Do personalized recommendations really work? If so, just how much? Most of the data comes from the world of online retailing where it’s easier to track than print, but there are certainly applications to print we can learn from. Yesterday, I ran across an actual A/B split test that provided the kind of detail you don’t normally see.
The test [PDF download] came from Nova Pontocom, the second largest Latin American online retailer. It ran an experiment for one month involving three portals, nearly 600,000 different users, and 50 million page views and resulted in 1 million online orders generating revenues of $230 million.
[NOTE: I am getting feedback that the PDF download link is not working, but when I paste it into my browser, it works fine. URL is wanlab.poly.edu/recsys12/recsys/p277.pdf.]
“To the best of our knowledge, this is the largest scale controlled experiment aiming to assess the business value impact of personalized recommendations published so far,” write the authors of the report.
Users were randomly assigned to a treatment group that received personalized recommendations and a control group that did not. The personalized recommendations were generated by seven different collaborative filtering techniques based on product views, purchases, and shopping cart composition.
At the end of the month-long test, researchers found (with 95% of statistical significance) that the personalized recommendations resulted in an overall increase in revenues in the order of 8-20%.
Although online shopping and print marketing are different animals, the concept of providing a relevant offer based on the recipient’s own behavior has universal application. For those who’ve wondered just how much personalization can affect the bottom line when all other factors remain the same, this is some pretty strong data.