Tag: Multinomial

A/B Testing

Are you a HiPPO? HiPPO stands for the Highest-Paid Person’s Opinion who designed websites or gives their opinion on how things ought to be (Christian, 2012). This may not be a good thing, because the HiPPOs may not be the best person to get the maximum traffic to and through your website. Proponents for A/B testing state that the advantages of using A/B testing are greater than the time it takes to conduct it in the first place (Christian, 2012; Patel, n.d.). Whereas, Patel (n.d.), further claims that  “A/B tests, done consistently, can improve your bottom line substantially.”  Therefore, A/B testing helps the data scientist to narrow down which element/variable makes an effective difference towards their goal, i.e. click-through rate within a website to generate more revenue (Christian, 2012; Patel, n.d.; Unbounce.com, n.d.).
First, you need to know what to test or which elements/variables you want to test. It is key to  know what you want to test, the current baseline result, what you are testing for, and the goal you want to reach (Patel, n.d.) If you have a click funnel for your audience and they are dropping out at a certain level, you may want to use that area to improve fallout rates.  Once you know what to test, make a list of all the variables you would like to test (Christian, 2012; Patel, n.d.; Unbounce.com, n.d.):
  • location
  • color
  • button type
  • surrounding type
  • text font
  • font size
  • any graphic you use
  • product descriptions
  • sales copy
  • verbiage
  • different offers (50% off, 35% off, free sample, etc.)
  • a whole page
  • a whole landing page

Then you set your control element/variable, and it is essentially what you have now, and you call that your A. Meanwhile, the element/variable you want to test as B, to be run simultaneously with the control (Christian, 2012; Patel, n.d.). The A and B variables are also known as variants, the challenger is the B variable, and the champion variable is the one that outperforms the others (unbounce.com, n.d.)  For instance, 100% of the audience will be split into 50% of your site with variable A and the other 50% of your site with variable B. The split can vary from 50/50 to 60/40 to 70/30, etc. and it depends on how much weight you want to assign to the challenger variable (unbounce.com, n.d).

Another thing to consider is what statistical test you want to apply to the A/B test:
  • If Gaussian is the assumed distribution (i.e. average revenue per paying user), you can use the Unpaired T-test and/or Student T-test (Amazon, 2015; Box et al., 1987; Pereira, 2007).
  • If Binomial is the assumed distribution (i.e.click through rate), you can use Fisher’s exact test and/or Bernard’s test (Amazon, 2015).
  • If Poisson is the assumed distribution (i.e. transactions per paying user), you can use the E-test and/or C-test (Krishnamoorthy & Thomson, 2004).
  • If Multinomial is the assumed distribution (i.e. the number of each product purchased), you can use the Chi-square test.
  • If the assumed distribution is unknown, you can use the Mann-Whitney U test and/or Gibbs Sampling.
Testing can go on for a few days to a few weeks depending on the amount of traffic you get (Patel, n.d.).  Something like Facebook can start an A/B test on Monday and have a reported result by Friday, whereas my current state of the blog may have to take about a month or two. However, too long of a test for the wrong set of traffic or just in general can include confounding variables, which will skew your results.
To test three variations, also known as a multivariate test, according to Patel (n.d.), you need to set up an A/B test, a B/C test, and a C/A test and you want to give it a bit more time to have enough data.  When doing a multivariate test, you want to give them equal weight to pick a champion variable quickly (unbounce.com).
Another great article to view is from Kolowich (n.d), which provides a checklist for successfully conducting an A/B Test.
Resources