- button type
- surrounding type
- text font
- font size
- any graphic you use
- product descriptions
- sales copy
- different offers (50% off, 35% off, free sample, etc.)
- a whole page
- a whole landing page
Then you set your control element/variable, and it is essentially what you have now, and you call that your A. Meanwhile, the element/variable you want to test as B, to be run simultaneously with the control (Christian, 2012; Patel, n.d.). The A and B variables are also known as variants, the challenger is the B variable, and the champion variable is the one that outperforms the others (unbounce.com, n.d.) For instance, 100% of the audience will be split into 50% of your site with variable A and the other 50% of your site with variable B. The split can vary from 50/50 to 60/40 to 70/30, etc. and it depends on how much weight you want to assign to the challenger variable (unbounce.com, n.d).
- If Gaussian is the assumed distribution (i.e. average revenue per paying user), you can use the Unpaired T-test and/or Student T-test (Amazon, 2015; Box et al., 1987; Pereira, 2007).
- If Binomial is the assumed distribution (i.e.click through rate), you can use Fisher’s exact test and/or Bernard’s test (Amazon, 2015).
- If Poisson is the assumed distribution (i.e. transactions per paying user), you can use the E-test and/or C-test (Krishnamoorthy & Thomson, 2004).
- If Multinomial is the assumed distribution (i.e. the number of each product purchased), you can use the Chi-square test.
- If the assumed distribution is unknown, you can use the Mann-Whitney U test and/or Gibbs Sampling.
Box, Joan Fisher. Guinness, Gosset, Fisher, and Small Samples. Statist. Sci. 2 (1987), no. 1, 45–52. doi:10.1214/ss/1177013437. https://projecteuclid.org/euclid.ss/1177013437
Christian, B. (2012) The A/B test: inside the technology that’s changing the rules of business. Retrieved from https://www.wired.com/2012/04/ff-abtesting
- Amazon.com (2015). The math behind A/B Testing. Retrieved from https://web.archive.org/web/20150921174256/https://developer.amazon.com/public/apis/manage/ab-testing/doc/math-behind-ab-testing
- Kolowich, L. (n.d). How to do A/B testing: A checklist you’ll want to bookmark. Retrieved from https://blog.hubspot.com/marketing/how-to-do-a-b-testing
- Krishnamoorthy, K. and Thomson, J. (2004). A more powerful test of comparing two Poisson means. Journal of Statistical Planning and inference. 119(1), 23-35. Retrieved from https://doi.org/10.1016/S0378-3758(02)00408-1
- Patel, N. (n.d.). A Beginner’s Guide to A/B Testing: An introduction. Retrieved from https://neilpatel.com/blog/ab-testing-introduction/
- Pereira, R. (2007). How beer influenced statistics. Retrieved from https://blog.gembaacademy.com/2007/06/20/how-beer-influenced-statistics/
- Unbounce.com (n.d.). What is A/B testing? Retrieved from https://unbounce.com/landing-page-articles/what-is-ab-testing/