Imagine you own an online business. You want to improve how the options are presented to potential buyers. Hence, you make minor adjustments in the displaying options.
In the above-mentioned picture, A is the control. A is the business-as-usual option. B refers to the option with minor changes (variant). You can observe in the picture that there is a minor colour variation. Now you want to know which one is better? A or B? For this, you use A/B testing. But how do you measure which one is best? Is it from profit (a continuous variable)? Or from bought or not bought (a discrete variable)? Let us understand both of these types below.
A/B testing with continuous variable
In this case, to know whether there is any significant improvement in revenue per customer between these two options, you run A/B testing. A/B testing is called split testing. You show options A & B to equal number of random visiting customers and collect the revenue per customer. As the revenue per customer is a continuous variable ($ 10, $ 15, $18, etc.), we can use a two-sample t-test or the Z-test to compare the results. The following illustration shows - which test to use under different situations.
A/B testing with discrete variable
Suppose you want to compare the click-through rates between A and B, then it is a discrete (binary) variable (clicked yes - 1, otherwise 0). For discrete variables, the chi-squared test or Fisher's exact test can be used in analysing the results of A/B testing.
- Helps in optimizing - one step at a time
- Provides statistical evidence
- Widely used in email marketing, website designing, product pricing, segmentation and targeting etc.
- Can take lots of time and resources
- Can help in comparing specific options, but may not be useful when developing an entirely new product