A: There are two main statistical analysis methods. The first is called a Student’s T-Test and the second is known as a multi-armed bandit.
The big benefit of using the multi-armed bandit approach is that you have a lot more flexibility in looking at the results, while a classic Student’s T-Test is pretty structured in its analysis. With a Student’s T-Test, you need to figure out ahead of time how much traffic you need for the test, how long you want to run the test for and how much statistical power you're expecting. You can look at all that data, use a sample-size calculator and then figure out how long you need to run the test. And then you should only look at the results at the end of that period.
Only checking the results at the end can be pretty challenging to do in business, because you have to teach everyone in your company enough about the statistical analysis to stay true to that requirement. Often what happens is people will want to see how it's going one hour in, two hours in, two days in. And after one or two hours, it can look like your test is really doing terribly and you're losing money. But there's a lot of variation at the beginning and you have to wait for the whole sample size to be reached at the end before you look at the results. So we see people cheat and not wait the whole time, which invalidates the test.
With the multi-armed bandit approach, you can look at the results whenever you want. You'll always see an accurate result and you can look at the results early without affecting the results of the test.
Another difference between the two tests is that with Student’s T-Tests you can understand a lot about each of the variations that you tested. So you can see which one was the winner and you can also see which one was the worst. With a multi-armed bandit, you can see which one's the winner, but you can't compare the other ones. It's really hard to tell which one was the second best or third best or fourth best. If you just want to know the winner, a multi-armed bandit is great. If you want to do a more scientific approach and understand each of your options, then the Student's T-Test is a better statistical method to use.