March 25, 2020

SAMPLE SIZE DETERMINATION

1. In order to determine the sample size necessary for an A/B Test, a sample size calculator requires:

   1. The baseline conversion rate

   2. The minimum detectable effect

   3. The statistical significance

2. Generally, sample size calculators use 4 parameters

   1. Margin of error.

   2. Confidence level.

   3. Population size.

   4. Expected proportion.

3. The Margin of Error is the furthest we expect the true value to be from what we measure.

4. Population Size - For large numbers, normally, 100,000 will suffice, as changes become negligible beyond that.

5. If we do not have historical data, we normally use 50%, which gives the most conservative sample size.

6. Baseline refers to the conversion rate of an A/B Test variant.

7. Minimum Detectable Effect refers to the percentage change we want to perceive between variants.

8. Confidence Level is how often the true value will lie within our specified range after multiple repetitions of the survey.

9. The more precise you want your answer to be, the more people you need to survey.