184. A/B Testing

Step 1: Problem Statement

• User Funnel (Journey)

• Success Metrics:

  • Measurable: Can the behavior be measured using collected data?
  • Attributable: Can the behavior (effect) be assigned to the treatment (cause)?
  • Sensitive: Does the metric have low variability that can distinguish the treatment from the control?
  • Timely: How long will it take?

Step 2: Hypothesis Testing

• Hypothesis Statement (, )
• Set Significance Level ( usually 0.05)
• Set Statistical Power (usually 0.8)
• Set Minimum Detectable Effect (MD usually 1% lift)

Step 3: Experiment Design

• Set Randomization Unit
• Target Population
• Determine Sample Size

Where:

• : sample standard deviation

• : difference between control & measurement

• Determine Duration of Experiment

Step 4: Run Experiment

Step 5: Validity Check

• Instrumentation Effect
• External Factors
• Selection Bias
• Sample Ratio Mismatch
• Novelty Effect

Step 6: Interpret Results

Step 7: Make Decision