A/B Testing Interview Questions

This document provides a curated list of A/B Testing and Experimentation interview questions. It covers statistical foundations, experimental design, metric selection, and advanced topics like interference (network effects) and sequential testing. Critical for roles at data-driven companies like Netflix, Airbnb, and Uber.

Code Examples

1. Power Analysis and Sample Size (Python)

Calculating the required sample size before starting an experiment.

from statsmodels.stats.power import TTestIndPower
import numpy as np

# Parameters
effect_size = 0.1  # Cohen's d (Standardized difference)
alpha = 0.05       # Significance level (5%)
power = 0.8        # Power (80%)

analysis = TTestIndPower()
sample_size = analysis.solve_power(effect_size=effect_size, power=power, alpha=alpha)

print(f"Required sample size per group: {int(np.ceil(sample_size))}")

2. Bayesian A/B Test (Beta-Binomial)

Updating beliefs about conversion rates.

from scipy.stats import beta

# Prior: Uniform distribution (Beta(1,1))
alpha_prior = 1
beta_prior = 1

# Data: Group A
conversions_A = 120
failures_A = 880

# Data: Group B
conversions_B = 140
failures_B = 860

# Posterior
posterior_A = beta(alpha_prior + conversions_A, beta_prior + failures_A)
posterior_B = beta(alpha_prior + conversions_B, beta_prior + failures_B)

# Probability B > A (Approximate via simulation)
samples = 100000
prob_b_better = (posterior_B.rvs(samples) > posterior_A.rvs(samples)).mean()

print(f"Probability B is better than A: {prob_b_better:.4f}")

3. Bootstrap Confidence Interval

Calculating CI for non-normal metrics (e.g., Revenue per User).

import numpy as np

data_control = np.random.lognormal(mean=2, sigma=1, size=1000)
data_variant = np.random.lognormal(mean=2.1, sigma=1, size=1000)

def bootstrap_mean_diff(data1, data2, n_bootstrap=1000):
    diffs = []
    for _ in range(n_bootstrap):
        # Sample with replacement
        sample1 = np.random.choice(data1, len(data1), replace=True)
        sample2 = np.random.choice(data2, len(data2), replace=True)
        diffs.append(sample2.mean() - sample1.mean())
    return np.percentile(diffs, [2.5, 97.5])

ci = bootstrap_mean_diff(data_control, data_variant)
print(f"95% CI for difference: {ci}")

Questions asked in Google interview

• Explain the difference between Type I and Type II errors.
• How do you design an experiment to test a change in the Search Ranking algorithm?
• How to handle multiple metrics in an experiment? (Overall Evaluation Criterion).
• Explain the trade-off between sample size and experiment duration.
• Deriving the variance of the difference between two means.
• How to detect if your randomization algorithm is broken?
• Explain how you would test a feature with strong network effects.
• How to measure the long-term impact of a UI change?
• What metric would you use for a "User Happiness" experiment?
• Explain the concept of "Regression to the Mean" in the context of A/B testing.

Questions asked in Meta (Facebook) interview

• How to measure network effects in a social network experiment?
• Explain Cluster-based randomization. Why use it?
• How to handle "Novelty Effect" when launching a new feature?
• Explain CUPED (Controlled-experiment Using Pre-Experiment Data).
• How to design an experiment for the News Feed ranking?
• What are the potential bounds of network interference?
• How to detect if an experiment has a Sample Ratio Mismatch (SRM)?
• Explain the difference between Average Treatment Effect (ATE) and Conditional ATE (CATE).
• How to optimize for long-term user retention?
• Design a test to measure the impact of ads on user engagement.

Questions asked in Netflix interview

• How to A/B test a new recommendation algorithm?
• Explain "Interleaving" in ranking experiments.
• How to choose between "member-level" vs "profile-level" assignment?
• How to estimate the causal impact of a TV show launch on subscriptions? (Quasi-experiment).
• Explain the concept of "Proxy Metrics".
• How to handle outlier users (e.g., bots, heavy users) in analysis?
• Explain "Switchback" testing infrastructure.
• How to balance "Exploration" vs "Exploitation" (Bandits)?
• Design a test for artwork personalization (thumbnails).
• How to measure the "Incremental Reach" of a marketing campaign?

Questions asked in Uber/Lyft interview (Marketplace)

• How to test changes in a two-sided marketplace (Rider vs Driver)?
• Explain "Switchback" designs for marketplace experiments.
• How to handle "Spillover" or "Cannibalization" effects?
• Explain "Difference-in-Differences" method.
• How to measure the impact of surge pricing changes?
• Explain "Synthetic Control" methods for city-level tests.
• How to calculate "Marketplace Liquidity" metrics?
• Design an experiment to reduce driver cancellations.
• How to test a new matching algorithm?
• Explain Interference in a geo-spatial context.

Additional Resources

• Microsoft Experimentation Platform (Exp) - Best technical papers.
• Netflix Tech Blog - Experimentation - Real-world case studies.
• Causal Inference for the Brave and True - Python handbook.
• Trustworthy Online Controlled Experiments (Book) - The "Bible" of A/B testing (Kohavi).
• Uber Engineering - Data - Marketplace testing concepts.