A/B Testing & Canary Deployments

Staging Never Mirrors Production

No matter how carefully you construct a staging environment, it differs from production in ways that matter for model performance.

• User demographics and behavior in staging are not representative
• Traffic volume differences mean cache behavior, serving latency, and batching dynamics differ
• Staging data may be anonymized or sampled in ways that change feature distributions
• Real-time feature values (current prices, live inventory) only exist in production
• Rare edge cases appear in production at much higher absolute volume

User Behavior Is the Ground Truth

The ultimate measure of a model's value is whether it improves user outcomes or business results. These can only be measured in production with real users making real decisions.

• Users respond to model outputs in complex, hard-to-simulate ways
• Recommendation model quality = did users click, engage, convert, return?
• Search ranking quality = did users find what they were looking for quickly?
• Only real user interaction data can measure these outcomes

The AUC-vs-Revenue Gap

Offline metrics (AUC, F1, NDCG) and online business metrics often move in opposite directions. A model with better AUC frequently performs worse in production because it optimizes the wrong objective.

• Higher AUC model may show less diverse recommendations → lower overall engagement
• Better calibrated probabilities may reduce clickbait → fewer clicks but higher satisfaction
• Models optimizing engagement can harm long-term retention
• The only way to measure the true objective is production A/B testing

Iterating Safely

Production testing allows rapid iteration while bounding risk. Small traffic percentages limit the blast radius of any single experiment.

• Test 5% of traffic → 95% of users are not affected if the model is bad
• Collect real signal in days rather than waiting for offline proxy metrics
• Fail fast: detect a bad model in hours, not weeks after full deployment
• Compound experiments: run multiple model variants simultaneously across user segments

Experimental Design Basics

A rigorous A/B test requires careful experimental design before collecting a single data point.

• Randomization unit: user-level (consistent experience) vs request-level (more samples); choose based on whether model affects session state
• Treatment vs control: new model (treatment) vs current production model (control)
• Primary metric: define ONE primary success metric before the test; secondary metrics are diagnostic only
• Guard rails: metrics that must not degrade (latency SLA, error rate, engagement floor)
• Test duration: at least one full week to capture day-of-week effects; ideally two weeks

Statistical Significance

Declare a winner only after achieving pre-specified statistical significance — not when the graph looks good or the numbers move in the right direction.

• Significance level α: typically 0.05; reject H0 when p-value < α
• Power (1-β): typically 80%; probability of detecting a real effect when it exists
• t-test: use for approximately normal continuous metrics (revenue, session duration)
• Mann-Whitney U: non-parametric alternative; robust to skewed distributions
• Chi-squared: for binary outcomes (click/no-click, convert/no-convert)

Sample Size Calculation

Determine minimum required sample size before starting the test, based on your MDE and baseline metrics.

from statsmodels.stats.power import TTestIndPower

# Parameters
alpha = 0.05       # significance level
power = 0.80       # desired power
baseline = 0.12    # current conversion rate
mde = 0.01         # minimum detectable effect (absolute)

# Effect size (Cohen's d)
effect_size = mde / (baseline * (1 - baseline)) ** 0.5

analysis = TTestIndPower()
n = analysis.solve_power(
    effect_size=effect_size,
    alpha=alpha,
    power=power,
    ratio=1.0,  # equal group sizes
    alternative='two-sided'
)
print(f"Required n per group: {int(n) + 1}")

Avoiding Common Pitfalls

• Peeking: stopping early when the test "looks significant" inflates false positive rate; use sequential testing methods (SPRT, always-valid p-values) if early stopping is needed
• Novelty effect: users interact differently with unfamiliar UI; run tests long enough for novelty to wear off (typically 1–2 weeks)
• Survivorship bias: if users churn during the test, their absence biases results
• Network effects: if users interact with each other, treatment and control are not independent
• Multiple testing: testing 20 metrics with α=0.05 means ~1 false positive by chance; apply Bonferroni correction

Progressive Traffic Routing

A canary deployment sends a small initial slice of traffic to the new model, then progressively increases as confidence builds.

• Stage 1 (1–5%): initial canary; watch for serving errors and latency regression
• Stage 2 (10–20%): expand if stage 1 is healthy; monitor quality metrics
• Stage 3 (50%): if metrics are positive at stage 2; collect statistical power for A/B conclusion
• Stage 4 (100%): full promotion after statistical significance confirmed
• Automated rollback at any stage if guard rail metrics breach thresholds

Traffic Splitting Infrastructure

• Istio (Kubernetes): VirtualService + DestinationRule define traffic weights between service versions; supports header-based routing for internal testing
• AWS CodeDeploy: native canary and linear traffic shifting for Lambda and ECS; built-in CloudWatch alarm integration for automated rollback
• Kubernetes ingress (NGINX/Traefik): weight-based traffic splitting via ingress annotations
• Envoy proxy: fine-grained traffic management; used by most service meshes under the hood

How Shadow Mode Works

The new model receives a copy of every production request, generates predictions, and logs them — but those predictions are never returned to users. The existing model continues serving all responses normally.

• Production proxy duplicates each request to both current model and shadow model
• Only the current model's response is returned to the user
• Shadow model predictions are logged alongside current model predictions
• Both sets of predictions are compared offline to detect divergence, errors, and distribution shifts

When to Use Shadow Mode

• High-stakes domains: healthcare, legal, financial decisions where a wrong prediction has severe consequences
• Regulated industries: models requiring audit trails and staged validation before any user impact
• Major architecture changes: switching model families (tree → neural network) where behavior may differ dramatically
• Latency validation: proving new model meets SLA under real production load before it affects users
• Cold start: validating a first-ever deployment of a model in a new serving stack

Limitations of Shadow Mode

• No real feedback loop: predictions not shown to users → cannot measure clicks, conversions, or engagement outcomes
• Double infrastructure cost: both models must handle full production request volume
• Latency validation only under observed load: cannot simulate traffic spikes beyond current volume
• No novelty effect data: cannot observe how users respond to the new model's specific outputs
• State divergence: models that update state (recommendation session context) will diverge over time in shadow mode

Epsilon-Greedy

The simplest bandit algorithm. With probability ε, route to a random variant (explore). With probability 1-ε, route to the current best variant (exploit).

• Simple to implement and understand
• ε is typically 0.05–0.1 in production
• ε-decay: reduce ε over time as confidence grows
• Limitation: treats all non-best variants equally; wastes exploration on clearly bad arms

Thompson Sampling

Maintain a Beta distribution posterior over each variant's true conversion rate. Sample from each posterior; route to the variant with the highest sample. Bayesian and naturally adaptive.

• Automatically concentrates exploration on promising variants
• Updates posteriors after each observation (fully online)
• Mathematically optimal for Bernoulli rewards under certain conditions
• Extends to continuous rewards via normal or log-normal priors

Upper Confidence Bound (UCB)

Select the variant with the highest upper confidence bound on its estimated reward. Exploration is driven by uncertainty — variants with less data get a confidence bonus.

• UCB1: add bonus term √(2 ln N / n_i) to each variant's mean reward
• Deterministic (unlike Thompson sampling) — useful for debugging and auditing
• Strong theoretical guarantees on cumulative regret
• LinUCB: contextual bandit variant — uses request features to personalize arm selection

Contextual Bandits

Standard bandits ignore request context — they give all users the same traffic allocation. Contextual bandits use features of the request (user segment, device, time of day) to route to the best arm for that specific context.

• Different model versions may be better for different user segments
• LinUCB and Thompson Sampling both have contextual extensions
• Vowpal Wabbit: production-grade contextual bandit library used at major tech companies
• Higher implementation complexity but dramatically more efficient