Sample Ratio Mismatch (SRM) & Telemetry Diagnostics

Identifying Silent Experiment Corruption

A Sample Ratio Mismatch (SRM) occurs when the observed ratio of visitors across experiment variants deviates significantly from the planned randomization split (e.g. 50/50 split resulting in 51,200 vs 48,800 visitors, (p < 0.001)). SRM is a severe indicator of systemic data corruption, indicating that one or more variants suffer from selective user attrition, crash events, redirection loop failures, or broken telemetry dispatch.

Any metric comparisons computed on a dataset afflicted with an unaddressed SRM are mathematically invalid and must not be used for business decisions.

Forensic Diagnostic Process

Step 1: Raw Split Verification (\chi^2 Test on Total Assignments)
Step 2: Multi-Dimensional Slicing (OS Version, Device Tier, Locale, Connection Type)
Step 3: Trigger Sequence Audit (Flag Evaluation vs Visual Render vs Event Dispatch)
Step 4: Attrition & Crash Correlation (Correlating Missing Samples with Exception Logs)
Step 5: Root Cause Isolation & Remediation Blueprint

What We Investigate

1. In-App Tracking Execution Order

We verify whether assignment events are recorded before or after the experimental UI component renders. In many mobile applications, if Variant B contains an unhandled exception that crashes the UI thread before sending the tracking event, Variant B appears falsely under-represented while appearing deceptively high in retention.

2. Client-Side Caching & Payload Loss

We evaluate local offline storage queues, background synchronization delays, and mobile network packet drop rates across experimental and control variants.

3. User Identifier Transitions

We audit state transitions when unregistered anonymous guest users log into registered profiles during an active test session, isolating double-counted units and allocation cross-contamination.

Deliverables & Outcomes

1. SRM Forensic Breakdown Report: Complete statistical analysis across all sub-cohorts identifying exactly where and why the allocation split degraded.
2. Telemetry Remediation Specification: Concrete code-level and architecture recommendations for mobile engineering teams to fix tracking order and event guarantees.
3. Validity Ruling: An explicit ruling on whether the historical experiment data can be salvaged through post-stratification weighting or must be discarded and re-run.

Engagement Intake

If your internal dashboards report an SRM warning or anomalous sample counts, contact our practice immediately via our Contact Form.