The Peeking Problem and Sequential Testing in Continuous Product Pipelines

Why checking dashboard p-values daily inflates false positive error rates from 5% to over 30%, and how sequential testing boundaries protect decision validity.

In modern experimentation culture, product managers and engineers frequently monitor live analytics dashboards daily. When a metric achieves a reported (p < 0.05) on day 4 of a planned 14-day test, teams often stop the experiment early to declare victory.

This widespread practice is known as the peeking problem (or continuous monitoring bias). In standard fixed-horizon hypothesis testing, stopping an experiment the moment (p < 0.05) inflates your actual Type I error rate (false positive rate) from 5% to over 30%.

Why Peeking Inflates False Positive Rates

A standard two-sample t-test assumes that sample size (N) is fixed in advance and that the test statistic is evaluated exactly once.

When you inspect the data continuously as new users arrive, the cumulative test statistic follows a stochastic random walk (Brownian motion). By the law of the iterated logarithm, even if the true treatment effect is exactly zero ((H_0) is true), the test statistic is guaranteed to cross the arbitrary (\alpha = 0.05) boundary at some point if observed frequently enough.

Number of Peeks (Inspections) | True Type I False Positive Rate
-----------------------------|--------------------------------
1 (Fixed-horizon end)        | 5.0%
5 intermittent checks        | 14.2%
10 intermittent checks       | 19.3%
Continuous (Daily across 30d)| 31.8%

Solutions: Sequential Testing & Alpha Spending

To allow continuous monitoring without corrupting decision validity, statistical science developed Sequential Testing methodologies:

1. Group Sequential Testing (O’Brien-Fleming Boundaries)

Instead of using a static critical value ((z = 1.96)), group sequential testing adjusts the threshold based on information fraction (t = n / N):

[ c(t) = \frac{z_{\alpha/2}}{\sqrt{t}} ]

Early in the test ((t = 0.2)), the required critical z-score is extremely conservative ((z \approx 4.38), (p < 0.00001)). As the test approaches completion ((t = 1.0)), the boundary converges back toward (z \approx 1.96).

2. Always-Valid p-Values (e-Values & Martingales)

Using mixture sequential probability ratio tests (mSPRT), analysts construct confidence sequences that maintain guaranteed coverage across any arbitrary stopping time:

[ P\left( \exists n \ge 1: \mu \notin C_n \right) \le \alpha ]

Operational Takeaways for Engineering Teams