Estimating long-term detection, win, and error rates in A/B testing

From statistical power to detection/win rates

You may wonder why the answer to long-term detection rates isn’t simply the Type II error rate or “We will detect (i.e., declare them as statistically significant) differences when they exist 80% of the time”.

That’s because the Type II error rate is conditional on the MDE. Assuming you use the same MDE all the time, you can claim that when the actual effect is at least as large as the MDE, we will detect it at least 80% of the time.

But what if it’s not? And how exactly does one think about the “at least” part of the statement? What if actual effects are larger/smaller than MDE? Your power will be exactly 80% only if you always choose MDE precisely equal to the actual unobservable difference for that specific experiment. If you can do that, you don’t need to experiment!

Frequentist approaches tend to develop tools that find upper/lower bounds of a measure of interest, and statistical power is an excellent example of that (I saw this clearly articulated in an essay by Jacob Steinhardt titled “Beyond Bayesians and Frequentists” - I highly recommend it).

However, we are after an expectation—we want to know the average expected detection rate. For that, we’ll borrow an idea from Bayesian statistics that tends to be better suited for such purposes. Instead of working with fixed values, we will assume a distribution of likely effect sizes (a.k.a., a prior distribution) and then integrate over that distribution to get expected rates.

To be precise, by integrating over a distribution of hypothesized treatment effect sizes $A$ (we’ll call the distribution $F_A$), we can calculate:

• Expected “detection rate” - the % of experiments we would declare statistically significant given they have a non-zero actual treatment effect.

• Expected “win rate” - the % of experiments we would declare statistically significant, but we count only experiments with a positive treatment effect.

Mathematically:

$$\text{expected detection rate} = \int_{-\infty}^{+\infty} F_A(x) \text{Pr}(\text{reject } H_0 | x \neq 0, \alpha, n, \sigma^2) dx$$ $$\text{expected win rate} = \int_0^{+\infty} F_A(x) \text{Pr}(\text{reject } H_0 | x \neq 0, n, \alpha, \sigma^2) dx$$

where $n$ is the sample size, $\sigma^2$ is the variance of the outcome metric, and $\alpha$ is the chosen statistical significance level.

One immediate thing to note is that detection/win rates do not depend on $MDE$ or $\beta$ (statistical power). The only levers you have to increase them are larger samples, variance reduction techniques (e.g., by using covariates, such as CUPED-like methods), better treatments (changes in $F_A$), and different $\alpha$ levels.

If your immediate reaction is, “OMG, a subjective assumption on effect size distribution (gasp!)”, I would like to remind you that choosing the MDE threshold for power calculations is subjective, too. Ideally, one would pick an MDE upfront and tune the sample size to achieve the desired statistical power. However, in practice, there’s always an incentive to cherry-pick an MDE itself, given a fixed sample size. It looks better to report a power of 80% with an MDE of 0.015 than a power of 50% with an MDE of 0.01.

Having said that, if you use MDE correctly and report statistical power together with the MDE, bad MDE choices are more transparent than flawed effect size distribution assumptions. But if we want to estimate average detection/win rates, that’s the cost we need to pay - there’s no free lunch!

eff_size_distrs = list(

  list(

    name = 'Startup team',

    distr = 'x ~ N(0.02, 0.03)',

    generator = function() rnorm(1, mean=0.02, sd=0.03),

    density = function(x) dnorm(x, mean = 0.02, sd = 0.03) 

  ),

  list(

    name = 'Mature team',

    distr = 'x ~ Gamma(2,200)) - 0.005',

    generator = function() rgamma(1, shape=2, rate = 200) - 0.005,

    density = function(x) dgamma(x + 0.005, shape=2, rate = 200)

  )

)

emp = lapply(

  eff_size_distrs,

  function(d) {

    r = list()

    r[[d$name]] = sapply(1:50000, function(i) d$generator())

    r

  }

) |> bind_cols() |> pivot_longer(everything(), names_to = "Assumed distribution")



ggplot(emp) + 

  #geom_density(aes(x=value, color=`Assumed distribution`)) +

  sapply(

  eff_size_distrs, 

  function(d) stat_function(

    aes(color=d$name, fill=d$name), 

    fun = d$density,

    data = tibble(name = d$name, full_name = paste(d$name, '', d$distr)),

    geom="area",

    alpha=0.5

  )

) + 

  xlim(-0.05, 0.1) + labs(

    y='Density',

    color='Assumed effect size distribution',

    x='Effect size',

    title='Some possible effect size assumptions'

  ) + 

  theme_light() + 

  theme(

    panel.grid.major = element_blank(),

    legend.position = "none"

  ) + facet_wrap(~full_name) +

  geom_vline(xintercept = 0, linetype = 'dotted')