Tag: statistical-power

  • A/B testing, zero-inflated (truncated) distributions and power

    A/B testing, zero-inflated (truncated) distributions and power

    Naive A/B testing just uses t-tests or proportion tests, with the assumption that at large sample sizes, the right statistical test does not matter that much. I explore the case of a zero-inflated upper-bounded Poisson distribution and find that using the wrong test can require 3x the sample size to achieve the same statistical power, a difference large enough to matter in a real business setting.