Analysis of 115 A/B Tests: Average Lift is 4%, Most Lack Statistical Power
Oct 2022 update: A newer, much larger and likely less biased meta analysis of 1,001 tests is now available! What can you learn from 115 publicly available A/B tests? Usually, not much, since in most...
View ArticleWhat Can You Learn From Running an A/B Test for 2 Years?
We just concluded an A/B test on Analytics-Toolkit.com that has been left to run for just over 2 years. And it failed, as in failing to demonstrate a statistically significant effect based on the...
View ArticleUpdates to Our A/B Testing Statistical Calculators
This short post presents several changes, updates, and new tool functionalities released in the past couple of months – some very recent, some less so. The updates concern all statistical calculators...
View ArticleWhat Can Be Learned From 1,001 A/B Tests?
How long does a typical A/B test run for? What percentage of A/B tests result in a ‘winner’? What is the average lift achieved in online controlled experiments? How good are top conversion rate...
View Article