It seems that A/B testing articles, tools and methods are everywhere, and that if you’re not testing you’re probably doing something wrong. This makes perfect sense: that’s probably the best way to ensure optimization of your product or offer.
Although this approach is mostly the right way do it, sometimes it can
get a little out of hand.
Yes: there is such a thing as over testing.
Death by testing is a situation where a company (or a team) will not decide on anything unless it is tested and tested to death: instead of picking out the color of the font in a text, for example, they would test it,
comparing 2 colors each time for about 5-10 times to make sure they have the perfect color. The list goes on and on, with every minor detail (along with the major ones, of course) would be subjected to testing.
So what’s the problem?
The problem with over testing isn’t the test itself, it’s the effort and time wasted on issues with a marginal effect on the bottom line. All companies have a limited supply of time and manpower which should be put into use in the most effective and efficient manner possible.
The results of putting too much effort and time in the wrong
place could be grave, which is why sometimes testing can cause more damage than good.
How to detect over testing
Remember that A/B tests should result in effect on a KPI. If you can estimate that the resulting changes to the bottom line will be marginal
even if the test will prove extremely successful, then you probably shouldn’t test.
When should you test
You should test and test again in some scenarios, the most common ones are:
- Changes to major revenue drivers (your best selling item, holiday sales, a high traffic landing page etc.).
- Any changes to high traffic pages.
- Unknowns: A completely new page, without any reference for comparison in the history of the company
A/B testing is a very effective way to optimize your
product, but if the test results are worth less than the effort put into the test, you probably shouldn’t do it.