AB Testing Do's & Don'ts for your eCommerce Store

Picture for category AB Testing Do's & Don'ts for your eCommerce Store

The ideal website doesn't exist. In case you were looking for it, we are sorry to disappoint you but there's no toolbox for the perfect landing page or form or product page. But "isn't UX just about guidelines & best practices?" you might wonder. According to Karl Gilis Els Aerts who hosted Sherpa Society's masterclass entitled "AB Testing & Optimization for eCommerce", the answer is "yes and no". Here is why"

  • Yes, there are many best practices and guidelines 
  • But there's much more than the guidelines 
  • Otherwise it would be pretty easy to make the perfect website 


You have to find out what works best for your website, for your audience and for your product/service. And for this to happen you need a process based on research and data, instead of wild guesses.

  1. Where & What? Where do people leave? What's the customer journey?
  2. Why? Why is that? You'll need to do some research for this.
  3. How to fix it? The bigger your knowledge of usability, UX, psychology, persuasion, …, the better you'll be at finding alternatives that will work.
  4. Measure & refine


So, with these in mind, let's focus on AB-testing! 

For starters, AB-testing is not a research tool. Use it to validate your homework. How can you create a good testing plan? 

1. Write solid hypotheses


Fyi, a hypothesis is a structured idea that tells you where it came from, how it's supposed to work & what the intended result is. And a simple formula to use is the following, as suggested by Craig Sullivan:

Because we saw (data/feedback), we expect that (change) will cause (impact). We'll measure this using (data metric). Simple, right?

2. Set your priorities. You can't test every idea. 

Most importantly though, always remember that...


Exactly! AB-testing without user research = gambling. 

Here are some of AB-testing do's & don'ts that Karl & Els highlighted during their session:

  1. For solid AB-testing you need at least 1.000 conversions per month. Otherwise, the result might be a statistical coincidence. 
  2. Make sure you know your sample size (use a size calculator like this one here: https://cxl.com/ab-test-calculator/).
  3. Take into account data pollution (often caused by cookies, seasonality, devices, etc). 
  4. Your AB-testing should run for at least 1 week and a maximum of 1 month.
  5. Don't be afraid to test big changes. Dare to test big.
  6. Don't test stupid stuff and/or details that don't matter.
  7. Test your test. For instance, does it work properly in all browsers?
  8. Don't stop a test too early unless results are alarmingly bad and are so consistently over 3+ days, or if you realize that the test has errors.
  9. Don't forget to segment. And especially don't use post-segmentation to turn a loser into a winner.
  10. Measure the right thing.
  11. Analyze results (and double-check them).
  12. Create follow-up tests.

Of course, this was just a glimpse of what our 3-day online training was all about. However, take into account their advice and  make sure you join our newsletter for more insights, tips and upcoming masterclasses. 


👉 Interested in hiring Karl Gilis & Els Aerts  for a custom in-house training? Let's chat!