Why Most Affiliate Site Split-Tests Fail (and how to fix it)

I have discussed this topic in previous posts but there is still a lot of confusion about the triple set up for affiliate split tests.

In this post I will walk through in more detail and show you the solution to a major issue for affiliate SEOs:

How to run a split test on your affiliate site the right way.

There is a great trick to split-testing that, as far as we know, we are the ONLY ones doing.

It’s something that we are extremely proud of and psyched to share with everyone for the first time.

Occasionally your split tests may return wonky results and you think you have a winning variation. But then you roll out the winning variation and it doesn’t line up with your split test data.  Why?

Use this technique, and you’ll finally get accurate data and great results.

Do you want a 20-100% conversion rate increase?


Why Your Split Tests Aren’t Providing Clear Answers

When making major changes to your Amazon affiliate sites (eg. Changing your comparison table layout), you absolutely have to run split tests to make sure those changes will drive more revenue. Otherwise you’re just speculating.

Some pages will see increases and some will see decreases. We run split tests to work out which ones are the winners and which ones are the losers so we can use data to increase our revenue.

The problem is that running split tests on affiliate sites is pretty tricky. In fact, a lot of SEOs get discouraged when they try to run their own tests for the first time and don’t get any conclusive results.

Split testing your sites correctly is an extremely effective way to multiply your revenue with existing traffic. You’ve just got to do it the right way.

In a ‘normal’ test, we’d just set the control (original page), create a variant (style we want to split test), allocate the visitors 50/50 to each variation, and watch as they battle it out for supremacy.

If only life were that easy….

This textbook way just doesn’t work in real life. Actually, it’s the reason why the majority of all split tests fail to show conclusive data.

Here’s why…..

If you’re running split tests in any of the split testing tools that use javascript to load the variations, and you set the traffic to split 50/50 to original and variant, then your original page will still end up getting 1.5x-2x the number of views. You may have noticed this discrepancy in your own tests and wondered why traffic isn’t being split equally. This preference for the original will skew your results MASSIVELY and give you a distorted picture of sales.

The Amazon sales data you get back will be so skewed that you won’t be able to draw accurate conclusions on the effect on revenue.

There are 3 reasons for this:

  • The testing software doesn’t sort visitors who have disabled javascript
  • The testing software doesn’t sort visitors who’ve disabled cookies
  • The testing software doesn’t sort visitors with a slow internet connection

We’ll call this the “excluded” traffic. This excluded traffic adds up to a huge visitor base that isn’t being subjected to your test, and are simply shown the original page. 

These ‘unsorted’ visitors aren’t being included in the 50/50 traffic split, and end up as ‘extra’ traffic on the original page.

Therefore, the reason your split tests are failing, isn’t that split-testing doesn’t work for Amazon affiliate sites. It’s that your traffic isn’t properly allocated and causing you to get skewed results.

After running test after test, we’ve devised a really badass, MacGyver-like hack to get your test results and Amazon sales data to line up right again. I’m super pumped to be sharing this insight with you.

Here’s what you’ve got to do….

Introducing Convertica’s affiliate site split-test traffic hack:  The Triple-Setup

We decided to share a technique we developed which has revolutionalized our split testing process.

Split-testing Amazon affiliate sites isn’t all that common to begin with.

If you want accurate results, instead of splitting traffic 50/50 between 2 pages, you’ve got to run the Convertica Triple-Setup. We call it that because you need 3 pages instead of 2:

You get this by duplicating the original page TWICE (don’t worry, it’s 100% safe for SEO. We’ll explain below).

This extra duplicate (control page) is an EXACT copy of the original, and all changes are made ONLY to the third page (the Variant).

Then you send 45% of the traffic each to the duplicate control page and the variant page, with only 10% to the original.

Still with me?

Here’s what the 3 pages look like:

  • Original: 10% of Traffic
  • Control (exact duplicate of the original): 45% of Traffic
  • Variant (the changed page): 45% of Traffic

Here’s a visual representation:

NOTE: Duplicating the original should be easy. Just use your theme’s default post duplication function.

This way, all the visitors who don’t get picked up by the split-testing software (the excluded visitors with disabled javascript, disabled cookies, and slow connections) get shown the original page. The remaining visitors that get picked up by the split-testing software are sorted into exactly 45%-45%-10% as above.

Here’s What You Do Next….

From there, you give the Control and Variant pages unique URLs (for example, postname-2018) and set up unique Amazon tracking-IDs for each.

By segmenting the traffic 45/45/10, we achieve perfect segmentation and calibrate the data to align with Amazon. Pretty cool, aye?

Do you want a 20-100% conversion rate increase?


Understanding the Distinction Between Traffic Segmentation Settings and Reality

If you recall, we determined that setting traffic segmentation to 50% – 50% would result in more than half the true traffic seeing the original. (The 50% assigned to original + the excluded traffic.)

Likewise when we set the traffic segmentation to 45% – 45% – 10%, more than 10% of the true traffic will see the original page. (The 10% assigned to original + the excluded traffic.)

But in this case, that’s ok, because you’re not using any of the sample data going to the original page. You’re only interested in the equal traffic that’s gone to your Control and Variant pages, and their corresponding revenue data over on Amazon (through their unique tracking-IDs).

What’s the 10% for? Shouldn’t we just allocate everything to the duplicates and make it 50-50-0?

We don’t want Google to think we’re gaming the system or trying any type of trickery. So as a precautionary measure, we want to make sure the original variation still gets some traffic, at least 10%. After all, that is the page that is ranking in the SERPS. From Googles perspective, if they allowed 100% redirection of traffic away from the ranking page in the name of split-testing, that could easily be abused.

Quick Review

We know this is a bit confusing so let’s review it quickly. It’s actually not that hard at all once it clicks.


  1. Duplicate the original twice
  2. Add a canonical tag on both duplicates back to the original page (to make safe for SEO)
  3. Keep the first duplicate an exact copy of the original: this becomes the Control page
  4. Make all your changes (that you want to test) to the second duplicate: this becomes the Variant page
  5. Send 45% of traffic to each Control and Variant, and only 10% to the Original
  6. This will calibrate your results to make up for the traffic that VWO doesn’t track, and ensure Control and Variant receive equal traffic.


Talk about life-hacks.

But hold on…

Duplicate Pages? Wait a Second. Is that Safe for SEO?

Telling an SEO to duplicate pages is like telling a vegan to eat beef. Don’t worry. This is 100% safe for SEO as long as you add a canonical tag to the original page. We’ve run well over 1,500 tests by now and never had any penalties as a result of duplication:

This tag tells Google to honor the original page ONLY and ignore the duplicates. The duplicates won’t affect your SEO at all.

Google has even confirmed this, and you won’t have any SEO problems.

OK, so now that you know how to run the test and make it safe for SEO, let’s run down how to set up the test in VWO.  The process will be similar across all split-testing platforms.

How to do the Triple-Setup Hack in VWO

As a default, VWO will only create a control and a variation, so you’ve got to do a bit of CRO mad-science to get the tri-setup configuration set up.

  • Go into your VWO dashboard
  • Click “Split URL” in the left-sided menu
  • Click “Create”

Click “Add Another Variation” to get your 3rd page, and give your duplicates a unique URL.

OK, almost there!

Next, Set up Your Goals

Goals are extremely important in CRO (and life too). There needs to be an identifier within the URL that shows us when a visitor clicks through to Amazon from that duplicate. This allows us to track revenue and match them to test data.

You must track Clicks on Link and Link Contains.

Add the Amazon ASIN number for each of the products in your comparison tables/mini review areas. You only need to add them once.

Finally – Allocate Your Traffic and Install the Header Code

Now that your test is set up, you just need to allocate the traffic according to our hack (45/45/10) and copy/paste the VWO Smart Code in just after the header.

Before you launch your test, be sure to preview it and make sure everything is in order. Once you’ve confirmed that it’s all good to go it’s time to launch your test and let the (accurate!) results roll in.

How to Analyse Your Test for Wins

As a rule of thumb, we don’t even think about calling a test until we’ve got at least 1,000 views to EACH Control and Variant, and above a 15% increase in conversions. The lower the conversion increase the more views you will need to ensure statistical significance. (In other words, the smaller the difference, the more sample data you need to be really sure.) If a test is suggesting under 5% increase in conversions, we scratch the test and start again.

In general, the higher the difference in conversions the fewer views required to draw statistical conclusions. For example, if a Variation is crushing the Original (actually, the duplicate of the original) by 30%, then we need fewer views to confirm the hypothesis than if it were leading by only 5%. Get what we mean?

VWO will show you the statistical significance of your test. (Statistical significance is the mathematical certainty that your test is NOT due to random chance. If statistical significance is at 80%, then you can be 80% certain that your test is actually showing correct results and not just a random fluke. In general, we look for 95% before calling a winner in a test.

TL,DR – Why Most Amazon Split Tests Fail

A lot of affiliate SEOs get discouraged when they run split tests on their Amazon affiliate sites and receive no insights.

The problem isn’t that split testing doesn’t work, it’s that tools like VWO by default don’t allocate traffic perfectly. This skews your results and leads to misinformed decisions. This is because:

  • VWO doesn’t track visitors who have disabled javascript
  • VWO doesn’t track visitors who’ve disabled cookies
  • VWO doesn’t track visitors with a slow internet connection

If you want more accurate results, you’ve got to run the “triple-setup” technique. Duplicate your original page twice and make all of your changes to one Variant page and keep the other copy as Control. Then, send 45% of your traffic to each, and only 10% to the original. Give each page a unique URL and tracking ID, and you are good to go!

This is 100% safe for SEO if you add a canonical tag to the original page.

REMEMBER: The Control page is an EXACT copy of the original. Make all of your changes ONLY to the Variant page.

There you have it! It might seem complicated at first, but using this awesome split testing hack will return accurate Amazon sales data and paint a clear picture of what is and is not working on your affiliate sites. Split-testing definitely works on affiliate sites, you’ve just got to do it right!

Do you want a 20-100% conversion rate increase?


Author: Kurt Philip


Submit a Comment

Your email address will not be published. Required fields are marked *

Hi, I’m Kurt Philip, the founder & CEO of Convertica. I live and breathe conversion rate optimization. I hope you enjoy our findings.

We’ve worked with over 1000 businesses in the last 6 years.
Let’s jump on a quick call to see how we can help yours.

Book FREE CRO Call

Client Case Studies

Follow us on Youtube