Affiliate testing
Amazon affiliate split testing: why most tests fail, and the triple setup that fixes the data
You run a split test on an Amazon affiliate page, find a winner, roll it out, and your Amazon earnings do not follow. It happened to Convertica too, often enough to find the cause. This is what goes wrong with Amazon affiliate split testing, and the three-page setup Convertica built to make test results line up with Amazon's sales data.
Short answer
Amazon affiliate split testing means showing different versions of an affiliate page to different visitors at the same time, then comparing which earns more from Amazon. Most of these tests fail because testing tools only sort visitors whose browsers run their script, so Amazon's reports cannot be matched cleanly to each version. The triple setup addresses this with three pages.
- Give the control and the variant their own Amazon tracking IDs, so orders and ad fees can be traced to each.
- The triple setup duplicates the original page twice and splits traffic 10/45/45 between the original, the control and the variant.
- Point each test copy's canonical link at the original, in line with Google's guidance on website testing.
- Judge the test on orders and ad fees in Amazon's reports, because a variant can win on clicks and lose on earnings.
What is Amazon affiliate split testing?
Amazon affiliate split testing means showing different versions of an affiliate page to different visitors at the same time, then comparing which version earns more from Amazon. The sale happens on Amazon, not on your site, so the test only works if every order and every dollar of ad fees can be traced back to the version the buyer saw.
Before you test, know your numbers: how to measure affiliate marketing conversion rates comes first. Split testing is the most reliable way to know whether a change to an affiliate page, such as a new comparison table, earns more. Some pages will gain and some will lose, and the test is how you tell them apart. The trouble is that a split test on an affiliate site is harder to run than the textbook version, and many site owners give up after a test or two that never gives a clear answer.
Why do Amazon affiliate split tests fail?
Most Amazon affiliate split tests fail because the visitors in the test and the sales in Amazon's reports do not match. JavaScript testing tools only sort visitors whose browsers run their script. Everyone else sees the original page, so the original's tracking ID collects sales from visitors who were never in the test, and the comparison is skewed.
In a textbook test you set the original page as the control, create a variant, split visitors 50/50 and compare. Tools that load the variant with JavaScript cannot sort three groups of visitors:
- visitors with JavaScript disabled,
- visitors with cookies disabled,
- visitors on connections too slow for the script to run in time.
Call them the excluded traffic. They are not split 50/50. They all land on the original page. The testing tool leaves them out of its own report, but Amazon does not know who was in the test: every sale made through the original page's links counts toward the original's tracking ID. In Convertica's tests at the time, the original page often ended up with 1.5 to 2 times the views of the variant at a 50/50 setting. Match that against Amazon's sales report and the original looks better, or the variant worse, than it really is.
Two more problems make it worse. A variant that shares the original's tracking ID cannot be measured in Amazon at all, so the test can only be judged on clicks, and clicks are a poor proxy for earnings. And a test called on a small, early sample will often reverse when more data arrives.
What is the triple setup?
The triple setup is a way to run an Amazon affiliate split test with three pages instead of two, so the two pages you compare get equal, clean traffic. You duplicate the original page twice. One copy stays exactly the same and becomes the control. The other copy gets your changes and becomes the variant. The original keeps a small share of traffic.
| Page | What it is | Traffic setting | Amazon tracking ID | Used in the comparison? |
|---|---|---|---|---|
| Original | The page that ranks in search | 10% | Its existing ID | No |
| Control | An exact copy of the original | 45% | Its own new ID | Yes |
| Variant | A copy with the changes you are testing | 45% | Its own new ID | Yes |
The excluded visitors, the ones the tool cannot sort, still land on the original page. That no longer matters, because you ignore the original's data. The visitors the tool does sort go 45/45 to the control and the variant, and each of those pages has its own URL (for example, the post name plus a suffix) and its own Amazon tracking ID. Their sales in Amazon's reports now line up with the traffic in your testing tool.
Why keep 10% of traffic on the original?
The original page keeps 10% of traffic because it is the page that ranks in search, and sending all of its visitors to test copies would look like an attempt to game the system. Keeping real visitors on the ranking page is a precaution. As with a 50/50 test, the original's true share will be more than 10%, because the excluded traffic lands there too.
You might ask why not 50/50/0. Sending every sorted visitor away from the page Google ranks, for the length of a test, is the kind of setup that can be abused. A small, steady share on the original avoids the question.
Is duplicating pages for a split test safe for SEO?
Duplicating pages for a split test is consistent with Google's own guidance on website testing, provided each test copy points back to the original with a rel="canonical" link. That tells Google the copies are variations of the original page and that the original is the one to index. The original article reported no SEO problems from the many tests Convertica had run this way.
Google Search Central's page on minimizing A/B testing impact in Google Search sets out four rules, and the triple setup fits all of them:
- Do not cloak. Show Googlebot the same pages and content that visitors see.
- Use rel="canonical" on the test URLs, pointing to the original. Google prefers this to noindex for test pages.
- Use temporary (302) redirects, not permanent (301) ones, if your test redirects visitors from the original URL to a test URL.
- Run the test only as long as necessary, then remove the test pages and redirects.
No setup can promise how a search engine will treat a site. Follow the guidance, and take the copies down when the test ends.
How do you set up the triple setup in VWO?
To set up the triple setup in VWO, create a split URL test, add a second variation so the test has three pages, set the traffic to 10/45/45 and track clicks on Amazon links by ASIN. The steps below follow the original walkthrough. VWO's screens have changed since, but the same options exist, and other split URL testing tools work in a similar way.
Duplicate the original page twice
Use your theme's or CMS's duplicate function. Give each copy its own URL and add a canonical link on each copy pointing to the original.
Make your changes on one copy only
The control stays an exact copy of the original. Every change goes on the variant.
Give each copy its own tracking ID
Create two new Amazon tracking IDs and use one in all the Amazon links on the control and the other on the variant.
Create a split URL test
In VWO, create a split URL test with the original as the base URL, then add another variation so you have three pages: original, control and variant.
Set up the goals
Track clicks on links whose URL contains each product's Amazon ASIN, one goal per product in your comparison table or mini reviews.
Allocate the traffic
10% to the original, 45% to the control, 45% to the variant. Install the testing tool's code in the page header.
Preview, then launch
Check every page and every link before the test goes live.
How do you read an Amazon affiliate split test?
To read an Amazon affiliate split test, wait until both the control and the variant have enough views, then compare their orders and ad fees in Amazon's reports by tracking ID, not only the clicks in your testing tool. The original page's numbers are left out. A variant can win on clicks and lose on earnings.
The original article gave Convertica's rules of thumb at the time:
- At least 1,000 views on each of the control and the variant before calling a test.
- A lift above 15% before calling a winner on that sample. The smaller the difference, the more views you need to be sure.
- If a test points to a lift under 5%, scrap it and test something bolder.
- Look for 95% statistical significance before calling a winner.
Those rules were a practical shortcut. A better way is to set the sample size before the test starts, from your conversion rate and the smallest lift worth detecting, and not to stop early. The A/B test sample size calculator gives that number, and the statistical significance calculator checks the result. Statistical significance tells you how unlikely your result would be if there were no real difference. It does not tell you how big the difference is.
What can still go wrong?
The triple setup fixes the traffic and data problem. It does not make every test a winner, and it does not fix every mistake. These are the failures Convertica still saw, and you will too:
- The same change wins on one page and loses on another. In the 10Beasts case study, one new table raised revenue on six of ten pages and lowered it on four. Roll out winners page by page.
- Peeking. An hour into a test, the tool can show a possible gain of several hundred percent. The affiliate income case study shows how that settles as data arrives.
- The wrong products. A better table cannot sell products buyers do not want, products sold only by third parties or products with high return rates. Check your Amazon reports before you test layout.
- Too little traffic. Small sites can only detect big differences. Test bold changes on the pages that earn most.
What has changed since this method was written?
Three things have changed since Convertica first published the triple setup in 2018, and all three make clean data harder, not easier:
- More visitors block or refuse scripts. Ad blockers, privacy settings and cookie consent banners can stop a testing script from running, which adds to the excluded traffic. The triple setup handles that the same way it handles visitors with JavaScript off.
- Server-side and edge testing is easier to get. Assigning visitors on the server, before the page loads, puts every visitor in the test and removes the excluded traffic problem. You still need one Amazon tracking ID per version.
- Bots and AI agents load pages. They can be counted as visitors in a test without ever buying. Check your testing tool's bot filtering, and judge tests on Amazon orders and ad fees rather than clicks or views.
AI agents also read pages differently from people. A variant built by a testing script after the page loads may never be seen by an agent or by AI search. Once a test has a winner, build it into the page itself. AI agent readiness is one of the eight checks in Convertica's free CRO audit, and CRO advisory, led by Kurt Philip, can help you plan a testing program for your affiliate site while your team builds the changes, or Convertica's team can build the fixes for you with full implementation.
Amazon affiliate split testing FAQ
What is Amazon affiliate split testing?
Amazon affiliate split testing means showing different versions of an affiliate page to different visitors at the same time, then comparing which version earns more from Amazon. Because the sale happens on Amazon, not on your site, each version needs its own Amazon tracking ID so the orders and ad fees can be tied back to it.
Why do Amazon affiliate split tests fail?
Most fail because the traffic and the sales data do not line up. JavaScript testing tools only sort visitors whose browsers run their script. Visitors with JavaScript or cookies disabled, or slow connections, see the original page, so the original's tracking ID collects sales from visitors who were never in the test, and the Amazon data is skewed.
What is the triple setup for affiliate split tests?
The triple setup uses three pages instead of two: the original page, an exact copy of it called the control, and a changed copy called the variant. The testing tool sends 10% of traffic to the original and 45% each to the control and variant. You compare only the control and variant, each with its own Amazon tracking ID.
Is duplicating pages for a split test bad for SEO?
Not if you follow Google's guidance on website testing: put a rel="canonical" link on each test URL pointing to the original page, show Googlebot the same content as users, use temporary (302) redirects rather than permanent (301) ones, and end the test as soon as you have an answer.
How long should an Amazon affiliate split test run?
Until each version has the sample size you planned, which depends on your conversion rate and the size of difference you want to detect. Convertica's rule of thumb at the time was at least 1,000 views on each of the control and variant before even looking. A sample size calculator gives the real figure for your page.
Can I split test an affiliate site with little traffic?
You can, but only big changes on your top pages will show up in a reasonable time, because small differences need a lot of traffic to detect. Start with the pages that earn most, test bold changes rather than small tweaks, and expect some tests to take weeks.
Do I still need the triple setup with server-side testing?
Less so. A server-side or edge test assigns every visitor before the page loads, so visitors without JavaScript are not left out. You still need a separate Amazon tracking ID for each version, because that is the only way to tie Amazon's orders and ad fees back to the version a visitor saw.