What does good CRO advisory look like?
Good CRO advisory does three things well: it keeps one ranked list of what to work on, it checks every test before launch, and it reads results honestly, including the losses. Most wasted CRO budgets fail at one of these three, not for want of ideas.
Prioritization: one ranked list, rebuilt as you learn
Every idea, from the audit, the team or the CEO, goes into one backlog and is scored the same way. Simple models such as ICE (impact, confidence, ease) or PIE (potential, importance, ease) are enough, as long as the scoring is written down. The prioritization section of the CRO audit guide shows both. Clear fixes, such as a broken form field, skip the queue: they get fixed, not tested. After each test, the list is re-ranked with what you learned.
Test design review before anything launches
Most failed tests were decided before launch. A consultant reviewing a test should be able to tick every line of this list.
- The hypothesis names the problem, the change and the evidence behind it.
- One primary metric is chosen in advance, tied to revenue, orders or qualified leads.
- Guardrail metrics are named, such as average order value or refund requests, so a "win" that hurts the business is caught.
- The sample size and the stop date are set before launch and cover full weeks, so weekday and weekend visitors are both counted.
- The variation has been checked on real phones and browsers, with tracking firing on both versions.
- Nobody changes the tested pages while the test runs. If someone does, the test restarts.
Reading results honestly
An inconclusive test is a result: it tells you the change did not matter much, which frees the team to try something bigger. Losing tests are reported as plainly as winners. Slicing results by device or source until something looks significant is not a finding; a segment worth trusting is one named before the test.
Good advisory also reports more than one number. In Convertica's dScryb case study, paid membership sign-ups rose 49.4% and revenue per visitor rose 23.8% in the same 17-day test. Both are true, they are not the same number, and the decision depends on which one the business needs.
Teaching your team as it goes
The best sign of good advisory is a team that writes better hypotheses and spots a broken test on its own after a few months. If your team is no more capable than when you started, the advice was not landing.