Crawlability test

What does the crawlability test check?

The crawlability test checks what one page hands to a crawler that reads HTML and runs nothing. It reports the response status, redirects and robots.txt rules in URL mode, and in both modes the indexing signals in the head, the title and description, the words of text, the headings, the links, the images and the structured data. Each item is stated as what was found.

How the text is counted: everything outside the head that is not inside a script, style, template, svg, iframe or textarea element. A word is a run of characters between spaces with at least one letter or digit in it. No CSS is applied, so text that a browser hides still counts, because a fetcher that applies no CSS reads it. There is no pass mark: how much text a page needs depends on the page.

What is the difference between crawlability and indexability?

Crawlability is whether a crawler may and can fetch a page. Indexability is whether a page that was fetched may be stored and shown in results. Robots.txt, the response status and redirects decide the first. A noindex and the canonical decide the second. A page needs both, and the two sets of rules can cancel each other out.

The most common trap is the first two rows used together. A page that is disallowed in robots.txt and also carries a noindex stays blocked from crawling, and Google's documentation on blocking indexing says the noindex only takes effect when the page is not blocked by robots.txt. To take a page out of a search index, let it be crawled and serve the noindex.

Passing both is still not a promise. Google describes search as three stages, crawling, indexing and serving, and says not every page makes it through each one. This test reads the first two from your side of the fence. It cannot tell you whether a page is in an index.

Which crawlers run JavaScript?

Googlebot runs JavaScript: Google documents that it queues pages and renders them in a headless Chromium. Apple says Applebot may render pages too. The crawler pages that OpenAI, Anthropic and Perplexity publish do not say whether their crawlers run JavaScript. Google's own advice covers the gap: it calls server-side rendering or pre-rendering a great idea because "not all bots can run JavaScript".

"Does not say" means exactly that: the page neither claims nor denies it. It is not evidence that a crawler skips JavaScript, and operators change how their crawlers work without notice. The practical reading is the same either way. Content in the HTML reaches every crawler on the first fetch, and content that needs JavaScript reaches only the ones that render, after a wait.

Rendering does not rescue everything, even for Google. Google says that when it meets a noindex in the HTML it may skip rendering and JavaScript altogether, so a script that removes a noindex may never run. The source a crawler receives first should already say what you mean.

How to fix what the test finds

Fix crawlability problems in the order a crawler meets them: access, then the response, then indexing signals, then content. A fix further down the list does nothing while something above it still stops the crawler. None of these steps guarantees a ranking or a citation. They remove reasons a page cannot be read.

1. Blocked in robots.txt. Remove or narrow the Disallow rule, and remember that a crawler with its own group ignores the * group. The AI visibility checker shows which rule decides for each AI crawler, and the robots.txt generator writes a file with the rules you choose.
2. An error status or a long redirect chain. Make the address answer 200 directly. Point internal links, the canonical and the sitemap at the final address, not at one that redirects.
3. A noindex you did not intend. Find where it comes from: the template, an SEO plugin setting, a staging flag left on, or a server rule that adds the X-Robots-Tag header.
4. A canonical that points elsewhere. If this page should be indexed, make its canonical its own address. If it should not, the canonical is doing its job.
5. Content that exists only after JavaScript. Render it on the server or pre-render it, so the facts are in the first HTML response: what is sold, the price, availability, policies and contact details. This is the fix Google itself recommends for crawlers that do not run JavaScript.
6. Links that are not links. Navigation built from buttons, click handlers or javascript: addresses gives a crawler nothing to follow. Use a elements with a real href.
7. JSON-LD that does not parse. Fix the syntax first, then check each type's properties with the schema markup validator. For a page with a visible FAQ, the FAQ schema generator writes the block.

After a fix, test again, then ask the search engine to look. Google says crawling can take anywhere from a few days to a few weeks after you ask it to recrawl, and that asking does not guarantee inclusion.

Why crawlability comes before conversion

A page converts only the visitors it gets, and a page that crawlers cannot read gets fewer of them from search and from AI answers. The same holds for AI agents that shop or research on a person's behalf: an agent that reads the HTML and finds no price cannot compare your offer. The AI agent readiness checklist covers that wider list, and AI traffic tracking shows how to see the visits AI assistants already send.

Being readable is the entry ticket, not the result. Once people arrive, the page still has to persuade them, which is what the free CRO audit measures: it loads the page in a real browser and scores the first screen, calls to action, forms, trust, mobile experience and speed, with AI visibility and agent readiness as one of its eight checks.