Robots.txt generator

How do you create a robots.txt file?

To create a robots.txt file, make a plain text file named robots.txt, add groups of rules that each start with a User-agent line, upload the file to the root of your site, and test it. Google's documentation gives the same four steps. The generator above does the second one and tests the result.

1. Create a file named robots.txt. Use a text editor, not a word processor, and save it as UTF-8. Word processors can add curly quotes and other characters that crawlers cannot read.
2. Add the rules. Each group starts with one or more User-agent lines and continues with Disallow and Allow lines, one rule per line.
3. Upload it to the root of the host. How depends on your hosting or platform. Some platforms write the file for you and offer a setting instead.
4. Test it. Before upload with the tester above, and after upload on the live address: the steps are under where robots.txt goes.

Sources: RFC 9309, Robots Exclusion Protocol, Google's How to write and submit a robots.txt file and How Google interprets the robots.txt specification.

Worked examples: what the generator writes, and why

The examples below are built and tested by the same code as the tools above. The site and its paths are made up. They show how the rules behave. They are not a recommended set of rules for any site.

1. The file the form starts with

Two blocked paths, one sitemap and no AI-specific rules give a file of 6 lines. No crawler is named, so all 20 listed AI crawlers read the * group: each one is allowed at / and blocked at /cart/, by Disallow: /cart/ on line 3.

2. Search and user-triggered crawlers allowed, training crawlers blocked

The same paths with the third preset give 32 lines:

# Every crawler that is not named below
User-agent: *
Disallow: /cart/
Disallow: /account/

# AI crawlers allowed by name. They do not read the * group, so its path rules are repeated here.
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Applebot
User-agent: DuckAssistBot
User-agent: MistralAI-Index
User-agent: Amzn-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: MistralAI-User
User-agent: Amzn-User
Disallow: /cart/
Disallow: /account/

# AI crawlers blocked by name
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: MistralAI-Training
User-agent: Amazonbot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

GPTBot is blocked at /. OAI-SearchBot is allowed at / and still blocked at /cart/, because the group that names it repeats the two path rules. Without that repetition, naming OAI-SearchBot would have opened the cart and the account area to it. CCBot is not named, so it follows the general rules: allowed at /. In all, 14 of the 20 may fetch the homepage.

3. The block preset

The fourth preset blocks 18 of the 20 listed crawlers by name, in one group with Disallow: /. Googlebot and Applebot are left on the general rules and stay allowed at /. They are the crawlers behind Google Search and Apple's search features, so blocking either one is a far bigger decision than an AI setting. Each has a switch of its own if that is what you want.

4. Lines that do nothing

Block /cart/, /cart/checkout/, /account/ and allow /account/help/ and /blog/, and the generator writes all five lines and adds two warnings:

• Disallow: /cart/checkout/ does nothing: a shorter Disallow already blocks every URL it matches.
• Allow: /blog/ does nothing: no path you block covers it, so those URLs are allowed without it.

Allow: /account/help/ gets no warning, because it opens part of a blocked path. Lines that do nothing are harmless to crawlers. They are worth removing because the next person to read the file will assume they matter.

5. One file, two crawlers

The tester starts with a file in which Googlebot has a group of its own. Tested for Googlebot and then for Bingbot, which the file does not name, the same six URLs give different results:

Googlebot is named in the file, so it reads its own group (User-agent line 8) and not the * group. So none of the four rules written for * applies to it, and the search results pages the file meant to block are open to Googlebot. Bingbot is not named in the file, so the * group applies (User-agent line 2). The staging folder, blocked only in Googlebot's group, is open to Bingbot. Whoever wrote this file probably wanted both crawlers to get all five rules: the fix is to repeat the four general rules in Googlebot's group, or to move the staging rule into the * group and delete Googlebot's group.

Should a robots.txt file have Crawl-delay, Noindex or Host lines?

Crawl-delay, Noindex and Host lines are not part of RFC 9309, the robots.txt standard, so each crawler decides for itself whether to read them. Google supports none of the three. This robots.txt generator writes User-agent, Disallow, Allow and Sitemap lines only, and the tester lists other lines as found and not evaluated.

What that means in practice:

• Crawl-delay is worth adding by hand only for a crawler whose operator says it reads the line, in that crawler's own group. For Google, a slower crawl is not a robots.txt setting.
• Noindex belongs on the page, as a robots meta tag or an X-Robots-Tag header, and the page must stay crawlable so that the tag can be seen.
• Host was not found on any current operator page we read, so the generator has no reason to write it.
• Clean-param is specific to Yandex. If Yandex matters to you, its documentation covers the syntax.

Microsoft's page on Bing's crawlers could not be read for this table, because its text is added by scripts, so nothing is said here about Bing and Crawl-delay.