BrowserTools

robots.txt Generator

Correct crawler rules for any site

By Guga — BrowserTools maintainer, Tbilisi, Georgia Updated 2026-09-11
Free Private No uploads
What is robots.txt Generator?

A sitemap field, a crawl-delay box, a block-everything checkbox and a repeatable list of rule blocks. Each block carries a user-agent name and two text areas — one for disallowed paths, one for allowed — taking a path per line, so /admin and /private entered in the same field become two Disallow directives under one agent. Blocks are added and removed with the buttons beside them, and the file is rebuilt as you type rather than on a submit. Ticking Block the entire site collapses the rule list to a single wildcard agent with a root disallow and hides the per-crawler editor. Crawl-delay, when set, is written into every block rather than only the first. The file starts with two blocks — a wildcard excluding /admin and /private, and one for GPTBot — so there is something to edit instead of a blank page.

Build a valid robots.txt with allow/disallow rules per crawler, crawl-delay and sitemap reference. Fill the form, copy the file, drop it in your site root — done.

Private robots.txt Generator In Your Browser

robots.txt Generator is processed locally with client-side JavaScript. Your files never touch a server, and there is no queue to wait in.

Step-by-Step

How to use robots.txt Generator

1

Enter the absolute URL of your sitemap in the Sitemap URL field; it becomes a Sitemap line at the end of the file.

2

Set Crawl-delay in seconds if you want one, then add or remove rule blocks with the Add rule and per-block remove buttons.

3

Fill in each block's user-agent name and type paths one per line into the Disallow and Allow text areas.

4

Copy the robots.txt panel and save the contents as robots.txt in the root directory of the site.

Key Capabilities
  • The Disallow and Allow fields take one path per line and emit a separate directive for each, all under the same user-agent block.
  • Block the entire site replaces the rule list with a single wildcard agent and Disallow: /, which is the correct way to close a site off.
  • Crawl-delay is written into every rule block when it is set, because that is where the directive has to sit to take effect.
  • The Sitemap line is appended last however many blocks exist, and runs of three or more blank lines are collapsed before output.
Common Use Cases
  • A site owner keeps an internal admin area out of crawler reach while leaving the public pages fetchable.
  • An operator blocks a training crawler in its own block, leaving the wildcard rules for everyone else untouched.
  • A developer closing a preview deployment uses Block the entire site to get a single wildcard rule with nothing else to review.
Clear Answers

Frequently Asked Questions

How is a matching rule decided when several could apply?
The most specific user-agent that matches the crawler wins, and inside that block the longest matching path settles it — so an Allow for /admin/public beats a Disallow for /admin. One block per agent is much easier to reason about than a single large block serving everybody.
Does blocking a page stop it appearing in results?
No. A crawler that already knows the URL can still list it as a bare link with no description, because taking a page out of stored results is a separate mechanism. Disallowing controls fetching, not what is already recorded.
Why did my crawl-delay make no difference?
The directive is a convention that several crawlers, including the largest, simply ignore. It is still worth setting for those that honour it, but it is not a way to protect a server from load; rate limiting does that job properly.
Is a missing file treated as allow everything?
Yes, and the rules are a request rather than a lock. Anything genuinely private needs real access control — authentication, network rules or an unguessable path. A crawler that chooses to ignore the file will do so whatever the file says.