Build a robots.txt file one crawler group at a time. The preview updates as you type, and the checks list flags common formatting mistakes before you copy the file.
Crawler groups
Each group starts with a User-agent line, and its rules apply only to the crawler it names. Use * for every crawler that has no group of its own.
Sitemap lines
Full addresses only, one per row. Empty rows are skipped.
Generated file
Checks
The file is built only from what you enter. It is not tested against your live site or against any crawler.
How a robots.txt File Works
A robots.txt file is a plain text file placed at the root of a host, so crawlers request it from an address such as /robots.txt. Well-behaved crawlers fetch it before they request other pages on that host, then follow the groups in the file to decide what they may fetch.
Each host needs its own file. A subdomain such as blog.example.com is a separate host, so it must serve its own robots.txt at its root. The file controls crawling only. It does not by itself keep a URL out of search results.
Directives at a Glance
| Directive | What it does | Example line |
|---|---|---|
| User-agent | Names the crawler a group applies to. An asterisk matches any crawler without a more specific group. | User-agent: * |
| Disallow | Asks matching crawlers not to fetch paths that begin with the value. An empty value allows everything. | Disallow: /drafts/ |
| Allow | Carves an exception out of a broader Disallow rule. | Allow: /drafts/launch-notes/ |
| Crawl-delay | Requests a pause in seconds between fetches. Support varies by crawler. | Crawl-delay: 5 |
| Sitemap | Points crawlers to an XML sitemap. It is not tied to a group. | Sitemap: https://www.example.com/sitemap.xml |
Mistakes the Checks Catch
Frequently Asked Questions
Not by itself. Robots.txt controls crawling, so a blocked URL can still be listed if other sites link to it. To keep a page out of results, leave it crawlable and add a noindex robots meta tag or an X-Robots-Tag header.
No. The rules, sitemap lines and warnings are produced by JavaScript running in your browser. Copy and download also happen locally, and nothing you type is stored.
Google has documented that Googlebot does not read the Crawl-delay line. Some other crawlers say they honor it, so the line can still help with them, but do not rely on it to slow Google down.