Robots.txt Generator

Robots.txt Generator

Build a robots.txt file one crawler group at a time. The preview updates as you type, and the checks list flags common formatting mistakes before you copy the file.

Crawler groups

Each group starts with a User-agent line, and its rules apply only to the crawler it names. Use * for every crawler that has no group of its own.

Sitemap lines

Full addresses only, one per row. Empty rows are skipped.

Generated file


                                

Checks

    The file is built only from what you enter. It is not tested against your live site or against any crawler.

    How a robots.txt File Works

    A robots.txt file is a plain text file placed at the root of a host, so crawlers request it from an address such as /robots.txt. Well-behaved crawlers fetch it before they request other pages on that host, then follow the groups in the file to decide what they may fetch.

    Each host needs its own file. A subdomain such as blog.example.com is a separate host, so it must serve its own robots.txt at its root. The file controls crawling only. It does not by itself keep a URL out of search results.

    Directives at a Glance

    DirectiveWhat it doesExample line
    User-agentNames the crawler a group applies to. An asterisk matches any crawler without a more specific group.User-agent: *
    DisallowAsks matching crawlers not to fetch paths that begin with the value. An empty value allows everything.Disallow: /drafts/
    AllowCarves an exception out of a broader Disallow rule.Allow: /drafts/launch-notes/
    Crawl-delayRequests a pause in seconds between fetches. Support varies by crawler.Crawl-delay: 5
    SitemapPoints crawlers to an XML sitemap. It is not tied to a group.Sitemap: https://www.example.com/sitemap.xml

    Mistakes the Checks Catch

    Paths without a slashA rule such as Disallow: admin/ may not match the URLs you meant. Paths should begin with a slash.
    Missing User-agentA group with no User-agent line has no crawler to apply to, so the tool leaves it out of the file.
    Repeated groupsTwo groups for the same crawler split the rules. Merge them so each crawler reads one set.
    Malformed sitemap linesA sitemap must be a full address that starts with https:// or http://. Relative paths are skipped.

    Frequently Asked Questions

    Does robots.txt stop a page from appearing in search results?

    Not by itself. Robots.txt controls crawling, so a blocked URL can still be listed if other sites link to it. To keep a page out of results, leave it crawlable and add a noindex robots meta tag or an X-Robots-Tag header.

    Is anything sent to a server when I build the file?

    No. The rules, sitemap lines and warnings are produced by JavaScript running in your browser. Copy and download also happen locally, and nothing you type is stored.

    Why does the tool note that Google ignores Crawl-delay?

    Google has documented that Googlebot does not read the Crawl-delay line. Some other crawlers say they honor it, so the line can still help with them, but do not rely on it to slow Google down.