Robots.txt Generator - Create Crawl Rules Online
Generate a clean, valid robots.txt file for your website in seconds with our free online Robots.txt Generator. Define which search-engine crawlers are allowed or blocked from specific paths, add a sitemap reference, and set a crawl delay - all produced instantly in your browser with no signup and no uploads.
The robots.txt file lives at the root of your domain (for example https://example.com/robots.txt ) and tells crawlers like Googlebot which parts of your site they may access. Our generator groups rules by user-agent automatically and outputs ready-to-publish text.
Key Features
- Live Output: Your
robots.txtregenerates instantly as you type or edit rules - no Generate button needed. - Allow / Disallow Rules: Add unlimited rules and choose whether each path is allowed or blocked.
- User-Agent Targeting: Use
*for all crawlers, or restrict rules to a specific bot likeGooglebot. Rules are grouped automatically. - Sitemap Reference: Append a
Sitemap:directive so crawlers discover your XML sitemap. - Crawl Delay: Optionally set a
Crawl-delayfor all robots to slow down aggressive crawling. - Copy & Download: Copy the result or download it directly as
robots.txt. - 100% Private: Everything runs in your browser - nothing is sent to a server.
How to Use the Robots.txt Generator
- Add an Allow or Disallow rule for each path you want to control, using a leading slash (for example
/private/). - Set the User-agent to
*for all crawlers, or a specific bot name for targeted rules. - Optionally enter your sitemap URL and a crawl delay value.
- Click "Copy" to copy the file, or "Download" to save it as
robots.txt. - Upload the file to the root directory of your website so it is reachable at
/robots.txt.
Frequently Asked Questions
1What is a robots.txt file?
A robots.txt file is a plain-text file placed at the root of your website (for example https://example.com/robots.txt) that tells search-engine crawlers which pages or directories they are allowed or disallowed to crawl.
2Where do I upload the generated robots.txt file?
Upload the file to the root directory of your website so that it is reachable at https://yourdomain.com/robots.txt. Crawlers fetch it from that exact location.
3What does User-agent: * mean?
An asterisk (*) matches all crawlers. Rules listed under User-agent: * apply to every search bot. To target a specific crawler, use its name, such as Googlebot or Bingbot.
4What is the difference between Allow and Disallow?
Allow permits a crawler to access a path, while Disallow blocks it. Disallow: / blocks the entire site, Disallow: /private/ blocks only the private directory, and an empty Disallow allows everything.
5What does Crawl-delay do?
Crawl-delay asks compliant crawlers to wait a number of seconds between requests to reduce server load. Note that major search engines like Google generally ignore this directive.
6Are my rules sent to a server?
No. All generation happens entirely in your browser. Your robots.txt content never leaves your computer, so it remains completely private.