1. Enter your details
Fill in the fields relevant to your task. Required information is checked before processing.
Back
Compose crawler groups, allow and disallow rules, crawl delay and sitemap directives.
Enter your details
Use your own page data or pasted content. Processing stays in your browser.
User-agent groups
Group 1
robots.txt gives crawler guidance; it is not access control and does not guarantee de-indexing. Crawl-delay support varies by crawler.
robots.txt is a public crawler instruction file, not an access-control mechanism. Disallow rules do not guarantee search de-indexing.
All processing happens locally in this browser. No page is crawled and no information is sent to an external API.
What this tool does
Create a robots.txt draft with multiple user-agent groups, allow and disallow paths, optional crawl-delay lines and a sitemap location. Review the public file before publishing it at the site root.
2. Generate the result
Select Generate or Process. Any validation issue appears above the action controls.
3. Copy or download
Review the output, copy it to your clipboard or download it as a local file.
Example
Add a wildcard crawler group, disallow /admin/, allow /, then provide an absolute sitemap URL. Copy the resulting plain text into your robots.txt file.
Frequently asked questions
How should I use this result?
No. robots.txt is a public crawler instruction file, not authentication or access control. Disallowed URLs may still be known or indexed through other signals.
Does this tool send a message or API request?
No message or API request is sent automatically. Any external action requires an explicit user click.