Robots.txt Generator
Generate a robots.txt file for your website to control search engine crawlers.
About Robots.txt Generator
Create a valid robots.txt file to control how search engines crawl your website. Download and place in your site root directory.
About this calculator
Choose which crawlers may access your site, list the folders to block and add your sitemap URL, then copy or download a ready robots.txt.
A wrong robots.txt can hide your whole site from Google, so test it before uploading.
How to use it
- Choose allow all, selected bots or block all.
- Add disallowed paths and the Sitemap URL.
- Press Generate robots.txt and upload the file to your site root.
How robots.txt works
The file lives at /robots.txt on your domain. User-agent: names a crawler (* means all); Disallow: blocks a path; Allow: re-permits a sub-path; Sitemap: gives your sitemap URL. Example: User-agent: * / Disallow: /admin/ / Sitemap: https://example.com/sitemap.xml. robots.txt is a request, not security: it does not hide private content, and a blocked URL can still appear in results if other sites link to it. To keep a page out of Google use a noindex tag, and let Google crawl it.
Worked example
Common patterns
| Allow everything | User-agent: * Disallow: |
| Block everything | User-agent: * Disallow: / |
| Block a folder | Disallow: /private/ |
Tips and common mistakes
- Never block CSS or JS files Google needs to render pages.
- Use noindex, not robots.txt, to remove pages from results.
- Include your Sitemap line.
- Test in Google Search Console.
- Keep the file under 500 KiB.
Frequently asked questions
Where does robots.txt go?
In the root: https://example.com/robots.txt.
Does it hide private pages?
No, it is public and advisory.
How do I stop a page being indexed?
Use a noindex meta tag.
What does Disallow: / do?
Blocks the whole site for that crawler.
Can I list my sitemap?
Yes, with a Sitemap line.
Is it free?
Yes.