Robots.txt generator
Tell search engines and other crawlers which parts of your site they may visit.
Runs entirely in your browser. Nothing you type is sent anywhere.
Crawler names change. Check each company’s own documentation before relying on this list.
Did this tool do the job?
Tell us privately. Ratings and messages come only to the Gauritanaya team and are never shown on the site.
Please don’t include personal details such as Aadhaar, PAN or account numbers. We use this only to fix and improve the site. How feedback is handled
How to use the robots.txt generator
robots.txt asks crawlers which paths not to visit. It doesn’t hide pages: a blocked page can still appear in search results if other sites link to it, so use a noindex tag for that.
- List the folders crawlers shouldn’t visit, such as admin or checkout pages.
- Add your sitemap address.
- Tick any AI crawlers you want to opt out of.
- Upload the file to the root of your site, so it opens at yourdomain.com/robots.txt.
Questions
Does robots.txt hide a page from Google?
No. It asks crawlers not to visit, but a blocked page can still appear in results if other sites link to it. Use a noindex tag to keep a page out of search results.
Do all crawlers obey robots.txt?
Reputable ones do. It’s a request, not a lock, so never rely on it to protect private information.
Is anything I type sent to a server?
No. Everything is generated in your browser.