Free tool
robots.txt generator
Create a simple robots.txt file for your site.
Prefer to talk it through? Get a quote or contact us.
Share
Put this tool on your own site
Free to use. Paste this where you want it to appear. It runs in the visitor's browser, so nothing they enter is sent to us, and the line under it links back to this page.
How it works
How it works
Choose whether to allow everything, list any paths to keep crawlers out of, add your sitemap address and copy the file to the root of your site as robots.txt.
FAQ
About this calculator
Does robots.txt hide pages from Google?
It asks well-behaved crawlers not to crawl a path, but a blocked page can still be indexed if other sites link to it. Use a noindex tag or a password to keep a page out of results.
Where does the file go?
At the root of the site, for example example.com/robots.txt.
Can a mistake hurt my rankings?
Yes. Blocking everything on a live site stops crawlers from reading it. Test before you publish.
How do I test my robots.txt?
Open yoursite.com/robots.txt after uploading it, and use the robots.txt report in Search Console to see how Google reads it.
Does robots.txt block bad bots?
No. It is a request that well-behaved crawlers follow. It does not protect private content.
Is anything saved?
No. The math runs in your browser and nothing you enter is sent to us.
See all free tools. More free tools: Keyword density checker, Word counter and reading time, Search result preview, or see all free tools.
General information, not tax or legal advice. How we write and check our tools.