What robots.txt is and what it controls
A robots.txt file is a plain text file placed at the root of a site, for example at /robots.txt. It follows the Robots Exclusion Protocol and asks crawlers which paths they should or should not request. Each group starts with a User-agent line naming a crawler (an asterisk means all of them), followed by Disallow and Allow lines that list URL path prefixes. A Sitemap line can point crawlers to your sitemap.
It is important to know what it does not do. The file is a request that well-behaved crawlers follow, not a security feature. It does not hide anything: the file itself is public, and a disallowed URL can still be visited by anyone who knows it. It also does not remove a page from search results. A blocked page can still be indexed from links alone, just without its content. To keep a page out of results, use a noindex directive on a page that crawlers are allowed to fetch, or put it behind a login.
When you need one
Most small sites can work with a very short file. You typically want one to keep crawlers away from areas that waste their time: internal search result pages, admin or checkout paths, staging folders, or endless filtered URLs. It also gives you a standard place to declare where your sitemap lives.
It is also worth having when a bad rule is already hurting you. A single Disallow: / under User-agent: * tells every compliant crawler to stay out of the whole site, which is a common accident after a staging site goes live.
How to use the Robots.txt Generator
- In the first rule box, set the User-agent. Use
*for all crawlers, or a specific crawler name. - In Disallow, list paths to block, separated by commas, such as
/admin, /cart. - In Allow, list any exceptions inside a blocked area, also comma-separated. Leave it empty if you have none.
- Click Add rule for another crawler group. Use Remove rule to delete one when more than one exists.
- Enter the full Sitemap URL, or clear the field to leave the line out.
- Review the robots.txt box and click Copy. Save the text as a file named robots.txt and upload it to your site's root.
You can build the sitemap itself with the Sitemap Generator.
Mistakes and limits to watch for
The default rule disallows /admin, which is only an example. Change it to match your own site before you publish. If a rule has no User-agent text, the tool writes *.
Each comma-separated path becomes its own line. The tool does not validate the paths or check that they start with a slash, so check them yourself. A rule with no Disallow or Allow entries produces only a User-agent line, and the tool does not add comments, crawl-delay or wildcard helpers on its own.
Test your finished file with the crawler tools provided by search engines before relying on it, and recheck it after any site redesign. Never list private URLs in it as a way to hide them, since the file is readable by everyone. The text is created in your browser, and nothing is uploaded.