Robots.txt Generator
Generate a clean robots.txt with a user-agent, allow or disallow paths and an optional sitemap line. Validate paths and download the file instantly.
What is the Robots.txt Generator?
A robots.txt file sits at the root of your website and tells search engine crawlers which sections they should not crawl. A single typo in it can hide important pages or leave admin and search pages open to bots, so it pays to generate the file carefully. This Robots.txt Generator writes a correctly formatted rule group from a few simple fields and lets you download it as a ready-made robots.txt file.
You choose one user-agent, either * for all crawlers or a single bot name such as Googlebot, decide whether your listed paths are allowed or disallowed, and optionally add your sitemap address. Every path is checked before the file is written, so values with spaces, missing slashes or stray characters are caught early. The generator works locally in your browser and does not inspect your live site.
How to use the Robots.txt Generator
- Enter * in User-agent token to address every crawler, or a single bot name such as Googlebot.
- Choose Disallow or Allow under Rule for listed paths; the choice applies to every path you list.
- Type one path per line in Paths, one per line, each starting with a slash, for example /wp-admin/ or /search/. Leave it empty to allow everything.
- Optionally paste your full sitemap address into Sitemap URL.
- Click Process to preview the file, then use Copy result or Download to save robots.txt.
- Upload the file to the root of your domain so it opens at yourdomain.com/robots.txt.
Features
- Writes a User-agent line followed by one Allow or Disallow line per path, in standard robots.txt syntax.
- Produces an allow-all file (User-agent: * with an empty Disallow:) when the path list is left blank.
- Adds a Sitemap: line when you supply an absolute HTTP(S) sitemap URL.
- Validates input inline: paths must start with / and contain no spaces, # or control characters, and the user-agent must be * or one crawler token.
- Offers a Download button that saves the output as a file named robots.txt, plus Copy result.
Why use this robots.txt generator
- Keep crawlers away from admin, cart, search and other low-value pages so crawling focuses on your content.
- Point search engines to your sitemap from the file they read first.
- Avoid syntax mistakes that could accidentally block your whole site.
- Get an uploadable file in seconds without editing text by hand.
How it works
Write User-agent plus one directive per path; an empty path list emits Disallow: to impose no restriction; optionally append Sitemap.
Examples
Block private and search pages for all crawlers
Input: User-agent token: * | Rule: Disallow | Paths: /private/ and /search/ | Sitemap: https://example.com/sitemap.xml
Result: User-agent: * Disallow: /private/ Disallow: /search/ Sitemap: https://example.com/sitemap.xml One crawler group. Install at the site root yourself. Robots rules are public crawl guidance, not access control or guaranteed deindexing; no live inspection is performed.
WordPress rules for Googlebot only
Input: User-agent token: Googlebot | Rule: Disallow | Paths: /wp-admin/ and /cart/ | Sitemap: empty
Result: User-agent: Googlebot Disallow: /wp-admin/ Disallow: /cart/ One crawler group. Install at the site root yourself. Robots rules are public crawl guidance, not access control or guaranteed deindexing; no live inspection is performed.
Allow everything
Input: User-agent token: * | Rule: Allow | Paths: empty | Sitemap: empty
Result: User-agent: * Disallow: One crawler group. Install at the site root yourself. Robots rules are public crawl guidance, not access control or guaranteed deindexing; no live inspection is performed.
Results are estimates for general information, not financial, medical, legal, tax, or property advice. Confirm important figures with the relevant institution or a qualified professional.
Robots.txt Generator: frequently asked questions
Can I create rules for several bots or mix Allow and Disallow?
Not in one pass. The generator creates a single crawler group: one user-agent and one rule type applied to every listed path. For a more complex file, generate each group separately and join them in a text editor, leaving a blank line between groups. Most small and medium sites only need one group for all crawlers, which is exactly what the default produces.
Why is my path rejected?
Each line must begin with a slash and must not contain spaces, the # character or invisible control characters, because these either break robots.txt syntax or start a comment. Write /my-page/ rather than my-page, and percent-encode special characters, for example %20 for a space. Empty lines are simply ignored, so spacing between entries is fine.
What does an empty Disallow line mean?
Disallow: with nothing after it means no restrictions at all, so compliant crawlers may visit every page. The tool writes it automatically when you leave the path list empty. It is the standard way to publish a valid robots.txt that allows everything while still giving you a place to add a sitemap line.
Will robots.txt remove pages from Google or protect private content?
No. The file is public, voluntary guidance that well-behaved crawlers follow; it is not a password or security measure, and blocked URLs can still appear in search results if other sites link to them. To keep a page out of search, use a noindex tag or remove it, and protect private areas with a login.
Where do I put the downloaded robots.txt file?
Upload it to the root folder of your domain so it opens at https://yourdomain.com/robots.txt. A file inside a subfolder is ignored by crawlers. Each subdomain needs its own copy. On WordPress, check whether an SEO plugin already serves a virtual robots.txt, since a physical file in the root normally takes priority.
Does the generator check my website?
No. It builds the file only from what you type and does not visit or test your live site. After uploading, open your robots.txt in a browser and use the robots.txt report in Google Search Console to confirm Google reads it as expected.