Robots.txt Generator
FREE TOOL · NO SIGN-UP
This robots.txt generator builds a ready-to-upload robots.txt file from a few simple choices. Pick a default rule for all robots, list the paths to disallow or allow, add an optional crawl-delay and your sitemap URL, and tick any AI crawlers or SEO bots you want to block. The file updates as you go, with warnings for risky settings. Copy it or download it as robots.txt, then upload it to the root of your domain.
What a Robots.txt File Does
A robots.txt file is a plain text file at the root of a website, for example https://example.com/robots.txt. It follows the Robots Exclusion Protocol, a standard that tells web crawlers which parts of a site they may request.
Search engines like Google and Bing fetch this file before they crawl. If a rule disallows a path, well-behaved crawlers skip it. That lets you keep bots out of admin pages, internal search results, cart pages and endless filter URLs that waste crawler access without adding anything useful to search.
Three points are often misunderstood:
- Robots.txt controls crawling, not indexing. A blocked URL can still appear in search results if other pages link to it. Google just can’t see its content.
- It is a request, not a lock. Reputable crawlers follow it. Scrapers and bad bots may ignore it.
- It is public. Anyone can read your robots.txt, so never use it to hide sensitive paths.
Every host needs its own file. blog.example.com and example.com are separate, and so are the HTTP and HTTPS versions.
Robots.txt Syntax and Directives Explained
A robots.txt file is made of groups. Each group starts with one or more User-agent lines naming the crawler, followed by rules for that crawler. The main directives are:
User-agent:which crawler the group applies toDisallow:a path the crawler should not requestAllow:an exception inside a disallowed pathSitemap:the full URL of your XML sitemapCrawl-delay:seconds to wait between requests (not supported by Google)
Lines starting with # are comments. Paths are case sensitive, so /Admin/ and /admin/ are different.
User-agent groups and crawler matching
User-agent: * is the catch-all group for every crawler. You can add named groups, such as User-agent: GPTBot, to give one bot different rules.
Crawlers pick the most specific group that matches their name and follow only that group. If Googlebot finds a group for Googlebot, it ignores the * group completely. That catches people out. Rules you put in the * group do not carry over to a named group, so a named group needs its own full set of rules.
This generator writes one User-agent: * group with your main rules. Any bots you choose to block are listed together in a second group that ends with Disallow: /, so those crawlers are kept out of the whole site.
Disallow, allow and wildcard path rules
Disallow: /wp-admin/ blocks every URL whose path starts with /wp-admin/. Allow: /wp-admin/admin-ajax.php creates an exception, so that one file stays reachable. When an allow and a disallow rule both match, Google follows the most specific (longest) rule.
An empty Disallow: line means nothing is blocked. Disallow: / blocks the entire site.
Google and Bing also support two wildcards:
*matches any sequence of characters, soDisallow: /*?sort=blocks any URL with a sort parameter.$marks the end of a URL, soDisallow: /*.pdf$blocks PDF files.
The generator’s default list already includes /wp-admin/, /?s= and /search/, which cover the WordPress admin area and internal search results.
Adding the sitemap URL and crawl-delay
The Sitemap: line helps crawlers find your XML sitemap. It must be a full absolute URL starting with https://, and the tool warns you if it isn’t. You can list more than one sitemap.
Crawl-delay asks a bot to wait a number of seconds between requests. Bing and Yandex respect it. Googlebot ignores it, and the tool shows a note when you set one. Use it only if a crawler is genuinely straining your server.
How to Create Robots.txt With the Generator
- Choose the default rule. “Allow everything” or “Custom rules below” both apply your allow and disallow lists to all robots. “Block everything” writes
Disallow: /, which is only for staging sites. The tool shows a red warning when you pick it. - Edit the disallow paths. Enter one path per line. The defaults suit most WordPress sites. Add paths such as
/cart/,/checkout/or/*?filter=if they apply to you. - Edit the allow paths. Add exceptions inside blocked folders, one per line. The default keeps
admin-ajax.phpopen, which many WordPress themes and plugins need. - Add crawl-delay if needed. Leave it blank unless a specific bot is overloading your server.
- Enter your sitemap URL. For example
https://yourdomain.com/sitemap.xml. - Block AI crawlers or SEO bots (optional). Tick individual bots, or use “Select all AI bots”. A second list covers other bots like AhrefsBot, SemrushBot, MJ12bot, DotBot, PetalBot, Baiduspider and YandexBot.
- Copy or download. The file includes a dated comment header. Use Copy, or Download to save it as
robots.txt.
The file is built live as you change settings, so you can see the effect of each rule before you use it. The generator runs 100% in your browser, and nothing you enter is uploaded.
Where to Upload the File and How to Test It
Robots.txt must sit in the root directory of the host it applies to: https://yourdomain.com/robots.txt. Crawlers do not look for it in subfolders, so /blog/robots.txt has no effect.
How you upload depends on your setup:
- Static or custom sites: upload the file through FTP, SFTP or your host’s file manager to the web root (often
public_html). - WordPress: WordPress serves a virtual robots.txt if no file exists. Uploading a real file replaces it, or you can paste the rules into the robots.txt editor in an SEO plugin like Rank Math.
- Blogger: paste the rules into Settings under “Custom robots.txt”.
After uploading, open the URL in a browser to confirm it loads as plain text. In Google Search Console, the robots.txt report shows which version Google fetched and flags any parsing problems. You can also use the URL Inspection tool to check whether a specific page is blocked.
Crawl Budget and Search Engine Indexing
Crawl budget is roughly how many URLs a search engine is willing and able to crawl on your site in a given period. For most small and medium sites it isn’t a real constraint; Google crawls them fully without trouble.
It starts to matter on large sites, online stores and sites that generate many URL variations. Faceted navigation, sort orders, session IDs and internal search pages can create thousands of near-duplicate URLs. When crawlers spend time on those, important pages may be crawled less often.
Robots.txt is the main tool for keeping crawlers away from that low-value space. Block the patterns, not individual pages, for example /*?sort= or /search/.
Keep in mind that blocking a URL stops crawling, not indexing. If the URL is already indexed or heavily linked, it may stay in results with no description. To remove a page from the index, leave it crawlable and use a noindex meta robots tag instead.
Blocking AI Crawlers Like GPTBot
Many AI companies run their own crawlers to collect content for training models or for AI-powered search answers. If you don’t want your content used this way, you can block them in robots.txt.
The generator includes common AI user-agents: GPTBot, ChatGPT-User and OAI-SearchBot (OpenAI), Google-Extended, CCBot (Common Crawl), anthropic-ai, ClaudeBot and Claude-Web, PerplexityBot, Bytespider, Applebot-Extended, Meta-ExternalAgent, Amazonbot, cohere-ai, Diffbot and Omgili.
Think about the trade-off. Blocking search-related AI bots such as OAI-SearchBot or PerplexityBot can stop your pages from being cited in those tools’ answers. Google-Extended controls use of your content for Google’s Gemini models, and blocking it does not affect Google Search rankings. As with any robots.txt rule, compliance is voluntary.
Robots.txt Mistakes That Hurt SEO
- Leaving a staging block live.
Disallow: /copied from a staging site to production stops crawling of the whole site. Check after every launch. - Blocking CSS and JavaScript. Google renders pages like a browser. Blocking theme or plugin assets can stop it from seeing your layout and content properly.
- Using robots.txt to deindex pages. Blocked pages can’t be crawled, so Google never sees a
noindextag on them. Google stopped supportingnoindexinside robots.txt in 2019. - Wrong case or missing slash.
/Blogdoesn’t block/blog/, and/privatealso blocks/private-offers/. - Forgetting named groups ignore
*rules. Adding a Googlebot group without repeating your main rules opens up everything for Googlebot. - Relative sitemap URLs. The Sitemap line must be a full URL.
- Placing the file in a subfolder. Only the root location counts.
- Listing secret paths. The file is public, so it can point attackers straight to admin or backup folders.
When Not to Rely on Robots.txt
Robots.txt is the wrong tool when you need to:
- Remove a page from search results. Use a
noindextag, or the Removals tool in Search Console for urgent cases. - Protect private content. Use password protection or server-side access rules. Robots.txt does nothing against people or bad bots.
- Fix duplicate content. Use canonical tags or redirects so search engines can see and consolidate the pages.
- Hide a staging site reliably. Password protection is safer than a disallow rule.
Keeping Crawl Rules Simple and Reviewed
Short robots.txt files are easier to trust. Block patterns that waste crawls, keep assets open, and review the file after every redesign or migration. Build the sitemap it points to with the XML Sitemap Generator, handle moved pages with the .htaccess Redirect Generator, set page-level indexing with the Meta Tag Generator, and find more on the FHSEO Hub free SEO tools page.