Meta Robots Tag Generator: Control Indexing with the Right Directives

Directives
Meta tag (HTML <head>)
X-Robots-Tag HTTP header
Apache .htaccess (for PDFs etc.)
Nginx

		

	
















FREE TOOL · NO SIGN-UP





This meta robots tag generator builds the exact <meta name="robots"> tag and matching X-Robots-Tag header for the directives you choose, such as noindex, nofollow, nosnippet or max-image-preview. It is for SEOs, developers and site owners who need to control what search engines index and show, page by page or file by file. Tick your settings and copy the meta tag, the HTTP header, or ready-made Apache and Nginx snippets.





A plain-language summary under the form tells you whether the page can appear in search results with your current choices, so mistakes like an accidental noindex stand out.













What the Meta Robots Tag Controls





The meta robots tag sits in a page’s <head> and gives search engine crawlers instructions about that page. When no tag is present, the default is index, follow.





Indexing and link directives





noindex tells search engines not to show the page in results. nofollow asks them not to follow links on the page. none is shorthand for both. noimageindex keeps images on the page out of image results. indexifembedded lets content embedded through an iframe be indexed with the parent page, even if the embedded URL has noindex. notranslate stops Google offering a translation link in results. You can target all crawlers with name="robots" or a single crawler, such as googlebot or bingbot, and the generator writes the correct name for each.





Snippet and preview directives





These control how much of your content appears in results. nosnippet blocks text snippets and video previews. max-snippet:50 limits snippets to 50 characters, and -1 means no limit. max-image-preview sets the largest image preview as none, standard or large. max-video-preview limits video previews in seconds. noarchive asks engines not to show a cached copy. unavailable_after tells Google to stop showing the page after a given date, which suits event pages and limited offers.













How to Generate a Meta Robots Tag





Every setting updates the code instantly, so you can experiment safely.





Step 1: Pick the bot





Choose who the rule applies to: all search engines, Googlebot only, Bingbot only or Googlebot-News. Most pages should use all search engines. Pick a single crawler only when you want different behavior, such as keeping a page out of Google News while leaving it in normal search.





Step 2: Choose index and follow settings





Tick noindex to keep the page out of results, nofollow to stop links being followed, or none for both. Leave them unticked to allow indexing. If you tick nothing at all, the tool outputs index, follow, which matches the default and is safe to leave out of your page entirely.





Step 3: Set snippet limits





Optionally set max-snippet in characters, max-image-preview and max-video-preview in seconds. Use -1 for no limit. Add an unavailable_after date for pages that should drop out of results later. Tick nosnippet or noarchive if needed.





Step 4: Copy the tag or header





Copy the meta tag for HTML pages. Copy the X-Robots-Tag header for server configuration, or use the ready Apache and Nginx snippets, which apply the directives to PDF and Word files. Paste the meta tag into your page’s <head>, or set it with your SEO plugin.













Meta Robots vs X-Robots-Tag





Both carry the same directives. The difference is where they live.





Header for PDFs and images





The meta tag only works in HTML. PDFs, images, Word documents and other files have no <head> to hold it. The X-Robots-Tag HTTP header carries the same instructions in the server response, so it works for any file type. It is also useful for applying rules across many pages at once from the server, without editing each page template.





Server config examples





The generator gives you two examples. On Apache, a <FilesMatch> block in .htaccess adds Header set X-Robots-Tag to every PDF, DOC and DOCX file. On Nginx, a location block with add_header X-Robots-Tag does the same. Edit the file extensions to match your needs. After deploying, check the response headers with your browser’s DevTools Network tab or curl -I to confirm the header is present.













Meta Robots vs robots.txt





These two are often confused, and using the wrong one can keep pages in search results you meant to hide.





Why blocked pages can’t be noindexed





robots.txt controls crawling. A meta robots tag controls indexing. If robots.txt blocks a URL, crawlers never fetch the page, so they never see its noindex tag. Google can still index the bare URL if other pages link to it, showing it with no description. To remove a page from results, allow crawling and add noindex. Block it in robots.txt only after it has dropped out, if at all.





Choosing the right tool





Use noindex for pages that should exist but not appear in search, such as thank-you pages, internal search results and thin tag archives. Use robots.txt to manage crawling of large sections that waste crawl resources, like faceted filters. Build those rules with the Robots.txt Generator.













Snippet Controls and AI Search





Snippet directives also matter for AI-generated answers in search.





max-snippet and nosnippet





Google’s documentation says its snippet controls, including nosnippet and max-snippet, also apply to AI features in Search such as AI Overviews. Content blocked from snippets will not be used as a direct input there. Using them can reduce how your pages are shown, so weigh visibility against control.





data-nosnippet





To hide only part of a page, wrap that text in an element with the data-nosnippet attribute, for example <span data-nosnippet>. The rest of the page remains eligible for snippets.













Robots Directive Mistakes





These mistakes cause the most painful indexing problems.





noindex left on after launch





Staging sites often use noindex, and it gets copied to production. Check key templates after every launch.





Conflicting tags





If a page has two robots tags, Google follows the most restrictive. Remove duplicate tags from themes and plugins.





noindex on canonical pages





Do not noindex a page that other pages name as their canonical. It sends mixed signals and may drop the whole set.













Decide What Search Engines Show





Choose your directives above and copy the exact tag or header you need in seconds. Manage crawling with the Robots.txt Generator, build the rest of your head tags with the Meta Tag Generator, and preview results with the SERP Snippet Preview tool. For server-level rules, see the Nginx Redirect Generator. Find more on FHSEO Hub.













Meta Robots Tag Generator: Control Indexing with the Right Directives FAQs









A meta robots tag is an HTML element in a page’s head that tells search engine crawlers how to treat the page, such as whether to index it, follow its links or show snippets. It looks like `<meta name=”robots” content=”noindex, follow”>`. Without a tag, the default is index, follow.









Noindex tells search engines not to show a page in results, while robots.txt disallow tells them not to crawl it. A disallowed page can still appear in results as a bare URL, because crawlers never see its noindex tag. To remove a page from search, allow crawling and use noindex.









Use noindex, follow on pages that should not appear in search but contain links worth discovering, such as paginated archives, internal search results or tag pages. The page stays out of results while crawlers can still reach linked pages. Over long periods, Google may crawl noindexed pages less often.









Noindex a PDF by sending an X-Robots-Tag: noindex HTTP header with the file, because PDFs cannot contain a meta tag. On Apache, add a FilesMatch block with Header set in .htaccess. On Nginx, use a location block with add_header. This generator outputs both snippets ready to paste.









Google says nosnippet and related snippet controls also apply to its AI features in Search, so content blocked with nosnippet should not be used as a direct input for AI Overviews. It also removes normal text snippets from results, which can lower clicks. Use data-nosnippet to protect only specific sections.