Control indexing with noindex, and avoid excluding important pages

The problem

Some pages carry noindex (or none), or restrict snippets with nosnippet or max-snippet:0, possibly by mistake.

How to spot it

• A page you expect in search never appears.
• The audit lists pages with noindex in a meta tag or HTTP header.

Why it matters

noindex removes a page from search results entirely. That is right for thank-you pages, internal search results and private areas, but a costly mistake on product or service pages. Snippet restrictions stop search and AI features from quoting the page.

What happens if you leave it

An accidentally excluded page gets no search traffic at all, however good it is.

How to fix it

1. Review every URL the audit lists under "Pages excluded from search".
2. For pages that should appear in search, remove the directive from:
   • the HTML: <meta name="robots" content="noindex">
   • the HTTP header: X-Robots-Tag: noindex (set by the server, CDN or plugin)
3. Keep noindex on pages that should stay out (login, cart, internal search, duplicates you cannot canonicalise).
4. Remove noindex pages from your XML sitemap.
5. Only use nosnippet or max-snippet where you deliberately want to limit quoting.

Platform-specific notes

• WordPress: check Settings → Reading → "Discourage search engines" (adds noindex site-wide), and per-page settings in your SEO plugin.
• Staging copies: make sure the live site did not inherit a staging site's noindex header.

Common mistakes

• Blocking a page in robots.txt and adding noindex: crawlers then cannot see the noindex at all.
• Forgetting HTTP headers set at the server or CDN level.

How to check the fix worked

Check the page source for a robots meta tag and the response headers (browser network tab or curl -I) for X-Robots-Tag. Use URL Inspection in Search Console to request indexing once fixed.