Create and check your robots.txt file

The problem

Your site has no robots.txt file at the root, or the file could not be read.

How to spot it

• Opening https://yourdomain.com/robots.txt shows a 404 page or an error.
• The audit reports "robots.txt is available" as an issue.

Why it matters

Crawlers request /robots.txt before anything else. The file is the standard place to point them to your XML sitemap and to keep them out of areas that should not be crawled, such as internal search results or admin pages. Without it, crawlers assume everything may be fetched, which is fine, but you lose the easiest way to announce your sitemap.

What happens if you leave it

A missing file is not an error in itself. A file that returns a server error (5xx) is worse: well-behaved crawlers then treat the whole site as off-limits until it is fixed.

How to fix it

1. Create a plain-text file named robots.txt (lowercase).

2. Start with a permissive file and add only the rules you need:

   User-agent: *
   Disallow: /wp-admin/
   Allow: /wp-admin/admin-ajax.php
   
   Sitemap: https://www.example.com/sitemap.xml
   

   Replace the paths and the sitemap URL with your own. If you have nothing to block, keep just User-agent: * and the Sitemap: line.

3. Upload it to the root of your domain so it is served at https://yourdomain.com/robots.txt. A robots.txt in a subfolder is ignored.

4. Make sure the server returns HTTP 200 with Content-Type: text/plain.

5. If your site exists on several hosts (for example shop.example.com), each host needs its own file.

Platform-specific notes

• WordPress: WordPress serves a virtual robots.txt. SEO plugins let you edit it; a physical robots.txt file in the web root takes precedence.
• Shopify: edit the robots.txt.liquid theme template rather than uploading a file.
• Static hosting / CDN: place the file in the published root folder (often public/ or dist/).

Common mistakes

• Disallow: / left over from a staging site. This blocks your whole site.
• Blocking CSS or JavaScript folders that pages need to render.
• Using robots.txt to hide private pages. It is public, and it is a request, not access control. Use authentication or noindex instead.
• Expecting Disallow to remove pages from search results. Blocked pages can still be indexed from links; use noindex to keep a page out.

How to check the fix worked

1. Open https://yourdomain.com/robots.txt in a private browser window. You should see the plain-text rules.
2. Check that your homepage and key pages are not disallowed for *.
3. Run a new WebGrowthEngine scan; the robots.txt check should pass and the sitemap should be found.