Decide how AI crawlers may access your site
Check which AI crawlers your robots.txt allows, make sure your content is readable without JavaScript, and understand llms.txt.
The problem
robots.txt blocks some AI crawlers, pages only show their content after JavaScript runs, or readable text is a tiny part of very heavy pages.
How to spot it
- The audit names the blocked crawlers (for example GPTBot or ClaudeBot).
- Viewing the page source shows almost no text, just an empty app container and scripts.
Why it matters
An AI service whose crawler you block cannot read your pages. Many crawlers do not run JavaScript, so content that only appears after scripts run may be invisible to them. Whether to allow AI crawlers is a business decision; this guide helps you make it deliberately.
What happens if you leave it
Blocking crawlers or depending on JavaScript can keep your content out of AI answers, but no setting guarantees that you will be cited.
How to fix it
-
Decide your policy for each crawler. Common tokens include
GPTBotandOAI-SearchBot(OpenAI),ClaudeBot(Anthropic),PerplexityBot,Google-Extended(Google's AI training control) andCCBot(Common Crawl). Check each operator's documentation, as tokens change. -
To allow a crawler, make sure no rule disallows it. To block one, add a group:
User-agent: GPTBot Disallow: / -
Make content readable without JavaScript: use server-side rendering or static generation so the main text is in the HTML response. Test by viewing the page source, not the rendered page.
-
Reduce very heavy pages: remove large inline data blobs and hidden markup.
-
Optional: publish
/llms.txt, a short Markdown summary of your site with links to key pages. It is an emerging proposal with limited support, so treat it as a bonus, not a requirement.
Common mistakes
- Blocking all bots with
User-agent: */Disallow: /while trying to block only one crawler. - Assuming that allowing a crawler means you will appear in its answers.
How to check the fix worked
Re-read your robots.txt with each crawler token in mind, then view the page source and confirm your main text is present. Re-run the audit.
References and further reading
Related tutorials
Answer customer questions clearly (FAQ content and markup)
Publish the questions customers actually ask, each followed by a short, direct answer, and mark them up so ans...
Identify your business with Organization structured data
Describe your business (name, logo, contact details, official profiles) in JSON-LD so search and AI systems ca...
Show who you are: About, contact, authors, dates and sources
Visible information about the people behind the site, how to reach them and where facts come from builds trust...