Should You Allow or Block AI Crawlers? GPTBot, robots.txt and llms.txt Explained

Many businesses have never looked at their robots.txt file, and some that have added blanket 'block all AI' rules — often copied from a template — without realising that 'AI crawler' covers several different bots doing different jobs. One trains a model. Another finds pages for a live search answer. Blocking all of them with one rule can switch off a channel a company intended to keep open.

This article sets out which crawlers do what, how to check what your own site currently allows, and the genuine, limited evidence on llms.txt.

Training crawlers and search crawlers are not the same thing

The most common misunderstanding is treating 'AI crawler' as one category. In practice, the major providers separate crawlers used to gather data for training a model from crawlers used to find and surface pages in a live search or answer feature. Blocking a training crawler is a decision about whether your content can be used to improve a model. Blocking a search crawler is a decision about whether your pages can appear in that platform's answers at all. Confusing the two is the single most common and consequential mistake.

The crawlers that matter, and what each one does

Should you block anything at all?

For most B2B companies seeking visibility and enquiries, the default of allowing all of these crawlers is the sensible starting point, because blocking them removes the possibility of appearing in the corresponding answers without any guarantee of a compensating benefit. There are legitimate reasons a business might still choose to block a training crawler specifically: concerns about proprietary pricing, technical documentation or client-identifiable material being used to train a third-party model. That is a defensible commercial decision — distinct from blocking a search crawler, which works directly against a visibility goal.

How to check what your website currently allows

1. 01Open yourdomain.com/robots.txt directly in a browser and read every 'Disallow' line against the crawler list above
2. 02Check whether a CDN, firewall or bot-protection service (e.g. Cloudflare, a WAF, or a hosting platform's bot rules) is blocking these user agents at the network level — robots.txt can look permissive while a separate service silently blocks the request
3. 03Check server or CDN access logs for the crawler names above to see whether they have actually been visiting, not just whether they are theoretically allowed
4. 04In Google Search Console and Bing Webmaster Tools, confirm key pages show as indexed, which rules out the most common underlying cause
5. 05If using a website platform or agency-managed site, ask explicitly whether any 'security' or 'bot blocking' feature has been enabled that affects these crawlers, since this is often switched on by default without discussion

Why Google can find a site but ChatGPT cannot

This is one of the most common support questions, and it usually comes down to one of three causes: OAI-SearchBot specifically blocked in robots.txt while Googlebot is allowed; a bot-protection or firewall service blocking OpenAI's user agents by default while allowing Google's (which many services treat as 'safe' by reputation); or content that only renders after client-side JavaScript runs, which Google processes (in a secondary rendering pass) more reliably than some AI crawlers are reported to. None of these is a penalty — they are configuration differences that are straightforward to check and fix.

Can AI crawlers read JavaScript-heavy websites?

Google documents that it renders JavaScript, but processes it in stages, and recommends not relying solely on client-side rendering for important content. It is widely reported — though not something every provider documents in full technical detail — that several AI-related crawlers do not reliably execute JavaScript, meaning content that only appears after a script runs may be invisible to them even though a human visitor sees it fine. The safest practice is ensuring core facts (what the company does, for whom, key services, contact details) exist in the server-rendered HTML, not only in a client-side-rendered component.

What is llms.txt, and do you need one?

llms.txt is a community-proposed file, similar in spirit to robots.txt or a sitemap, intended to give AI systems a concise summary of a site's key pages. It is not an official standard from any AI provider. As of current public information, no major AI search platform has publicly committed to reading or using it, and Google has stated it does not use it for Search. Creating one is low-cost and causes no harm, but there is no publicly documented evidence it improves AI visibility. Treat it as a minor, optional addition — not a priority ahead of crawlability, clear content and indexing fundamentals.

A simple decision framework

• Visibility is the goal: allow GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Google-Extended, Googlebot and Bingbot unless there is a specific reason not to
• Concerned specifically about model training on proprietary content: consider blocking GPTBot and Google-Extended only, leaving the search-facing crawlers (OAI-SearchBot, PerplexityBot, Googlebot, Bingbot) allowed
• Not sure what is currently configured: check robots.txt, bot-protection settings and server logs before assuming either way
• Site is JavaScript-heavy: confirm core facts exist in server-rendered HTML regardless of the robots.txt decision

Mistakes to avoid

• Copying a blanket 'block all AI bots' robots.txt template without checking which named crawlers it actually lists
• Assuming a CDN or security plugin's default settings allow these crawlers without checking
• Treating llms.txt as a substitute for being properly indexed by Google and Bing
• Reviewing robots.txt once and never again — platforms add and rename crawlers, and the right configuration is not fixed forever