Engineering6 min read

Is your site blocking ChatGPT? Bot protection and AI crawlers

A site can load perfectly in a browser, rank in Google and still return 403 to every AI assistant. How to check in two minutes, and where the block is switched on.

One of our own sites looked perfect. It loaded fast, it was indexed, it ranked for its name. Then a routine scripted check requested the home page and got a 403.

So did every other request that did not come from a real browser. The home page, the articles, even robots.txt. We do not know for how long, because nothing we normally look at had shown it. Anyone who pasted the address into an AI assistant during that time got an error instead of an answer.

This is how a site ends up invisible to assistants while looking healthy, how to check yours in two minutes, and how to decide on purpose what gets in.

Why nothing looked wrong

The host had a protection mode switched on that puts a JavaScript challenge in front of every visitor. A browser solves it in the background and shows the page. A person testing the site sees nothing unusual, because a person tests with a browser.

Search kept working too. The mode lets known bots through, search engines among them, so Google went on crawling and the usual dashboards stayed green.

What did not get through was everything else that reads the web without running JavaScript: plain HTTP clients, monitoring scripts, link preview generators, and the fetch tools assistants use when a user hands them a URL. All of them received a 403 and a header saying why:

HTTP/2 403
x-vercel-mitigated: challenge

We confirmed it from two separate networks, on four different URLs, with a second site on the same host as a working control. From outside we could not tell which named crawlers were on the allow list and which were not. That is the uncomfortable part: you do not know until you test.

The cost is not abstract. A prospect who asks an assistant to compare three agencies gets two summaries and one error. The assistant does not say "this company blocked me". It moves on to the ones it could read.

Check yours in two minutes

Start with the file a well-behaved crawler requests before anything else:

curl -sI https://example.com/robots.txt

Then a real page:

curl -sI https://example.com/

You are looking at the first line and at a few headers.

What you seeWhat it means
200 and your content typeA plain client can read the page
403 with x-vercel-mitigated: challengeA Vercel challenge is in front of the site
403 with cf-mitigated: challengeA Cloudflare challenge
429Rate limiting, often tuned too tightly
200 but the body is a "checking your browser" pageA challenge served with a success code, the hardest one to spot

For the last row, look at the body rather than the status:

curl -s https://example.com/ | head -n 30

If that is not your page, it is not what a crawler sees either.

Two honest limits. A request from your laptop is not the real crawler: the large ones are verified by IP address, so a rule can treat them differently from you. A pass on this test is therefore not proof. A fail is still a clear signal, because the tools that fetch a page on a user's behalf look much more like your curl than like a browser.

The second check costs nothing. Give an assistant with browsing the address of one of your inner pages and ask it to quote the second paragraph. If it paraphrases the home page from memory, or apologises, you have your answer.

Where it gets switched on

Almost never by a decision. Usually by a default or by a bad afternoon.

An attack mode on the host. Meant for the hours of a DDoS attack. Somebody turns it on during a traffic spike, the spike passes, and nobody turns it off.

A default on the CDN. In July 2025 Cloudflare began blocking AI crawlers by default for new domains. If your domain was added after that and nobody changed the setting, the choice was made for you.

A bot-fight toggle. One switch in a dashboard that challenges anything automated, with no list of what "anything" includes.

A security plugin or a firewall rule written by a previous developer to stop a scraper, matching far more than that scraper.

robots.txt. A blanket Disallow copied from a staging site, or a list of blocked agents pasted from a blog post in 2023.

Decide on purpose

Blocking is a legitimate choice. The mistake is not knowing you made it.

The agents are not all the same thing, and the useful distinction is what each one is for.

PurposeExamplesIf you block it
TrainingGPTBot, ClaudeBot, Google-ExtendedYour pages are not used to train future models
Search indexOAI-SearchBot, Claude-SearchBot, PerplexityBotYou stop appearing in assistant search results
Fetch for a userChatGPT-User, Claude-User, Perplexity-UserAn assistant cannot open your page when someone asks it to

Google-Extended is a token you put in robots.txt, not a separate crawler. It controls AI training use and has no effect on Google Search.

A publisher whose product is the text has good reasons to block the first row. A company whose website exists to be found has almost none to block the second and third. A reasonable position for a business site is therefore to allow search and user fetches, and to make its own call on training:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

Names change and new agents appear, so check each vendor's own documentation rather than trusting a list in an article, this one included.

And remember the order of things. robots.txt is a request that polite crawlers honour. A firewall is a wall. If the firewall returns 403 for robots.txt itself, whatever you wrote in that file is never read.

After you fix it

Verify from outside, not from your own browser. Run the two curl commands again, then ask an assistant to open an inner page and describe it.

Check again after any change to security settings. These modes are easy to re-enable and nothing in a browser will tell you.

And put it in monitoring. A scheduled request for robots.txt that alerts on anything other than 200 would have caught ours on the first day. It is the kind of check that reports nothing for a year and then pays for itself in an afternoon.

This sits next to a wider question, whether the machines that read your site can tell what you sell. For shops that is already concrete: your catalogue may be in ChatGPT today. For everyone else, access comes first. A page nobody can fetch is not summarised badly. It is not summarised at all.

Whether assistants can reach and read a site is one of the six areas in our website audit, alongside the checks that only fail quietly.