Is your site blocking AI crawlers without knowing?

This is the failure that makes every other effort pointless, and the site owner is almost always the last to find out. If an answer engine gets a 403 at your door, nothing you publish behind it exists.

This is step 1 of how to improve your AI search visibility. Nothing else on that list matters until this one passes.

Test it yourself in one command

You do not need a tool for this. Run these and look at the status code. Anything other than 200 is a problem.

curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot" https://yoursite.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "ClaudeBot" https://yoursite.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "PerplexityBot" https://yoursite.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "ChatGPT-User" https://yoursite.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "Google-Extended" https://yoursite.com/

Then compare against a normal browser user agent. If Googlebot and a browser get 200 while the AI crawlers get 403, you have found it, and you have found it in about thirty seconds.

Also read your robots.txt with your own eyes at yoursite.com/robots.txt. Check it is the file you wrote. Several hosts and CDNs will serve a managed one instead, and it can disallow crawlers you never chose to block.

The settings that cause it

Grouped by where people actually find them.

Cloudflare

Three separate settings can do this independently, which is why people fix one and stay blocked.

• Block AI bots under Security. A single toggle that refuses the lot. Set it to do not block, and allow mixed-purpose crawlers.
• Managed robots.txt under AI Crawl Control. When on, Cloudflare serves its own robots.txt over yours, commonly with a content signal that opts you out.
• Bot Fight Mode. Useful protection, and it can challenge crawlers. Verified bots are normally exempt, so check the crawler counts in AI Crawl Control rather than assuming.

Cloudflare shows per-crawler allowed and blocked counts. That screen is the fastest way to see the truth, and it is also the first place a fix shows up.

WordPress and security plugins

Wordfence, All In One WP Security and similar tools ship bot-blocking rules that often predate these crawlers and catch them as unknown agents. Look for an allowlist and add the five user agents above.

Hosting platforms and WAFs

Vercel, Netlify and AWS WAF all have bot rules. Check any managed rule set for an AI or scraper category, since that is usually where these agents are grouped.

Two failures that look different but are the same

noindex. A stray <meta name="robots" content="noindex"> tells every engine to leave the page out. It usually arrives from a staging site that got copied to production, and it is invisible unless you look at the source.

JavaScript-only content. If your page is an empty shell until scripts run, an engine that does not execute JavaScript sees nothing. Turn JavaScript off in your browser and load your own homepage. Whatever is left is what a large part of the machine web sees. If it is blank, that is your real content problem, and no amount of writing fixes it.

We did this to ourselves

On 11 July 2026 we ran our own scanner on showuplabs.com and scored 0 out of 100. The reason was not our content. Our own Cloudflare configuration was returning a 403 to ClaudeBot, GPTBot and PerplexityBot. We were an AI visibility company that AI could not read, and we had no idea until the tool told us.

We publish the score daily on our proof page, including the months where it has not moved. If it can happen to a company that sells the fix, it is worth thirty seconds of your time to check.

After you unblock it

Nothing happens instantly. Engines re-crawl on their own schedule, so expect days to weeks before it shows anywhere. The first evidence is in your logs rather than in any score: AI crawler hits that were previously refused start returning 200. Watch that, not the headline number, or you will conclude it failed while it is working.

Once the door is open, the next question is whether what is behind it can be quoted. That is writing content AI will quote.

Keep reading

• How to improve your AI search visibility, the full five-step method this is step one of.
• Writing content AI will quote, step two.
• The 12-point AI visibility checklist.
• What is AEO? if you want the concept before the practice.

Judging a crawler you do not recognise

Blocking every unknown bot is a blunt instrument, and the useful question about any crawler is what it fetches, how often, and whether it obeys your robots.txt. A crawler worth allowing publishes that somewhere you can find from its user agent string. Ours is at showuplabs.com/bot, including the two lines that block it, and it is a fair template for what to expect from anyone else's.

Questions, answered