Uday Rajaram | Updated September 2026
A large share of medical practice websites are invisible to ChatGPT, Perplexity, and Claude, not because of anything in their content, but because their hosting or CDN returns a 403 error to AI crawlers by default. The robots.txt file often explicitly allows those crawlers, so every online checker gives the site a passing grade while the server refuses the request. The fix takes about ten minutes once you know where the setting is. We know because it happened to our own site: roughly two of every three AI crawler requests were being refused for weeks while our robots.txt scored perfectly.
What we found on our own site
In August our inbound leads slowed to nothing. Our rankings were flat, our content was fresh, and our robots.txt explicitly allowed GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, and ClaudeBot by name. Every AI-readiness checker we ran gave us a clean bill of health. Then we opened the crawler logs on our CDN and found the real story: GPTBot was being refused on 100 percent of its requests, OAI-SearchBot on 70 percent, ChatGPT-User on 60 percent, and PerplexityBot and ClaudeBot on every single one.
The cause was a default setting on our CDN that had been switched on for months without anyone touching it. The file that declares who is allowed in said yes; the server that answers the door said no. Search engines and answer engines only ever hear the server. Within a day of fixing it, allowed requests went from a trickle to hundreds per day, and within a week our pages were being cited in AI answers at the highest rate we had ever recorded.
Why the checkers miss it
Nearly every free AI crawler checker works the same way: it downloads your robots.txt and reads the rules. That is a check of what you declared, not what your server does. A CDN firewall rule, a bot-protection feature, a managed challenge, or a hosting provider's default block will refuse a crawler before robots.txt is ever consulted, and no amount of reading the file will reveal it. The only test that tells the truth is to request a page the way the crawler would and look at the status code that comes back.
How to check your site in 60 seconds
Open a terminal (on a Mac, the Terminal app; on Windows, PowerShell) and request your homepage while identifying as an AI crawler. Replace the domain with yours:
curl -A "GPTBot" -o /dev/null -s -w "%{http_code}" https://yourpractice.com/
A 200 means the crawler got in. A 403 means your server refused it. A 503 or a page full of JavaScript usually means a challenge page, which is a refusal by another name. Repeat with OAI-SearchBot, ChatGPT-User, PerplexityBot, and ClaudeBot. If you do not want to use a terminal, send us the domain and we will run it for you as part of a free audit.
Two caveats. Some hosts verify crawlers by IP address rather than by name, so a spoofed request from your laptop can be refused even when the real crawler would be allowed; treat a 403 as a strong signal to check your settings, not as final proof. And a 200 on the homepage does not guarantee every page; test a service page and a blog post too.
Where the block usually lives, and how to fix it
Cloudflare: the AI Crawl Control section has separate policies for search, agent, and training crawlers, plus a legacy Block AI Bots rule and a managed robots.txt that can override your own. Set the policies to allow, disable the legacy rule and managed robots.txt, and make sure the mixed-purpose crawler setting is also set to allow. Bot Fight Mode and Browser Integrity Check will also refuse crawlers and should be off for a content site.
WP Engine, Kinsta, and similar managed WordPress hosts: many ship with AI crawler blocking on by default, usually in a security or bot-protection panel. Look for anything labeled AI bots, scrapers, or LLM crawlers and switch it to allow.
Security plugins and WAFs: Wordfence, Sucuri, and similar tools have bot rules that can catch AI user agents. Whitelist the crawlers you want.
After the fix, confirm it the same way you found it: rerun the curl tests and watch your CDN or server logs for allowed requests. Then submit your key pages to Bing Webmaster Tools, since ChatGPT reads Bing's index and the sooner Bing recrawls, the sooner ChatGPT can cite you.
Why this matters more than it sounds
Patients now ask AI assistants who to see. In our own analytics, ChatGPT is the second-best source of booked consultations after brand-name Google searches, and the visitors it sends are the most engaged on the site. A practice that is blocked at the server level is not competing for those patients at all; it is invisible in the conversation where the decision gets made. And because the block is silent, most practices never find out. They just see fewer leads and assume the market got harder.
Related: AEO for Medical Practices, How to Get Your Cosmetic Practice on ChatGPT, Editorial Standards.