There’s a report that blocking AI training bots with Cloudflare is also blocking legitimate Googlebot and Bingbot from indexing a website. Cloudflare recently announced that this would happen beginning in September. It’s unclear whether this is a fluke or user error but it does serve as a reminder to be aware of the pending change and to adjust settings accordingly.
Redditor Claims Cloudflare Blocked Googlebot
A Redditor in the r/SEO community asked if anyone else has experienced Cloudflare blocking Googlebot as an AI trainer.
They asked:
“Has anyone else run into Cloudflare AI Crawlers blocking Googlebot/Bingbot?
I’m testing Cloudflare’s new AI Crawlers & Scrapers feature and noticed something unexpected.
When I set AI Training = Block, both Googlebot and Bingbot start receiving HTTP 403 responses when trying to fetch my sitemap. As soon as I disable the AI Training block, the sitemap is accessible again.
From what I understand, Cloudflare now classifies Googlebot and Bingbot as “Search + Training” bots, so enabling the training block appears to affect them as well.
This leaves me with a dilemma:
- Allow AI Training so search engines can access my sitemap.
- Block AI Training and risk Google/Bing getting 403s.
Has anyone found a clean workaround?”
Google’s John Mueller responded to ask for a DM to take a closer look. Another Redditor asked if maybe Cloudflare was blocking rogue bots that were spoofing Googlebot’s user agent.
The original poster replied that wasn’t the case and offered more information:
“I understand that Cloudflare blocks fake Googlebots, but in my case I’m referring to what Cloudflare itself shows in the dashboard.
When I enable Bot Fight Mode, uploading my sitemap returns a 403 error. As soon as I disable it, the 403 disappears.
Also, in the AI Crawlers section, Cloudflare shows Googlebot and Bingbot as blocked automatically. That’s why I’m wondering if Bot Fight Mode is interfering with legitimate crawlers or if the dashboard is reporting something else.”
Cloudflare’s Official Documentation
In it’s official announcement states that beginning September 15 Cloudflare will begin blocking mixed-usage bots, which are bots like Googlebot that simultaneously index websites for search and collect data AI training. The effect is that Cloudflare will block Googlebot.
Cloudflare’s announcement explains:
“On September 15, 2026, Cloudflare will set updated defaults for new domains: bots classified as Training or as Agent will be blocked on pages that display ads, and Search will remain allowed.
Mixed-purpose crawlers that combine Search and Training will also be blocked by all configurations to block AI training, including the legacy “Block AI bots” option. Before September 15, all customers can opt out of these new defaults.”
That kind of blocking is supposed to go into effect in September so it’s curious that someone is reporting that this is happening right now.
Featured Image by Shutterstock/Erlin Diah