Last updated September 2026.
GPTBot and ClaudeBot are not minor bots anymore. Cloudflare’s own crawler analysis, published in August 2025, put GPTBot at 28.1% of AI-only crawler traffic in July 2025, up from 11.9% a year earlier. ClaudeBot held 23.3% of that same traffic, up from 15%. Both bots run as single-purpose Training crawlers, so both lose default access to ad-monetized pages the moment the new rule takes effect.
What changes on September 15
Cloudflare confirmed the new defaults in its own policy update, Your site, your rules: new AI traffic options for all customers, published July 1, 2026. The deadline itself sits in the table below.
| Detail | What Cloudflare confirmed |
|---|---|
| Effective date | September 15, 2026 |
| Who is affected first | Domains onboarding to Cloudflare on or after that date |
| Blocked by default | Training and Agent crawlers, on any page that carries ads |
| Still allowed by default | Search crawlers |
| Mixed-use bots affected | Googlebot, Applebot, and Bing, once a site already blocks Training |
| Opt-out window | Any time before September 15, 2026, in Security settings |
An ad on a page is the trigger, not the page’s topic. Cloudflare treats an ad as a signal that a human, not a bot, was meant to land there. On pages without ads, none of this changes.
Which crawlers count as mixed-use
Cloudflare sorts every bot into three use cases: Search, Agent, and Training. A bot that runs more than one of those at once, most often Search paired with Training, is what the company calls mixed-use. Cloudflare names Googlebot, Applebot, and Bing directly as its clearest examples, and it enforces the most restrictive rule across all of a mixed-use bot’s behavior.
GPTBot and ClaudeBot are not part of that mixed-use group. Both run Training only, so both already sit inside the category Cloudflare blocks by default on ad-monetized pages, mixed-use label or not.
| Crawler | Operator | Classification | Effect on September 15 |
|---|---|---|---|
| GPTBot | OpenAI | Training | Blocked by default on pages that carry ads |
| ClaudeBot | Anthropic | Training | Blocked by default on pages that carry ads |
| ChatGPT-User | OpenAI | Agent | Blocked by default on pages that carry ads |
| Googlebot | Search plus Training (mixed-use) | Loses search crawling too, once the site blocks Training | |
| Applebot | Apple | Search plus Training (mixed-use) | Loses search crawling too, once the site blocks Training |
| Bingbot | Microsoft | Search plus Training (mixed-use) | Loses search crawling too, once the site blocks Training |
The pre-deadline checklist
-
Confirm whether your domain counts as new. The automatic block on Training and Agent crawlers applies to domains onboarding to Cloudflare on or after September 15, 2026, not to every existing zone by default. Verify it: check the “added to Cloudflare” date on your zone overview page.
-
Open your AI Bots controls. Find the three toggles under Security > Settings: Search, Agent, and Training. Verify it: record the current state of each toggle with today’s date, so you have a snapshot from before the deadline.
-
Check your existing Block Training status. If you already block Training, through either the new controls or the legacy Block AI Bots service, mixed-use crawlers such as Googlebot lose their search access too. Verify it: read the classification tags Cloudflare lists for each bot in BotBase.
-
Decide whether to opt out of the mixed-use change. A site that wants Googlebot, Applebot, and Bing to keep crawling for search, even while blocking Training, has to set that preference before September 15. Verify it: look for the explicit opt-out control next to the AI Bots settings.
-
Map which pages carry ads. The new default block only applies to ad-monetized pages, so content-only sections of a site are not affected the same way. Verify it: cross-check your ad-tag placements against your sitemap.
-
Read your robots.txt for Content Signals. Cloudflare-managed robots.txt files now carry search, ai-train, and use fields that state a preference for how a crawler may use your content. Verify it: fetch yourdomain.com/robots.txt and confirm the Content-Signal line matches what you intend.
-
Pull 30 days of crawler visits by user agent. Isolate GPTBot and ClaudeBot specifically, since they are today’s two largest single-purpose Training crawlers and the most likely to disappear from your logs first. Verify it: export bot traffic from your CDN or from an AI SEO platform’s bot analytics module.
-
Schedule a post-deadline recount. Set a reminder for the week of September 15 to compare crawler visits before and after the cutover. Verify it: rerun the same 30-day pull and confirm the bots you expected to lose access actually stopped requesting pages.
Confirm the crawl stopped
Reading raw server logs bot by bot is slow. A handful of AI SEO platforms fold crawler evidence into the same dashboard as citation tracking, so a before-and-after comparison takes minutes instead of a log-parsing script.
Temso includes AI bot visit tracking on every plan, from 10,000 monthly visits on Starter to 100 million on Professional, next to citation and content data in the same account. Profound ties its Agent Analytics feature to GA4 and its citation maps, so a drop in GPTBot visits lines up against the URLs that used to get cited. BrightEdge keeps AI-search visibility inside the same enterprise workspace its customers already use for technical crawl reporting. Scrunch’s Agent Traffic module reads crawler visits at the CDN layer, broken out by bot, frequency, and page. Full profiles and Index scores for all four sit in the AI SEO tools index.
Check your Cloudflare Security settings against this list before September 15, 2026, then rerun your crawler count the week after to confirm the block worked. If you want that verification next to your citation and content data instead of a separate export, Temso’s bot traffic tracking is included on every plan, starting at $89 a month.