Cloudflare Pay Per Crawl: How It Works, What Changed in 2026 and What It Means for SEO

Cloudflare pay per crawl lets sites charge AI crawlers. How it works, what changed in 2026, and why blocking AI training can also block Googlebot.
Cloudflare pay per crawl: a crawler on a web page linked to a dollar coin, with the words pay-per-crawl Cloudflare pay per crawl: a crawler on a web page linked to a dollar coin, with the words pay-per-crawl

The costly mistake with Cloudflare’s AI crawler settings is not about money. It is blocking Googlebot by accident. Cloudflare says that if you block “Training” crawlers, multi-purpose crawlers such as Googlebot, Applebot and Bingbot are blocked too [5]. Googlebot crawls for Google Search, so that puts your search traffic at risk.

Cloudflare pay per crawl lives in the same part of the dashboard. It lets a site charge AI crawlers for each page they fetch. Here is how it works, what changed in 2026 and what to check today.

Updated 17 November 2026: rebuilt with Cloudflare’s 2026 changes and current sources. They include Search, Agent and Training controls, new defaults from 15 September 2026, and Pay Per Use.

If you only do one thing: check whether “Training” crawlers are blocked in your Cloudflare settings. If they are, run a live test in Search Console’s URL Inspection tool on an important page. It tells you whether Google can still fetch it.

Cloudflare announced pay per crawl on 1 July 2025, in private beta [1]. It lets a website owner charge AI crawlers for access. The charge uses HTTP status code 402, “Payment Required”, which the web had barely used before [1].

The owner sets one flat price per request for the whole site. For each crawler, there are three choices [1]:

  • Allow: the crawler gets free access.
  • Charge: the crawler must pay the site’s price.
  • Block: no access, and no option to pay.

Cloudflare acts as the Merchant of Record and runs the technical side [1]. A crawler with no billing relationship can still be set to “charge”. In practice that works like a block, but it tells the crawler that paying is possible later [1].

On the same day, Cloudflare said it now blocks AI crawlers by default when they access content without permission or payment [2].

Expert quote: “AI crawlers have been scraping content without limits.” — Matthew Prince, co-founder and CEO, Cloudflare [2]

How does a paid crawl work?

With Cloudflare pay per crawl, the exchange happens in HTTP headers. This is the flow Cloudflare describes, with the price left as a placeholder [1]:

  1. The crawler asks for a paid page. Cloudflare answers with a 402 and a crawler-price header.
  2. The crawler retries with a crawler-exact-price header. That means it agrees to pay.
  3. Cloudflare serves the page with a 200 and a crawler-charged header.
GET /example.html
HTTP 402 Payment Required
crawler-price: USD XX.XX

GET /example.html
crawler-exact-price: USD XX.XX
HTTP 200 OK
crawler-charged: USD XX.XX

A crawler can also offer a ceiling first, with a crawler-max-price header [1]. To stop impostors, paying crawlers sign their requests using the Web Bot Auth proposals [1].

What changed in 2026?

A lot. Cloudflare widened the controls, changed the defaults for new sites and started moving from paying per fetch to paying per use. Here is the old model against the new one:

TopicJuly 2025November 2026
StatusPrivate beta [1]Closed beta, per Cloudflare’s docs updated 28 July 2026 [3]
PricingOne flat price per request, site-wide [1]Free pages by URL pattern, and dynamic prices set by your server or a Worker (16 June 2026) [4]
Crawler controlsOne “block AI bots” style choice [5]Separate Search, Agent and Training options, on every plan including Free (1 July 2026) [5]
Defaults for new domainsAI crawlers blocked unless allowed [2]From 15 September 2026: Training and Agent blocked on pages that show ads; Search allowed [5]
Payment modelPay per crawl: paid when a page is fetched [1]Pay Per Use: paid when content is used, starting with Ceramic.ai and You.com [6]

Two smaller changes are worth knowing. Since 17 April 2026, Cloudflare can redirect verified AI training crawlers with a 301. When they ask for a duplicate page, they go to your canonical URL [4]. And Enterprise Bot Management customers now get BotBase, a searchable directory of known bots and how Cloudflare classifies them [5].

The Pay Per Use shift is the biggest change in thinking. With Ceramic.ai, Cloudflare says a publisher who opts in is paid every time its content appears in Ceramic’s AI search results. The publisher also gets back the queries, citations and ranking [6].

Why can blocking AI training block Googlebot?

Some crawlers do more than one job. Cloudflare calls these multi-purpose crawlers, and it applies the most restrictive rule that fits. So if you block Training, it says Googlebot, Applebot and Bingbot are blocked as well [5].

For SEO, that is the part to get right. Google has a separate way to opt out of AI training. Its Google-Extended token in robots.txt controls whether your content may train Gemini models. Google says Google-Extended does not affect your inclusion or ranking in Google Search [7].

The two companies see this differently. Cloudflare argues that a small site faces a bad bargain. It can show up in search and let AI train on it, or risk not being found. It says this favours search providers that use the same bots for search and training [5]. Google’s position, in its own documentation, is that Google-Extended already separates the two, because it controls training without touching Search [7].

Watch out: Cloudflare said owners could opt out of the new defaults in their Security settings before 15 September 2026 [5]. If your domain joined Cloudflare after that date, check what it is set to now.

Who is affected?

  • New Cloudflare domains since 15 September 2026. They get the new defaults on pages that show ads [5].
  • Existing sites that block Training. Through either the new options or the older “Block AI bots” setting, Cloudflare says Googlebot, Applebot and Bingbot are blocked too [5].
  • Enterprise customers. BotBase shows which known bots reach you and how each is classified [5].
  • Publishers who want to charge. Cloudflare pay per crawl is still a closed beta. You join through Cloudflare’s sign-up page or, on Enterprise, your account executive [3].
  • Sites not on Cloudflare. None of these settings apply. robots.txt rules, including Google-Extended, still do [7].

Our analysis: access matters more than income

This section is our own judgement, based on the sources above.

Most sites will not earn from this soon. Cloudflare pay per crawl is still a closed beta after 16 months [3]. We found no public figures on what any publisher has earned from it.

The access settings matter now. They reach every plan, including Free [5]. One wrong choice can stop search crawlers, which costs far more than any crawl fee pays.

The model is still moving. Cloudflare is turning pay per crawl into Pay Per Use [6]. To us, that suggests charging per fetch alone did not settle the market. Treat any price you set today as an experiment.

What should you do now?

Match the setting to your goal. This table sums up the options from the sources above:

Your goalWhat to useWatch out
Stay in Google Search and AI OverviewsKeep Googlebot allowedBlocking Training in Cloudflare also blocks Googlebot [5]
Keep content out of Gemini trainingGoogle-Extended in robots.txtDoes not affect Search inclusion or ranking [7]
Charge AI crawlersCloudflare pay per crawl (closed beta)Join through Cloudflare first [3]
Charge only for some pagesConfiguration Rules or dynamic pricingAdded 16 June 2026 [4]
  1. Open your Cloudflare AI crawler settings. Note what Search, Agent and Training are set to.
  2. Test Google’s access. Run a live test in URL Inspection on two or three key pages.
  3. Check robots.txt. Make sure it says what you mean for Googlebot and Google-Extended.
  4. Decide on charging later. If you want to, ask Cloudflare to join the beta. Start with a few pages.

From our own site: aiseojournal.net is served through Cloudflare. On 17 November 2026 we requested a page using crawler names. Requests named GPTBot got HTTP 429 four times out of four. OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Bytespider and CCBot got 200. Fake Googlebot and Bingbot names got 403, which is how Cloudflare stops impostors. Limit: we could only send the names. Real crawlers are checked by IP and signatures, so Cloudflare may treat them differently.

Professional tip: A live test in URL Inspection uses Google’s own crawler, so it is the most reliable check you can run. Our own live test on 17 November 2026 showed “URL is available to Google”.

What we still don’t know

  • Whether the 15 September defaults rolled out as planned. Cloudflare’s AI Crawl Control changelog shows no entry after 16 June 2026 [4].
  • When Cloudflare pay per crawl leaves beta. The docs still say closed beta [3].
  • How Pay Per Use will be priced. Cloudflare names partners but gives no rates [6].
  • What publishers earn. We found no published figures.

Frequently asked questions

It is a Cloudflare feature that lets a website charge AI crawlers for each page they fetch. It uses HTTP status code 402, “Payment Required”. It is still in closed beta.

It can. Cloudflare says that if you block Training, multi-purpose crawlers such as Googlebot, Applebot and Bingbot are blocked too.

Use the Google-Extended token in robots.txt. Google says it does not affect inclusion or ranking in Google Search.

It is Cloudflare’s newer model, announced on 1 July 2026. Publishers who opt in are paid when their content is used, not just fetched. One example is content that appears in an AI search result.

Not yet. Cloudflare’s documentation says it is in closed beta, and you join through Cloudflare.

Where to go next

To set your crawler rules, read our guide to managing AI crawlers with robots.txt. For the training opt-outs from Google and Bing, see AI training opt-outs explained. For crawling and indexing basics, start with our technical SEO guide.

Sources

Add a comment

Leave a Reply

Your email address will not be published. Required fields are marked *

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use