Google AI Training Opt Out Robots.txt: What the New Rule Means
Short answer
On October 30, 2026, Google announced it is working with other AI companies on a shared robots.txt signal that would let website owners opt out of AI training in one place, instead of listing each AI crawler separately. The control is proposed, not live, and no rollout date is confirmed. Existing robots.txt rules for crawlers like Google-Extended and GPTBot still work the same way until it ships.
A site owner reads a headline about Google letting websites opt out of AI training through robots.txt.
They open their robots.txt file, ready to add the new line.
There isn't one to add yet.
That's the gap between the headline and the actual announcement. Google said it is working on a Google AI training opt out robots.txt signal, developed with other AI companies, but it hasn't shipped, and there's no new directive published to use today.
Here's what Google actually announced on October 30, 2026, what already works right now to control AI crawlers, and what to check on your own site while the proposal is still a proposal.
What Google Announced on October 30, 2026
On October 30, 2026, Google said it is developing a shared signal that website owners could add to their robots.txt file to opt out of having their content used to train AI models.
The announcement described this as a proposal in development, not a feature that went live that day. Google framed it as an early step, something it's working through with other companies that operate AI crawlers, rather than a finished specification ready to publish.
That distinction matters for anyone updating a robots.txt file this week. Nothing changed in how robots.txt works on October 30. No new token exists to add, and no existing rule behaves differently because of the announcement.
The core idea behind the proposal
The idea, as described, is a single directive that multiple AI companies would recognize as meaning "don't use this content to train AI models." A site owner would add it once, in one file, instead of maintaining a separate rule for every AI crawler that might read the site.
That's a meaningfully different idea from blocking a crawler outright. Today, blocking a crawler from fetching a page at all is a separate decision from telling a company not to use already-fetched content for training. The proposal is aimed at the second case: a shared way to express a training preference, not a replacement for existing crawl permissions.
Why Google is framing this as collaborative, not Google-only
Google's announcement repeatedly framed the effort as cross-industry rather than something it would define alone and expect others to follow. A signal only works if the companies reading robots.txt files agree on what it means, so Google said it has been talking with other companies that run AI crawlers about adopting the same approach.
That framing is also a practical necessity. Robots.txt has worked for decades because crawlers across the web agree, informally, to read and respect it the same way. A training opt out that only one company honored would tell a site owner very little about what happens to their content everywhere else.
Why a Shared Signal Instead of Separate Rules per Crawler
Right now, a site owner who wants to express a preference to multiple AI crawlers has to do it one user agent at a time.
The current patchwork of per-crawler rules
Each AI company that crawls the web for training or for live retrieval publishes its own crawler, identified by its own user-agent token in robots.txt. Blocking one doesn't block another. A rule written for one crawler's token has no effect on a different company's bot, even if the intent behind both rules is the same. Our guide to user-agent examples walks through which tokens belong to which crawlers and how to tell them apart in a robots.txt file.
For a site owner who wants to opt out of AI training broadly, that means listing every known AI training crawler by name, and revisiting the list whenever a new one appears. There's no single line that currently covers all of them at once.
What a shared signal would change in practice
If the proposal ships as described, a site owner would write one directive meant to signal a training preference to every company that agreed to honor it, rather than repeating the same intent across a growing list of separate tokens.
That would simplify maintenance, but it wouldn't eliminate the need to understand what's actually being crawled. A shared training signal, as described, would only be read by companies that choose to honor it. A crawler that doesn't recognize or respect the directive would simply ignore it, the same way any crawler can choose to ignore a robots.txt rule it doesn't recognize today.
Which AI Providers Have Signaled Support
Google's announcement named a handful of other companies it said are involved in early discussions about the proposal. It's worth being precise about what that means and what it doesn't.
What's been confirmed so far
Google described the effort as a conversation with other AI companies that operate crawlers, rather than a signed agreement already in place. Being named in that conversation isn't the same as a company publishing its own commitment to honor a specific directive once one exists.
At this stage, the most reliable confirmation of any company's position is that company's own documentation, published on its own site. Google's announcement is the right place to learn what Google is proposing. It isn't a substitute for a separate AI provider stating, on its own terms, how it plans to read or respect the signal.
What's still unclear about participation
Several things aren't settled yet, based on what's public. Whether every company named in early discussions will adopt the final specification once it's published, whether all of them will honor it the same way, and whether companies not mentioned at all might adopt it later are all open questions.
It's also not yet clear whether participation would be uniform. A company could, in principle, agree to recognize the signal for some of its products or crawlers but not others, the way AI companies already draw different lines between a crawler used for training and one used for live retrieval at the moment someone asks a question. None of that has been spelled out in detail yet.
The Planned Rollout: What's Confirmed and What Isn't
Google's announcement didn't include a launch date. It described the proposal as something still being worked out, with a feedback or testing period ahead rather than an immediate release.
What Google says comes next
What Google has said is that it plans to continue refining the proposal with the other companies involved before publishing a final specification that site owners could actually implement. No syntax for the directive itself has been published. That means there's no confirmed name for the new line, no confirmed format, and no confirmed answer for exactly how it would interact with existing Allow and Disallow rules in a robots.txt file.
Until that specification is published, anything written about the exact wording of the new directive would be a guess, and it's worth treating any such guess, from any source, with caution until Google documents it directly.
Why there's no new directive to add yet
This is the part most worth repeating plainly: there is nothing new to add to a robots.txt file today because of this announcement. The existing rules for Google-Extended, GPTBot, ClaudeBot and other named AI crawlers work exactly as they did before October 30, 2026. A site owner who wants to express an AI training preference right now still has to do it the way it's always been done, crawler by crawler, using the tokens that already exist.
The rest of this guide covers those existing controls, since they're the only ones that actually do anything on a live site today.
How the Proposed Control Would Work
Google hasn't published the exact syntax for the new directive, so nothing here describes a confirmed format. What follows is based only on how Google described the proposal, and it's worth reading as a general shape rather than a spec to copy.
Where it would sit inside robots.txt
Robots.txt already has a well-documented structure: a user agent line, followed by Allow and Disallow rules that apply to that user agent. Google's own introduction to robots.txt covers how that structure works and what it's built to control. A shared training signal, as described, would sit inside that same file format rather than replace it. The most likely shape, based on how Google framed the idea, is a new kind of directive recognized across user-agent blocks, rather than a separate file or a setting outside robots.txt entirely.
That matters because it means a training opt out, once published, would probably live alongside the Allow and Disallow rules a site already has, not somewhere new that owners would need to go looking for.
How it differs from a Disallow rule
A Disallow rule tells a crawler not to fetch a page at all. It's a crawling instruction: the crawler either requests the page or it doesn't.
A training opt out, as Google described it, is a different kind of instruction. It would apply to content a crawler may still be permitted to fetch, and instead say something about what that content can be used for afterward. That's a meaningful difference. A company could, in principle, still crawl a page to serve some other purpose, like checking whether a page exists or indexing it for a product separate from training, while still honoring a request not to use that content as training data.
Until the final specification is public, it's not confirmed whether the new signal would stand alongside a Disallow rule, replace the need for one in some cases, or work independently of crawling permissions altogether. Google's documentation on how robots.txt works is the best current reference for the rules that do exist today, and it's worth checking back there once Google updates it with the new directive.
The Crawler Controls That Already Exist Today
None of the uncertainty above should stop a site owner from using the controls that are already documented and already work. These apply right now, regardless of when the shared signal ships.
Google-Extended and Google's AI training
Google already publishes a separate crawler control called Google-Extended, specifically for AI training and related Google products, distinct from Googlebot itself. Google's own documentation on Google-Extended explains what blocking it does and doesn't affect. Disallowing Google-Extended in robots.txt doesn't remove a site from Google Search. It only affects whether that content is used for the specific products Google-Extended governs.
Other AI crawler tokens worth knowing
Other AI companies publish their own crawler tokens the same way, each with its own name and its own documented purpose. GPTBot, ClaudeBot and PerplexityBot are among the more commonly seen ones, and each behaves independently of the others. Blocking one has no effect on whether the others still crawl a site. RepGet's free AI Crawler Checker tests a site against several of these at once, which is a faster way to see what's currently allowed than checking each token by hand.
llms.txt as a separate convention, not the same thing
A related but different idea is llms.txt, a convention described at llmstxt.org for giving AI systems a clear summary of a site's content in a single file. It's worth being clear that llms.txt isn't the same mechanism as robots.txt, and it isn't the same as Google's proposed training signal either. It's an informal convention some sites publish to make their content easier for an AI system to read and summarize, not a crawling permission or a training opt out.
Crawler name | Company it belongs to | What blocking it actually affects |
|---|---|---|
Googlebot | Whether pages appear in Google Search results | |
Google-Extended | Whether content is used for AI training and related features, separate from Search | |
GPTBot | OpenAI | Whether content is crawled for that company's AI training, documented on its own site |
ClaudeBot | Anthropic | Whether content is crawled for that company's AI training, documented on its own site |
PerplexityBot | Perplexity | Whether content is crawled for that company's AI products, documented on its own site |
AI Training vs Search Indexing vs AI Visibility: Three Different Things
A lot of confusion about this announcement comes from treating these three as one thing. They aren't, and they're controlled separately.
Search indexing: Googlebot
Search indexing is handled by Googlebot. It's what determines whether a page can appear as a result in Google Search. Blocking Googlebot removes a site from Search results. It has nothing to do with AI training, since Googlebot and Google-Extended are separate crawlers with separate documented purposes.
AI training: Google-Extended and similar tokens
AI training is a separate process, governed by tokens like Google-Extended and the other crawlers listed above. These control whether content is used to train a model, not whether that content shows up in search results. A site can block Google-Extended and still rank normally in Google Search, because the two are independent settings.
AI visibility: what happens at query time
AI visibility is different again. It's what happens when someone asks an AI assistant a question and the assistant retrieves or references information to answer it, often at the moment the question is asked rather than from a model trained months earlier. Blocking a training crawler doesn't necessarily change whether a page can still be read live for this purpose, since some assistants retrieve information separately from how they were trained. Our guide to getting mentioned in ChatGPT covers that distinction in more detail, including why no provider guarantees a specific page will appear in a specific answer.
Concept | Mechanism or crawler involved | What blocking or allowing it changes |
|---|---|---|
Search indexing | Googlebot | Whether pages appear in Google Search results |
AI training | Google-Extended and similar crawler tokens | Whether content is used to train AI models |
AI visibility | Live retrieval at the moment a question is asked | Whether content can still be referenced in an assistant's answer, independent of training settings |
A Worked Example: Checking What a Site Currently Allows
Imagine a small accounting firm that set up its robots.txt file two years ago and hasn't touched it since. This example is illustrative.
The owner reads about Google's announcement and decides to check what the file actually says before assuming anything needs to change. They open the file directly at their site's root address and read through it line by line.
What the example site's file actually showed
The file blocks Google-Extended, added at some point after reading advice about AI training. It says nothing at all about GPTBot, ClaudeBot or PerplexityBot. No rule exists for any of them, which means none of those three are currently restricted from crawling the site for whatever purpose they crawl for.
Running the same file through RepGet's free Robots.txt Checker confirms the same thing: a clear verdict on Google-Extended, and an open verdict on the others, with no rule written for them either way. That gap wasn't intentional. It was simply what the owner had read about two years ago, when Google-Extended was the only AI crawler token getting much attention.
Nothing about this example requires waiting for Google's new proposal. The gap it reveals is something that's fixable today, using tokens that are already documented and already work, which is the subject of the checklist that follows.
A 20-Minute Checklist to Prepare Your Site
None of this requires waiting for Google to publish anything. A short review now means there's nothing left to scramble over once the new signal actually ships.
Step | Time | Tool or action to use |
|---|---|---|
Open your current robots.txt file and read every line | 5 minutes | Visit the file directly at your site's root address |
Check which AI crawlers are blocked, allowed, or not mentioned at all | 5 minutes | RepGet's free AI Crawler Checker |
Decide a plain policy: block AI training crawlers, allow them, or leave the decision for later | 5 minutes | A short written note, even a line in a shared document |
Check whether an llms.txt file exists or is worth adding | 3 minutes | RepGet's free llms.txt Generator |
Set a reminder to revisit robots.txt once Google publishes the final specification | 2 minutes | A calendar note, since no date has been confirmed yet |
Twenty minutes is enough to turn an assumption into an actual decision. Most site owners haven't looked at their robots.txt file since it was first set up, and the gap between what it says and what they think it says is usually small but worth closing before adding anything new on top of it.
Common Mistakes Site Owners Should Avoid
A few mix-ups come up often enough with this topic that they're worth naming directly, separate from the checklist above.
Mistake | What it leads to | How to fix it |
|---|---|---|
Confusing Google-Extended with Googlebot | Blocking the wrong token and accidentally affecting Search visibility, or assuming a training block already covers Search | Treat them as two separate rules with two separate purposes, and check Google's own documentation if unsure which token does what |
Assuming an AI training opt out removes a page from AI assistant answers | A false sense that blocking training settles the question of AI visibility entirely | Remember that live retrieval at query time is a separate mechanism from training, and isn't necessarily affected the same way |
Waiting on the unreleased Google proposal before doing anything | Months of no crawler policy at all, even though working controls already exist | Use the documented tokens available today, and revisit the file once the new signal ships |
Treating llms.txt as the same thing as the proposed training signal | Publishing a file that doesn't actually express a training preference and expecting it to | Keep the two separate: llms.txt is a content summary convention, not a crawling or training permission |
The pattern underneath most of these mistakes is the same one that caused the confusion in the first place: treating a proposal still in development as though it were already a working feature. It isn't, and nothing about reading this article changes that.
Where RepGet fits
RepGet doesn't set AI training policy for a site, and it doesn't track Google's unreleased proposal on a business's behalf. There's nothing to configure yet, so there's nothing RepGet or any other tool can switch on early.
What RepGet does help with is seeing what a site's robots.txt file actually says right now, rather than what an owner assumes it says. Its free AI Crawler Checker tests a site against crawlers like GPTBot, ClaudeBot and PerplexityBot together, and its free Robots.txt Checker gives a verdict on each rule already in the file. Its free website audit goes wider still, crawling the whole site and giving it a 0 to 100 SEO score with technical issues worth reviewing, including crawler access problems that have nothing to do with AI at all. No account or credit card required.
RepGet doesn't guarantee a ranking, a training opt out being honored by any specific AI company, or a mention in any AI assistant's answer. The free tools above show what a site currently allows. The decision about what to allow is still the site owner's to make.
The bottom line
Google's October 30, 2026 announcement describes a shared robots.txt signal for opting out of AI training, developed alongside other companies that run AI crawlers. It's a proposal, not a shipped feature. No syntax has been published, no launch date is confirmed, and there's no new line to add to a robots.txt file because of the announcement itself.
What hasn't changed is everything that already worked before October 30. Google-Extended, GPTBot, ClaudeBot and PerplexityBot are all documented crawlers with their own tokens, and a site owner can set a clear policy on each of them today. Search indexing, AI training and AI visibility remain three separate things, controlled separately, and no single setting covers all three at once. The practical move right now is using the controls that exist, checking what a site's file actually says rather than assuming, and watching for Google's follow-up documentation before changing anything further.
Find out what your website needs first
Run a free website audit with RepGet to see what your site's crawler rules and content actually look like today. No account or credit card required.
FAQ
Is Google's AI training opt out in robots.txt live yet?
No. Google announced it is developing the signal with other AI companies on October 30, 2026, but no syntax, specification or launch date has been published, so there is nothing new to add to a robots.txt file yet.
How do I block AI training crawlers from my site today?
Add Disallow rules for the specific crawler tokens you want to restrict, such as Google-Extended, GPTBot, ClaudeBot or PerplexityBot, since each works independently and none is covered by a single shared rule yet.
What's the difference between Google-Extended and Googlebot?
Googlebot handles whether a page appears in Google Search results. Google-Extended is a separate, documented crawler that governs AI training and related features, and blocking one has no effect on the other.
Will opting out of AI training remove my site from ChatGPT or Gemini answers?
Not necessarily. Many AI assistants retrieve information live at the moment a question is asked, separate from training, so a training opt out doesn't automatically change what shows up in a specific answer.
Which AI companies support Google's proposed robots.txt signal?
Google said it has been talking with other companies that run AI crawlers, but it described this as early discussion rather than a confirmed agreement, so check each company's own documentation for its actual position.
Does blocking AI training crawlers hurt my Google Search rankings?
No. Blocking a training-specific crawler like Google-Extended doesn't affect Googlebot or Search indexing, since the two are separate, independently documented controls.
What is llms.txt and is it the same as Google's new proposal?
No. llms.txt is an informal convention for summarizing a site's content for AI systems to read, described at llmstxt.org. It isn't a crawling permission or a training opt out like the signal Google proposed.
When will the new AI training control actually be available?
Google hasn't given a date. The announcement described ongoing work with other companies and a specification still being refined, with no confirmed timeline for when site owners could actually implement it.