Google Explains How Robots.txt Affects AI Crawlers

Key Highlights

• Robots.txt Controls Crawling, Not AI Training: Google clarifies that robots.txt manages crawler access but does not automatically control how collected content is used afterward.

• Google-Extended Offers AI Usage Controls: Website owners can use Google-Extended to manage certain Gemini-related content uses without affecting Google Search rankings or inclusion.

• IETF Proposes a New AI Content Rule: The proposed Content-Usage directive could help websites communicate AI training preferences, but it is not yet a universally adopted standard.

Google has clarified an important limitation of robots.txt: the file controls crawler access to website URLs, but it does not automatically control how AI systems use content after collecting it.

The clarification came from Google Search Relations representatives Gary Illyes and Cherry Prommawin in an October 2026 episode of Google's The Non-Canonical Cabinet video series. The discussion covered AI crawler restrictions, supported robots.txt directives, and the correct handling of URLs containing non-English characters.

The announcement is relevant to website owners, SEO professionals, publishers, and businesses working to improve their visibility across traditional search engines and AI-powered platforms. It also comes as the Internet Engineering Task Force (IETF) develops a proposal for communicating AI content usage preferences.

What Did Google Say About Robots.txt and AI Training?

According to Gary Illyes, robots.txt is an access preference protocol defined by RFC 9309, the Robots Exclusion Protocol. Its primary purpose is to communicate which URLs a crawler may request.

However, restricting access and controlling subsequent content usage are different things.

For example, a website can use robots.txt to request that a particular AI crawler avoid specified pages. That restriction does not automatically establish what happens to content the system has already collected.

Google recommends identifying the specific crawler through its recognized user-agent token when a website owner wants to restrict one AI crawler while permitting others to access the website.

The clarification was also covered in Search Engine Journal's report on Google's robots.txt guidance.

Why This Matters for Website Owners

Consider a business that publishes product guides, industry reports, and educational articles. It wants Google Search to discover these resources but would prefer to restrict access by a particular third-party AI crawler.

A crawler-specific robots.txt rule can communicate that preference to compliant crawlers.

However, this approach has limitations. It does not automatically apply to other AI providers, guarantee compliance, or determine how information obtained before the restriction will be used.

Website owners should therefore identify their objectives before changing their robots.txt configuration.

How to Restrict Specific AI Crawlers Using Robots.txt

Website owners can create separate robots.txt groups for different crawlers.

For example, the following illustrative rule requests that a hypothetical AI crawler avoid the entire website:

User-agent: ExampleAI
Disallow: /

ExampleAI is a placeholder, not a real crawler token. Replace it with the appropriate token published by the provider you want to restrict.

The Disallow: / directive requests that the specified crawler avoid crawling all paths covered by that group.

If the website owner wants Googlebot to retain access, a separate group can specify its crawling preferences:

User-agent: Googlebot
Allow: /

These examples illustrate crawler-specific rules. Actual configurations should be checked for conflicts with existing directives and the intended behavior of each crawler.

Google documents the supported syntax in its official robots.txt specification.

Which Robots.txt Directives Does Google Support?

Google documents four primary robots.txt fields.

Website owners should not assume that adding an unfamiliar directive will create a new capability.

For example, a custom rule intended to prohibit AI training will not automatically be recognized simply because it appears in robots.txt. Google may ignore unsupported instructions, while other crawlers can have their own parsing behavior and supported extensions.

Before implementing restrictions, consult Google's robots.txt documentation and the relevant AI provider's official crawler guidelines.

What Is Google-Extended?

Google-Extended is a dedicated control token that website owners can use to manage certain AI-related uses of content collected by Google.

It does not have a separate HTTP crawler user agent. Instead, Google uses its existing crawlers, while Google-Extended acts as a control in robots.txt.

Google documents Google-Extended as a way to manage whether content Google crawls can be used for future Gemini model training and certain grounding applications involving Gemini Apps and Grounding with Google Search on Vertex AI.

Google also states that Google-Extended does not affect a website's inclusion in Google Search and is not a Google Search ranking signal.

Website owners can read more in Google's official documentation about common crawlers and Google-Extended.

Example: Restrict Google-Extended Uses

A website owner who wants to restrict the relevant Google-Extended uses can add this rule:

User-agent: Google-Extended
Disallow: /

This rule expresses a restriction on the uses covered by Google's documentation.

It does not block Googlebot from crawling the website for Google Search. However, website owners should review Google's current documentation before implementing changes, particularly when their website relies on Google products beyond conventional search.

Google-Extended is also not a universal AI content-control mechanism. It does not automatically apply to independent AI providers.

The distinction is important for businesses that want to retain Google Search visibility while managing their preferences for Google's AI-related uses of content.

IETF Proposal Could Introduce a Standard for AI Content Usage

Another important development is an Internet-Draft from the Internet Engineering Task Force (IETF).

The proposal, titled Associating AI Usage Preferences with Content in HTTP, was co-authored by Google's Gary Illyes and Mozilla's Martin Thomson.

It proposes a Content-Usage directive for robots.txt and a corresponding HTTP header field. The objective is to let content publishers communicate preferences about how their material should be used by automated systems.

Under the proposal, crawling permissions would remain separate from preferences concerning the subsequent use of collected content.

The current proposal can be reviewed through the official IETF Datatracker entry.

How Would the Proposed Content-Usage Directive Work?

The draft includes an example that expresses a preference against AI training while allowing crawling:

User-Agent: *
Allow: /
Content-Usage: train-ai=n

The example illustrates the proposed approach, not a universally supported robots.txt configuration.

The intended distinction is straightforward: a website can remain accessible to crawlers while expressing a preference about whether its content should be used for AI model training.

This could give publishers a more consistent way to communicate their preferences if the proposal is finalized and adopted by the systems that retrieve and process web content.

Is Content-Usage an Official Standard Yet?

No. As of October 26, 2026, the proposal remains an Internet-Draft rather than a finalized, universally adopted standard. The current draft is listed as expiring in February 2027.

Website owners should not assume that adding Content-Usage to their robots.txt file will cause Google or other AI providers to enforce the preference.

The proposal's progress can be followed through the IETF's AI usage preferences draft.

Until compatible implementations become available, website administrators should rely on controls supported by the individual providers they work with.

Google Recommends Encoding Non-English URL Paths

The October discussion also addressed robots.txt rules involving non-English characters and emoji.

Google's Cherry Prommawin recommended percent-encoding such characters in rule paths to improve compatibility with different crawlers.

Percent-encoding represents characters using a percentage sign followed by hexadecimal values. It is commonly used when URLs contain characters outside the standard ASCII character set.

For example, a URL path containing Hindi or Marathi characters may have a percent-encoded representation.

Google's parser can process raw UTF-8 characters and their encoded equivalents, but other or older crawlers may interpret raw characters differently.

The relevant technical requirements for URL path matching are defined in RFC 9309.

Why Multilingual Websites Should Pay Attention

Businesses operating multilingual websites should verify that their robots.txt rules match the actual paths they intend to control.

Incorrect encoding or mismatched paths could result in a crawler interpreting a rule differently from what the website owner intended.

A technical audit should include:

• Checking robots.txt rules for URLs containing non-English characters.

• Verifying that percent-encoded paths match the intended URLs.

• Testing rules against the relevant crawler's documented behavior.

• Reviewing existing Allow and Disallow directives for conflicts.

This is particularly relevant to publishers and businesses serving users across multiple languages and regions.

What Does Google's Clarification Mean for SEO?

The announcement is relevant to SEO because robots.txt is often used to manage how search engines crawl websites.

However, website owners must distinguish between crawling, indexing, and AI content usage.

Crawling occurs when a crawler requests a URL and retrieves its content.

Indexing involves processing content for possible inclusion in a search engine's index.

AI content usage refers to activities such as model training or using retrieved information to support generated answers.

These activities are related, but they are not interchangeable.

Robots.txt Does Not Guarantee Search Exclusion

Google explicitly warns against using robots.txt as a mechanism for keeping webpages out of its search results.

If a URL is blocked from crawling, Google may still discover and index the URL without accessing its content.

If the objective is to prevent a webpage from appearing in Google Search, website owners should review Google's guidance on noindex and blocking search indexing.

Importantly, Google must be able to crawl a page to read a noindex directive contained in that page. Blocking the URL in robots.txt can prevent Google from seeing the instruction.

This distinction is important for websites that want to retain organic visibility while managing crawler access for other purposes.

What Does This Mean for Generative Engine Optimization?

Generative Engine Optimization (GEO) focuses on improving a brand's visibility across AI-powered search and answer experiences.

The latest clarification reinforces the importance of understanding technical access controls, but it does not mean that allowing AI crawlers guarantees visibility in AI-generated answers.

An accessible webpage may never be selected as a source for a particular response. Similarly, restricting a crawler does not guarantee that previously collected content will disappear from every AI system.

For businesses investing in GEO, the following actions can help create a more informed strategy.

1. Audit Robots.txt Before Making Changes

Review your existing robots.txt file before introducing new AI crawler restrictions.

Check whether important service pages, product pages, articles, and other landing pages are accessible to the crawlers required for your SEO objectives.

Avoid applying broad restrictions without understanding which URLs and products they affect.

2. Identify AI Crawlers Individually

Create a list of relevant AI providers and their documented crawler tokens.

Determine whether each crawler is associated with search, AI-assisted browsing, training, or another activity.

Use the provider's official documentation rather than relying on assumptions about what a particular bot does.

3. Review Google-Extended Settings

If your business wants to manage Google's documented Gemini-related content uses, review Google-Extended.

Consider your objectives carefully before applying restrictions, particularly if your business relies on Google products that use retrieved web content.

Remember that Google-Extended is not a Google Search ranking signal.

4. Protect Important SEO Pages

Confirm that your robots.txt rules do not accidentally prevent search engines from crawling pages that should remain available in search results.

Review indexing directives, canonical tags, XML sitemaps, and Search Console reports alongside your crawler configuration.

A robots.txt change should support your wider technical SEO strategy rather than operate in isolation.

5. Monitor AI Visibility Separately

Track brand mentions, website citations, referral traffic, and visibility for relevant queries across AI-powered platforms where measurement is available.

Keep these reports separate from conventional organic search metrics.

This helps your team understand whether changes in technical access coincide with changes in search or AI visibility without assuming that one directly causes the other.

Robots.txt Audit Checklist for Website Owners

Use this checklist when reviewing your website's crawler and AI content-use policies.

One final precaution: robots.txt is not a security mechanism. It communicates crawling preferences to compliant crawlers but does not prevent unauthorized access to publicly accessible URLs. Sensitive information should be protected with authentication or other suitable security measures.

The technical basis for this distinction is explained in the Robots Exclusion Protocol standard.