What Is robots.txt? Crawling Rules, noindex and Access Control

What is robots.txt?

robots.txt is a plain-text file at a website's root that tells automated crawlers which paths they may or may not fetch.

A typical location is https://example.com/robots.txt. A User-agent line identifies the crawler group, Disallow identifies restricted paths and Allow identifies permitted paths. A crawler applies the matching rules under the protocol's selection rules.

The convention dates to 1994. IETF published it as the formal standard RFC 9309 in 2022.

Crawling, indexing and authorization are different controls

RFC 9309 states that these rules are not access authorization. Private information needs actual access control, such as authentication. A file asking cooperative bots not to fetch a path is not a lock on that path.

For Google search indexing, noindex is a separate directive. If robots.txt prevents Google from retrieving a page, it may also prevent Google from seeing that page's noindex instruction.

CDN bot protection can independently deny a request. Allowing a path in robots.txt does not override that network-layer decision.

AI providers can use different agents for different purposes

Providers document agents for uses such as model development, search and user-requested page retrieval. A rule aimed at one named agent does not automatically state a policy for every other use.

Check the relevant provider's purpose and behavior before writing a rule. User-requested retrieval may be documented differently from automated crawling. The AI crawler policy article discusses the distinction; its historical observations should retain their capture dates.

Related terms

Web Crawler · noindex directive · Content Delivery Network