Robots.txt

Websites are visited by more than human readers. Search engines, feed readers, archives, independent discovery services, monitors, and AI-related systems may retrieve publicly accessible resources. Those systems have different purposes: some discover links and index pages, some check for updates, and others retrieve material for automated processing.

robots.txt is a long-established way for website operators to publish rules for cooperative crawlers. The file communicates expectations about which resources a crawler should request. It is not authentication, authorization, or a substitute for security controls.

Three concepts must stay separate:

1. Crawling — whether a crawler should request a resource (robots.txt Allow/Disallow).
2. Indexing and presentation — whether and how a search engine should represent a resource in results (HTML robots meta tags, HTTP X-Robots-Tag).
3. Content usage — what the publisher prefers about downstream uses such as AI training or AI-assisted answers (emerging drafts and vendor conventions).

These mechanisms are not interchangeable. Blocking a fetch is not the same as asking not to index a page, and neither is a copyright license.

1. Crawl robots.txt says which paths cooperative crawlers should request.
2. Index Meta tags and X-Robots-Tag guide how engines may present retrieved pages.
3. Usage Draft and vendor signals express preferences about AI and related processing.

How websites learned to communicate with crawlers

The Robots Exclusion Protocol emerged in 1994 as a simple text file that early web crawlers agreed to consult. A root-level robots.txt became a widely recognized coordination point because it was easy to publish and parse.

In September 2022, RFC 9309 formalized the protocol: location, encoding, user-agent groups, Allow/Disallow matching, redirects, HTTP status handling, caching, and processing limits. Original conventions, the standardized protocol, and crawler-specific extensions remain distinct. Growth in automated content processing has renewed interest in preferences beyond ordinary crawl access—those ideas are drafts or vendor systems unless an RFC says otherwise.

The Robots Exclusion Protocol

RFC 9309 uses normative language carefully. MUST means a requirement for compliant implementations. SHOULD is a strong recommendation. MAY is optional. Do not treat every SHOULD as a universal law of every commercial crawler.

Key requirements and recommendations:

See RFC 9309 for authoritative wording on each point.

Create your first robots.txt file

1. Choose the resource location

Publish at the origin root:

https://joshua.example/robots.txt

The filename is lowercase. A file at /blog/robots.txt does not govern the entire website.

2. Create the text file

User-agent: *
Allow: /

Sitemap: https://joshua.example/sitemap.xml

User-agent: * selects the default group for crawlers without a more specific match. Allow: / permits the whole path space for that group. Sitemap: advertises an XML sitemap location (widely recognized; not an Allow/Disallow rule).

3. Publish

Deploy with ordinary static hosting, or generate the file in a build (Jekyll, Hugo, Eleventy, WordPress, custom servers). Confirm the live URL, not only the file in the repository.

4. Check the response

curl -i https://joshua.example/robots.txt

Expect HTTP 200, plain text, UTF-8, and the intended body.

5. Test the policy

For a given crawler product token and URL path, determine which group applies and which Allow/Disallow rule wins. Test before changing production policies that block large path trees.

Different crawlers, different rules

Clients send an HTTP User-Agent header. In robots.txt, User-agent lines name product tokens (often the first token of that header). Matching is case-insensitive for the product token.

Wildcard group:

User-agent: *
Disallow: /private/

Named group:

User-agent: ExampleBot
Disallow: /experiments/

A named group is not simply stacked on top of every * rule. The protocol selects which group(s) apply. Typically a more specific named match is used instead of falling back to *. HTTP user-agent strings can be spoofed; they are not authentication. Some operators publish verification mechanisms (IP ranges, reverse DNS, signed tokens)—those are separate from claiming a product token in a header.

Allow and Disallow

Disallow identifies a path prefix a cooperative crawler must not request:

User-agent: *
Disallow: /drafts/

Allow can carve an exception from a broader restriction:

User-agent: *
Disallow: /projects/private/
Allow: /projects/private/public-demo/

Empty versus root:

Disallow:

does not establish a path restriction (no pattern).

Disallow: /

matches the entire path space. Accidentally omitting or adding the slash changes the meaning completely.

How a crawler decides which rule wins

Matching starts from the beginning of the URI path. Among applicable rules, the longest matching pattern wins (octet/byte length). Path comparisons are case-sensitive. When Allow and Disallow match with equal length, Allow wins under RFC 9309. Percent-encoding and reserved characters matter; encode patterns carefully when matching literal special characters. Query strings are part of URI handling—follow the RFC rather than inventing search-engine folklore.

/admin and /admin/ can match different URL sets: the trailing slash narrows or expands which paths share the prefix.

Wildcards and end-of-path matching

RFC 9309 defines special characters inside path patterns:

• * — matches any sequence of characters
• $ — matches the end of the path

Examples:

User-agent: *
Disallow: /*.pdf$
Allow: /public/*.pdf$
Disallow: /tmp/*

A path * is not the same as User-agent: *. Percent-encode when you need a literal reserved character in a pattern. Always check the RFC; undocumented search-engine quirks are not the protocol.

Multiple groups and conflicting rules

Consecutive User-agent lines before the first Allow/Disallow share one group:

User-agent: ExampleBot
User-agent: OtherBot
Disallow: /lab/

RFC 9309 describes how matching groups are selected and how rules combine. Do not assume “the first matching group always wins.” Specific named groups typically supersede the wildcard group. Rules that appear outside a group, unknown record names, and legacy parsers may behave differently—prefer clear groups and known field names.

(See the Joshternet note below for how this site’s hub subset differs from full group merging.)

A policy belongs to a particular origin

robots.txt is scoped to one origin: scheme, host, and port. HTTP and HTTPS are different. www.joshua.example and joshua.example are different. Subdomains each need their own file if you want rules there. Nondefault ports are separate origins. A policy on one host does not automatically govern related domains. Migrations often need redirects and correct robots files on each origin that still receives traffic.

HTTP responses and crawler behavior

Successful retrieval

A valid 200 response with a parseable body supplies the policy for that origin.

Redirects

RFC 9309 discusses following redirects (at least five is the documented recommendation). Interpret rules in relation to the authority originally requested—read the RFC section before assuming cross-host redirect semantics.

Unavailable responses

Many 4xx responses mean the robots resource is unavailable; the RFC’s unavailable rules apply. Do not invent a single meaning for every 4xx.

Unreachable responses

Server or network failures, including many 5xx responses, are treated as unreachable under the RFC—again, not identical to “file missing.”

Persistent failures

Unusually long unreachability has additional guidance in the RFC. Commercial crawlers may also apply their own retry policies.

Parsing failures

Use parseable rules. Do not discard an entire file solely because one line is malformed.

Missing robots.txt, 403, 404, and 500 are not interchangeable. Distinguish RFC requirements from particular vendor behavior.

Caching and processing limits

HTTP caching (Cache-Control, ETag, Last-Modified) affects how quickly policy changes propagate. RFC 9309 recommends caching behavior for robots resources. Parsers must process at least the first 500 KiB; larger files may be truncated by consumers. Stale caches and recovery after failures mean a change is not instantly universal. Keep policies concise.

Sitemap directives and other extensions

User-agent: *
Allow: /

Sitemap: https://joshua.example/sitemap.xml

Sitemap: advertises sitemap locations and is widely recognized. It is not part of RFC 9309’s Allow/Disallow access grammar, though many crawlers collect it. Historically deployed records such as Crawl-delay and Host are not standardized in RFC 9309; support varies. Do not claim a crawler implements an extension without current documentation. See XML Sitemaps.

Crawling and indexing are different decisions

Preventing a request is not the same as instructing a search engine not to display a URL.

A blocked URL may still appear as a URL-only result if discovered through external links—engines often cannot read noindex on a page they are not allowed to fetch. To apply meta robots, the crawler generally must be permitted to retrieve the resource.

Robots meta tags

Common directives (consumer support varies; verify current engine docs):

• index / noindex
• follow / nofollow
• none (typically noindex, nofollow)
• nosnippet
• max-snippet, max-image-preview, max-video-preview
• noarchive (where still supported)
• notranslate
• unavailable_after

Crawler-specific tags such as googlebot exist. Directives may be combined; conflict resolution is consumer-specific.

<meta name="robots" content="noindex">

<meta name="robots" content="max-snippet:160">

Do not present obsolete or engine-only directives as universally effective. See Google’s robots meta documentation.

X-Robots-Tag HTTP headers

HTTP response headers can carry the same class of directives for PDFs, images, and other formats:

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex

Configure via the origin server, application middleware, or a CDN. Inspect with:

curl -I https://joshua.example/files/report.pdf

Headers and HTML meta tags are different delivery channels for similar indexing ideas.

Automated access in the age of AI

Automated systems retrieve content for search indexing, monitoring, archiving, research, retrieval-augmented generation, AI-assisted answers, model training, fine-tuning, and agent-driven browsing. Permission to retrieve may differ from preferences about downstream use. Standards work is active; consensus is not finished. Publisher preferences and enforceable technical restrictions are not the same thing.

IETF AI Preferences Working Group

The IETF AI Preferences (AIPREF) working group aims to standardize machine-readable preferences about automated uses of content. Keep three layers distinct:

1. A vocabulary of usage categories and preference values.
2. Attachment mechanisms (how preferences bind to HTTP content or robots policies).
3. Existing robots access rules (Allow/Disallow).

As of Nov. 3, 2026, both working-group documents remain Internet-Drafts, not RFCs:

• draft-ietf-aipref-vocab-08 — vocabulary
• draft-ietf-aipref-attach-05 — HTTP / robots attachment

Drafts may change, expire, or be replaced before standardization.

AIPREF vocabulary

Per the current vocabulary draft, categories include:

• train-ai — AI training preferences
• ai-use — AI use (broader automated AI processing as defined in the draft)
• search — search-related processing as defined in the draft

Preference values:

• y — preference to allow
• n — preference to disallow
• Omitted or unrecognized — preference unknown

Absence of a preference is not explicit permission or prohibition.

Draft-labeled example (not a deployment requirement):

train-ai=n, ai-use=y, search=y

The draft uses HTTP Structured Fields concepts. Expressing a preference is not the same as enforcing it.

The proposed Content-Usage mechanism

Status: Internet-Draft (draft-ietf-aipref-attach) — proposed, subject to change.

HTTP response header (draft syntax):

Content-Usage: train-ai=n

The header applies to the representation described by that response.

robots.txt record (draft pattern):

User-agent: *
Allow: /
Content-Usage: train-ai=n

Ordinary crawl permission remains separate from content-usage preferences. Path-specific preferences, group selection, retrieval-time binding, and non-retroactive updates are draft topics—read the current attachment document. Do not treat this as identical to Content Signals or the training policy-token proposal.

AI training policy-token proposal

Status: Individual Internet-Draft (not a WG consensus document) — draft-kelley-aipref-training-preference-signal-00, published Oct. 30, 2026 (Kevin Kelley / Anthropic).

This proposal uses an ordinary robots user-agent group keyed by a special policy token. Allow/Disallow are interpreted as preferences about AI training, not ordinary crawler access:

User-agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

It introduces no new directive names; it reuses path matching. That differs from the working-group Content-Usage approach. Do not combine these syntaxes into one invented standard.

Content Signals

Status: Vendor/community mechanism — contentsignals.org, with documented Cloudflare support via managed robots / Bot Preference Sync. Not an IETF RFC and not the same as Content-Usage.

Categories:

• search — building a search index and ordinary search results (not AI-generated summaries, per their definition)
• ai-input — inputting content into AI models (e.g. RAG / grounding)
• ai-train — training or fine-tuning AI models

Values are typically yes or no. Omission neither grants nor restricts that category via Content Signals.

User-agent: *
Allow: /
Content-Signal: search=yes, ai-input=no, ai-train=no

This expresses a preference. It is not a universal crawler instruction. Cloudflare products may inject or manage these lines; that does not mean every crawler on the web honors them.

Compare the AI preference approaches

Syntax, scope, semantics, and support differ. They are not all “robots standards.” Some publishers publish more than one mechanism during transitions; avoid contradictory declarations.

AI crawler user agents

Operators may run separate crawlers for search, training, and user-initiated browsing. Product tokens change—verify against current official documentation before writing named groups. A claimed User-Agent string does not prove identity; use provider verification when you need assurance. Independent indexers (including Joshternet’s JoshBot) publish their own tokens.

Examples change often; treat any short list as illustrative, not exhaustive. Check each operator’s current crawler docs before relying on a token.

Other ways websites communicate with automated agents

Independent proposals include llms.txt, agents.txt, AGENTS.md, and various well-known capability manifests. Purposes and maturity differ. Instructions for coding agents are not a website-wide crawl policy. Guidance for interpreting content is not authorization to fetch or train. None of these replace RFC 9309.

What these instructions do and do not authorize

A robots directive is not a copyright license. Absence of a restriction does not waive copyright or other obligations. Creative Commons and similar licenses are separate. Legal enforceability varies by jurisdiction; this handbook does not give legal advice.

Robots policies and website security

Do not use robots.txt to protect private resources. Authentication, authorization, and server-side controls protect admin interfaces, drafts, personal documents, sensitive APIs, and credential-bearing URLs. Listing a path in robots.txt can reveal that the path exists. Noncompliant clients and ordinary browsers may still request it.

A crawl restriction is not an HTTP 401/403/404. Serve appropriate status codes and authentication challenges when a resource must stay private. Obscure URLs and Disallow lines are not security.

Complete implementation example

Goals

1. Allow ordinary discovery of public pages.
2. Advertise the XML sitemap.
3. Restrict utility crawl paths.
4. Permit selected public demos under a private tree.
5. Keep designated pages unindexed where supported.
6. Optionally express AI training preferences with a labeled proposed mechanism.
7. Keep standardized crawl rules separate from draft preferences.

RFC 9309-oriented robots.txt

User-agent: *
Allow: /
Disallow: /drafts/
Disallow: /projects/private/
Allow: /projects/private/public-demo/

Sitemap: https://joshua.example/sitemap.xml

Indexing controls (separate from robots.txt)

<meta name="robots" content="noindex">

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex

Draft IETF Content-Usage example (proposed)

User-agent: *
Allow: /
Content-Usage: train-ai=n

Content Signals example (vendor/community)

User-agent: *
Allow: /
Content-Signal: search=yes, ai-input=no, ai-train=no

Do not concatenate contradictory policies into one production file without understanding how each consumer reads them.

Static hosting and CMS implementation

Handwritten sites can ship a single robots.txt in the site root. Jekyll, Hugo, and Eleventy can emit static files from static/ or public/. WordPress and custom apps may generate rules from templates. Reverse proxies and CDNs can override the origin file—verify the edge response. Staging environments sometimes use Disallow: / for the whole site; never deploy that template to production by mistake. Always curl the live URL.

Validate robots.txt

1. Accessibility — curl -i the published URL.
2. Encoding and type — UTF-8 / plain text as expected.
3. Syntax — groups, directive names, paths, comments.
4. Precedence — sample paths and user agents.
5. Redirects and errors — confirm expected behavior for missing or failing robots resources.
6. Indexing — inspect meta and X-Robots-Tag separately.
7. AI preference syntax — validate against the exact draft or vendor doc version you copied.
8. Production — CDN/proxy must not silently replace your file.

Vendor testing tools help one consumer; they do not guarantee every crawler.

Worked policy scenarios

Personal blog

Allow articles and feeds; Disallow /drafts/ and build-only paths. Verify with curl and a path checklist.

Photography portfolio

Allow galleries and project pages for discovery. Express AI training preference only with a labeled draft or Content Signals block if you choose—do not assume universal enforcement.

Software project

Allow /docs/ and release notes; Disallow staging mirrors and internal tooling paths on that origin.

Multidomain website

Separate robots.txt per origin; do not assume one file covers every hostname.

Public blog with noindex pages

Keep thank-you or thin utility pages crawlable if you need engines to see noindex; block only paths that should never be fetched.

Website migration

Maintain robots and redirects on both old and new origins until traffic settles.

AI usage preferences

State the same intention three ways only for comparison—RFC crawl rules, draft Content-Usage, and Content Signals—without asserting identical support.

Common mistakes

Robots and the Joshternet

A valid Joshternet declaration (/.well-known/josh) and crawler-access permission are separate decisions. You can participate while restricting what automated clients may fetch.

JoshBot identifies as:

Joshternet-Joshbot (+https://joshternet.org/joshbot)

Product token: Joshternet-Joshbot. The hub loads each origin’s /robots.txt, treats missing or unreadable files as allow, prefers a named group over *, and applies longest prefix Allow/Disallow (equal length prefers Allow). It uses the first matching named group and does not implement full RFC path wildcards (* / $), nor does it interpret Content-Usage, Content-Signal, or __usage_ai-training. Details: JoshBot and Reference.

Continue through the handbook

• Semantic HTML
• SEO fundamentals
• XML Sitemaps
• Feeds
• Links and relations
• Creative media
• Validation
• Reference
• JoshBot

Handbook navigation below follows content_nav (previous: Sitemaps; next: Links and relations).

Specifications and references

Internet-Drafts may change, expire, or be replaced. Verify status before relying on draft syntax in production.