Public-site indexing
These rules apply to a site meant to be found in search. If
## Project specifics in AGENTS.md names a repository-owned skill for
indexing or search, or says the site is not public, that governs.
A public site serves robots.txt and a sitemap from the App Router metadata
files robots.ts and sitemap.ts. Allow all crawlers, apart from the AI
crawler policy below, and link the sitemap; disallow only paths with a reason,
such as API routes that are not pages. Give personal or search-result pages
robots: { index: false, follow: true } in their metadata, and leave them out
of the sitemap.
Keep AI crawler policy in one place: the hosting platform's bot controls when
it has them, otherwise per-bot rules in robots.ts.
When the host also serves the site on a default or per-deployment hostname,
turn those hostnames off and send X-Robots-Tag: noindex, nofollow on them as
a backstop, so only the canonical domain is indexed. For a static export, make
the CI export check require out/robots.txt and out/sitemap.xml.