Index of every page on this site: llms.txt
All skills
bydradev avatar

/public-site-indexing

@20b39bd
Run it yourself

Use before adding or changing `robots.ts`, `robots.txt`, `sitemap.ts` or the sitemap, a page's metadata `robots` setting or `noindex`, the site's hostname or canonical URL, or its AI crawler policy in this Next.js repository. Covers what a public site serves to crawlers, which pages stay out of the index and the sitemap, where AI crawler policy lives, and keeping default and per-deployment hostnames out of search results.

  • 1 file
  • 1.5 KB
  • Updated last month
  • GitHub
Use this Skill: https://skilld.dev/gh/bydradev/gh-repo-bootstrapper/public-site-indexing

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈112 tokens for metadata: the name and description. ≈265 when used: this file.

Description uses 5.6% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈448 characters, or 5.6%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Public-site indexing

These rules apply to a site meant to be found in search. If ## Project specifics in AGENTS.md names a repository-owned skill for indexing or search, or says the site is not public, that governs.

A public site serves robots.txt and a sitemap from the App Router metadata files robots.ts and sitemap.ts. Allow all crawlers, apart from the AI crawler policy below, and link the sitemap; disallow only paths with a reason, such as API routes that are not pages. Give personal or search-result pages robots: { index: false, follow: true } in their metadata, and leave them out of the sitemap.

Keep AI crawler policy in one place: the hosting platform's bot controls when it has them, otherwise per-bot rules in robots.ts.

When the host also serves the site on a default or per-deployment hostname, turn those hostnames off and send X-Robots-Tag: noindex, nofollow on them as a backstop, so only the canonical domain is indexed. For a static export, make the CI export check require out/robots.txt and out/sitemap.xml.

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No third-party reports yet.

Signed by skilld at 20b39bd. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated last month

README badge

README badge for bydradev/gh-repo-bootstrapper/public-site-indexing