Index of every page on this site: llms.txt
All skills
tavily-ai avatar

/tavily-crawl

@778122e official
by Tavily AItavily-ai/skills488 stars
42
Run it yourself

Crawl websites and extract content from multiple pages via the Tavily CLI. Use this skill when the user wants to crawl a site, download documentation, extract an entire docs section, bulk-extract pages, save a site as local markdown files, or says "crawl", "get all the pages", "download the docs", "extract everything under /docs", "bulk extract", or needs content from many pages on the same domain. Supports depth/breadth control, path filtering, semantic instructions, and saving each page as a local markdown file.

Use this Skill: https://skilld.dev/gh/tavily-ai/skills/tavily-crawl

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈133 tokens for metadata: the name and description. ≈969 when used: this file.

Description uses 6.7% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈534 characters, or 6.7%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

tavily crawl

Crawl a website and extract content from multiple pages. Supports saving each page as a local markdown file.

Before running

Crawl requires authentication. Run the requested command directly when tvly is already authenticated; do not add a status check to every invocation.

If tvly is missing, follow the tavily-cli setup. If an installed CLI reports an authentication error, use tvly login for authentication only, or tvly init --skip-skills when guided verification is also useful. Browser-based OAuth is preferred when an interactive user can complete it. --no-browser prints the sign-in link instead of opening it, but still waits for a localhost callback. In an unattended agent or CI environment, leave authentication to the user or use a securely provided TAVILY_API_KEY. Do not start a second login immediately after guided setup has completed.

When to use

  • You need content from many pages on a site (e.g., all /docs/)
  • You want to download documentation for offline use
  • Step 4 in the workflow: search → extract → map → crawl → research

Quick start

# Basic crawl
tvly crawl "https://docs.example.com" --json

# Save each page as a markdown file
tvly crawl "https://docs.example.com" --output-dir ./docs/

# Deeper crawl with limits
tvly crawl "https://docs.example.com" --max-depth 2 --limit 50 --json

# Filter to specific paths
tvly crawl "https://example.com" --select-paths "/api/.*,/guides/.*" --exclude-paths "/blog/.*" --json

# Semantic focus (returns relevant chunks, not full pages)
tvly crawl "https://docs.example.com" --instructions "Find authentication docs" --chunks-per-source 3 --json

Options

Option Description
--max-depth Levels deep (1-5, default: 1)
--max-breadth Links per page (default: 20)
--limit Total pages cap (default: 50)
--instructions Natural language guidance for semantic focus
--chunks-per-source Chunks per page (1-5, requires --instructions)
--extract-depth basic (default) or advanced
--format markdown (default) or text
--select-paths Comma-separated regex patterns to include
--exclude-paths Comma-separated regex patterns to exclude
--select-domains Comma-separated regex for domains to include
--exclude-domains Comma-separated regex for domains to exclude
--allow-external / --no-external Include external links (default: allow)
--include-images Include images
--timeout Max wait (10-150 seconds)
-o, --output Save JSON output to file
--output-dir Save each page as a .md file in directory
--json Structured JSON output

Crawl for context vs. data collection

For agentic use (feeding results to an LLM):

Always use --instructions + --chunks-per-source. Returns only relevant chunks instead of full pages — prevents context explosion.

tvly crawl "https://docs.example.com" --instructions "API authentication" --chunks-per-source 3 --json

For data collection (saving to files):

Use --output-dir without --chunks-per-source to get full pages as markdown files.

tvly crawl "https://docs.example.com" --max-depth 2 --output-dir ./docs/

Tips

  • Start conservative — --max-depth 1, --limit 20 — and scale up.
  • Use --select-paths to focus on the section you need.
  • Use map first to understand site structure before a full crawl.
  • Always set --limit to prevent runaway crawls.

See also

Source: SKILL.md on GitHub

Skill behaviors

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

Third-party checks

1 warning1mo4 checks · Risk SAFE
  • Gen Agent Trust Hub1mo

    This skill enables the agent to crawl websites and extract content using the Tavily CLI. It is a legitimate tool for web-based research and data collection. As with any web-crawling tool, the agent will process untrusted content from external sources.

  • Socket1mo

    No alerts

  • Snyk1mo

    Risk: MEDIUM · 1 issue

  • ZeroLeaks6mo

    1 finding · Score: 82/100

Provenance

Signed by skilld at 778122e. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub last week.

Steadyupdated last month

Capability

What it can do
Runs commands
All 1 allowed tools
Bash(tvly *)

Topics

  • Documentation
  • CLI
  • tavily
  • web-crawling
  • content-extraction
  • markdown
  • scraping
  • semantic-search

README badge

README badge for tavily-ai/skills/tavily-crawl

What it does

Crawl multiple pages from a website and extract content as markdown files or JSON, with options for depth limits, path filtering, and semantic instructions. Targets documentation downloads, bulk page extraction, and multi-page site research via the Tavily CLI.

Generated from the current SKILL.md.

Frequently asked

Does this crawl external links or just the same domain?
By default it allows external links, but you can restrict it to the starting domain using `--exclude-domains` or disable external links entirely with `--no-external`.
How do I prevent the crawl from getting too large?
Use `--max-depth`, `--max-breadth`, and `--limit` to control scope. Start conservative (depth 1, limit 20) and scale up. Always set `--limit` to prevent runaway crawls.
Should I use `--chunks-per-source` or save full pages with `--output-dir`?
Use `--instructions` + `--chunks-per-source` when feeding results to an LLM (returns only relevant chunks). Use `--output-dir` when saving for offline use or data collection (returns full pages as markdown files).
Can I focus the crawl to specific URL paths?
Yes, use `--select-paths` and `--exclude-paths` with comma-separated regex patterns, e.g. `--select-paths '/api/.*,/guides/.*'`.
What's the difference between this and the extract skill?
Crawl handles multiple pages on a site with depth/breadth control and path filtering. Extract is for individual pages. Map is for discovering URLs before deciding to crawl.

Generated from the current SKILL.md. These answers refresh after source changes.