skilld
All skills
firecrawl avatar

/firecrawl-crawl

@509dd9c official
by firecrawlfirecrawl/cli643 stars
110

Bulk-extract many pages from one site or section. Use for "crawl", "everything under /docs", or content spanning linked pages.

Ask your Agent

Use this Skill: https://skilld.dev/gh/firecrawl/cli/firecrawl-crawl

This session only. Nothing lands on disk.

SKILL.md

≈36 tokens always: the name and description. ≈409 when used: this file.

firecrawl crawl

Bulk extract content from a website. Crawls pages following links up to a depth/limit.

Prerequisite: crawl requires authentication (no keyless free tier); without credentials the CLI prompts an interactive login.

Quick start

# Crawl a docs section
firecrawl crawl "<url>" --include-paths /docs --limit 50 --wait -o .firecrawl/crawl.json

# Full crawl with depth limit
firecrawl crawl "<url>" --max-depth 3 --wait --progress -o .firecrawl/crawl.json

# Check status of a running crawl
firecrawl crawl <job-id>

Run firecrawl crawl --help for the full option list.

Done when: the crawl reaches a terminal status and the saved output under .firecrawl/ contains the expected pages.

Tips

  • Use --wait when you need the results immediately. It has no default timeout; use --timeout <seconds> to bound polling. Without --wait, crawl returns a job ID for async polling.
  • Scope crawls with --include-paths whenever the request names a section — crawl only the pages you need.
  • Crawl consumes credits per page. Check firecrawl credit-usage before large crawls (credit-usage requires authentication).

See also

Source: SKILL.md on GitHub

Third-party checks

1 warning16d5 checks · Risk SAFE
  • Gen Agent Trust Hub16d

    The skill provides standard documentation and instructions for bulk-extracting web data using the Firecrawl CLI. It introduces a potential indirect prompt injection surface through the ingestion of untrusted web content.

  • Socket16d

    No alerts

  • Snyk16d

    Risk: MEDIUM · 1 issue

  • Runlayer6mo

    1 file scanned · No issues

  • ZeroLeaks5mo

    Score: 93/100 · 2 sections analyzed

Provenance

Signed by skilld at 509dd9c. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 13 hours ago.

Activeupdated last month

Capability

What it can do
Runs commands
All 2 allowed tools
Bash(firecrawl *)Bash(npx firecrawl-cli *)

Topics

README badge

README badge for firecrawl/cli/firecrawl-crawl

What it does

Crawls a website following links up to a specified depth or page limit, extracting content from each page. Use this when you need to bulk-extract all pages from a docs section, site directory, or discover and scrape multiple linked pages at once. Supports path filtering, concurrency control, and async job polling.

Generated from the current SKILL.md.

Frequently asked

Does firecrawl crawl work asynchronously?
Yes. By default, crawl returns a job ID immediately. Use --wait to block until completion; without it, you get back a job ID that you can check later with firecrawl crawl <job-id>.
How do I limit a crawl to a specific section like /docs?
Use --include-paths /docs to crawl only URLs matching that path prefix. Combine with --limit to cap the total pages crawled.
What's the difference between crawl and scrape?
Scrape extracts a single page. Crawl follows links across multiple pages up to a depth or limit, making it suitable for extracting entire site sections.
Does crawl consume credits?
Yes, crawl consumes credits per page extracted. Check firecrawl credit-usage before running large crawls.
Can I control how fast the crawl runs?
Yes. Use --delay <ms> to add delay between requests and --max-concurrency <n> to limit parallel workers.

Generated from the current SKILL.md. These answers refresh after source changes.