Index of every page on this site: llms.txt
All skills
huggingface avatar

/huggingface-datasets

@5291c2b official
by Hugging Facehuggingface/skills11k stars
797
Run it yourself

Use this skill for Hugging Face Dataset Viewer API workflows that fetch subset/split metadata, paginate rows, search text, apply filters, download parquet URLs, and read size or statistics.

Use this Skill: https://skilld.dev/gh/huggingface/skills/huggingface-datasets

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈53 tokens for metadata: the name and description. ≈1.1k when used: this file.

Description uses 2.6% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈212 characters, or 2.6%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Hugging Face Dataset Viewer

Use this skill to execute read-only Dataset Viewer API calls for dataset exploration and extraction.

Core workflow

  1. Optionally validate dataset availability with /is-valid.
  2. Resolve config + split with /splits.
  3. Preview with /first-rows.
  4. Paginate content with /rows using offset and length (max 100).
  5. Use /search for text matching and /filter for row predicates.
  6. Retrieve parquet links via /parquet and totals/metadata via /size and /statistics.

Defaults

  • Base URL: https://datasets-server.huggingface.co
  • Default API method: GET
  • Query params should be URL-encoded.
  • offset is 0-based.
  • length max is usually 100 for row-like endpoints.
  • Gated/private datasets require Authorization: Bearer <HF_TOKEN>.

Dataset Viewer

  • Validate dataset: /is-valid?dataset=<namespace/repo>
  • List subsets and splits: /splits?dataset=<namespace/repo>
  • Preview first rows: /first-rows?dataset=<namespace/repo>&config=<config>&split=<split>
  • Paginate rows: /rows?dataset=<namespace/repo>&config=<config>&split=<split>&offset=<int>&length=<int>
  • Search text: /search?dataset=<namespace/repo>&config=<config>&split=<split>&query=<text>&offset=<int>&length=<int>
  • Filter with predicates: /filter?dataset=<namespace/repo>&config=<config>&split=<split>&where=<predicate>&orderby=<sort>&offset=<int>&length=<int>
  • List parquet shards: /parquet?dataset=<namespace/repo>
  • Get size totals: /size?dataset=<namespace/repo>
  • Get column statistics: /statistics?dataset=<namespace/repo>&config=<config>&split=<split>
  • Get Croissant metadata (if available): /croissant?dataset=<namespace/repo>

Pagination pattern:

curl "https://datasets-server.huggingface.co/rows?dataset=stanfordnlp/imdb&config=plain_text&split=train&offset=0&length=100"
curl "https://datasets-server.huggingface.co/rows?dataset=stanfordnlp/imdb&config=plain_text&split=train&offset=100&length=100"

When pagination is partial, use response fields such as num_rows_total, num_rows_per_page, and partial to drive continuation logic.

Search/filter notes:

  • /search matches string columns (full-text style behavior is internal to the API).
  • /filter requires predicate syntax in where and optional sort in orderby.
  • Keep filtering and searches read-only and side-effect free.

For CLI-based parquet URL discovery or SQL, use the hf-cli skill with hf datasets parquet and hf datasets sql.

Creating and Uploading Datasets

Use one of these flows depending on dependency constraints.

Zero local dependencies (Hub UI):

  • Create dataset repo in browser: https://huggingface.co/new-dataset
  • Upload parquet files in the repo "Files and versions" page.
  • Verify shards appear in Dataset Viewer:
curl -s "https://datasets-server.huggingface.co/parquet?dataset=<namespace>/<repo>"

Low dependency CLI flow (npx @huggingface/hub / hfjs):

  • Set auth token:
export HF_TOKEN=<your_hf_token>
  • Upload parquet folder to a dataset repo (auto-creates repo if missing):
npx -y @huggingface/hub upload datasets/<namespace>/<repo> ./local/parquet-folder data
  • Upload as private repo on creation:
npx -y @huggingface/hub upload datasets/<namespace>/<repo> ./local/parquet-folder data --private

After upload, call /parquet to discover <config>/<split>/<shard> values for querying with @~parquet.

Agent Traces

The Hub supports raw agent session traces from Claude Code, Codex, and Pi Agent. Upload them to Hugging Face Datasets as original JSONL files and the Hub can auto-detect the trace format, tag the dataset as Traces, and enable the trace viewer for browsing sessions, turns, tool calls, and model responses. Common local session directories:

  • Claude Code: ~/.claude/projects
  • Codex: ~/.codex/sessions
  • Pi: ~/.pi/agent/sessions

Default to private dataset repos because traces can contain prompts, file paths, tool outputs, secrets, or PII. Preserve the raw .jsonl files and nest them by project/cwd instead of uploading every session at the dataset root.

hf repos create <namespace>/<repo> --type dataset --private --exist-ok
hf upload <namespace>/<repo> ~/.codex/sessions codex/<project-or-cwd> --type dataset

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

1 warning2mo4 checks · Risk SAFE
  • Gen Agent Trust Hub2mo

    This skill facilitates integration with the Hugging Face Dataset Viewer API and repository upload workflows. It includes potential considerations such as running command-line utilities to upload agent session logs and ingesting external dataset content. While these functions support legitimate workflows, appropriate data boundaries should be maintained.

  • Socket2mo

    No alerts

  • Snyk2mo

    Risk: MEDIUM · 1 issue

  • ZeroLeaks7mo

    Score: 93/100 · 2 sections analyzed

Signed by skilld at 5291c2b. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 3 days ago.

Activeupdated 7 months ago
  • API
  • huggingface
  • datasets
  • pagination
  • search
  • filtering
  • parquet
  • metadata
  • data-exploration

README badge

README badge for huggingface/skills/huggingface-datasets

Fetches dataset metadata, rows, and statistics from Hugging Face Dataset Viewer API without modifying data. Supports pagination, text search, filtering, parquet URLs, and column statistics for exploration and extraction workflows.

Generated from the current SKILL.md.

Does this skill support write operations like creating or modifying datasets?
No. This skill is read-only and covers Dataset Viewer API calls for exploration and extraction. Dataset creation and upload are handled separately via the Hub UI or the `hf-cli` skill.
How do I access gated or private datasets?
Pass an `Authorization: Bearer <HF_TOKEN>` header with your Hugging Face API token in requests to gated or private datasets.
What is the maximum number of rows I can fetch in a single request?
The `length` parameter maxes out at 100 rows per request. Use `offset` and `length` together to paginate through larger datasets.
Can I use SQL or CLI commands to explore datasets with this skill?
No. For CLI-based parquet discovery or SQL queries, use the separate `hf-cli` skill with `hf datasets parquet` and `hf datasets sql` commands.
Does this skill support filtering and searching?
Yes. Use `/search` for text matching on string columns and `/filter` with predicate syntax for row-level filtering, both read-only operations.

Generated from the current SKILL.md. These answers refresh after source changes.