Index of every page on this site: llms.txt
All skills
Run it yourself

Use when working with structured data files (CSV, JSON, YAML, TOML, Parquet) — querying, transforming, filtering, aggregating, or converting between formats

  • 1 file
  • 5.1 KB
  • MIT
  • Updated 6 months ago
  • GitHub
Use this Skill: https://skilld.dev/gh/jeremylongshore/tons-of-skills-marketplace/data-processing

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈44 tokens for metadata: the name and description. ≈1.2k when used: this file.

Description uses 2.2% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈174 characters, or 2.2%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Data Processing

When to Use

  • Querying, filtering, or transforming JSON files
  • Reading or converting YAML, TOML, or XML config files
  • Analyzing, aggregating, or joining CSV/TSV/Parquet files
  • Running SQL queries against local data files without a database server
  • Converting between data formats (JSON to YAML, CSV to JSON, etc.)
  • Exploring deeply nested JSON structures

Tools

Tool Purpose Structured output
jq Query and transform JSON Native JSON
yq Query and transform YAML, TOML, XML -o json for JSON output
gron Flatten JSON into greppable path.to.key = value lines --ungron to reverse back to JSON
miller (mlr) Transform CSV/JSON/TSV records with awk-like verbs --json for JSON output
xsv Fast CSV slicing, searching, joining, statistics CSV native (pipe to xsv table for display)
DuckDB SQL queries on CSV, JSON, Parquet files -json flag for JSON output

Patterns

JSON: Filter array elements by field value

jq '.items[] | select(.status == "active")' data.json

JSON: Extract specific fields from array

jq '[.users[] | {name: .name, email: .email}]' data.json

JSON: Count items grouped by field

jq '[.events[] | .type] | group_by(.) | map({type: .[0], count: length})' data.json

YAML: Read a nested value

yq '.spec.containers[0].image' deployment.yaml

YAML: Convert entire file to JSON

yq -o json config.yaml

TOML: Read a value

yq -p toml '.database.host' config.toml

JSON: Explore unknown structure by grepping paths

gron data.json | grep -i "error"

CSV: Column statistics (min, max, mean, stddev)

xsv stats data.csv | xsv table

CSV: Search rows matching a pattern in a column

xsv search -s status "active" data.csv

CSV: Select specific columns

xsv select name,email,created_at users.csv

CSV: Sort by column

xsv sort -s revenue -R sales.csv

SQL: Query a CSV file

duckdb -c "SELECT department, COUNT(*) as cnt, AVG(salary) as avg_sal FROM 'employees.csv' GROUP BY department ORDER BY avg_sal DESC"

SQL: Query a JSON file

duckdb -c "SELECT * FROM read_json_auto('events.json') WHERE type = 'error' LIMIT 20"

SQL: Query Parquet files

duckdb -c "SELECT * FROM 'data/*.parquet' WHERE created_at > '2026-01-01'"

CSV/JSON: Transform records with miller

mlr --csv filter '$revenue > 1000' then sort-by -nr revenue sales.csv

Format conversion: CSV to JSON

mlr --icsv --ojson cat data.csv

Pipelines

YAML config → JSON → SQL query

yq -o json config.yaml | duckdb -c "SELECT key, value FROM read_json_auto('/dev/stdin') WHERE env = 'production'"

Each stage: yq converts YAML to JSON, DuckDB runs SQL on the JSON stream.

Grep nested JSON paths → reconstruct matching subset

gron large.json | grep "\.errors\[" | gron --ungron

Each stage: gron flattens JSON to paths, grep filters, ungron reconstructs valid JSON from matches.

CSV filter → aggregate with SQL

xsv search -s region "EU" sales.csv | duckdb -c "SELECT product, SUM(revenue) as total FROM read_csv_auto('/dev/stdin') GROUP BY product ORDER BY total DESC"

Each stage: xsv filters rows by region, DuckDB aggregates the filtered stream.

Join two CSV files with SQL

duckdb -c "SELECT u.name, u.email, o.total, o.date FROM 'users.csv' u JOIN 'orders.csv' o ON u.id = o.user_id ORDER BY o.date DESC"

Multi-format pipeline: JSON → CSV → stats

jq -r '.records[] | [.name, .score] | @csv' data.json | xsv stats

Each stage: jq extracts fields to CSV format, xsv computes statistics.

Prefer Over

  • Prefer DuckDB over Python/pandas for ad-hoc SQL queries on files — single command, no script, handles large files
  • Prefer jq over Python json module for one-off JSON transforms — single pipeline vs. multi-line script
  • Prefer xsv over awk/cut for CSV operations — correct CSV parsing, handles quoted fields and escapes
  • Prefer miller over awk for format-aware record transformations — understands CSV/JSON headers natively
  • Prefer yq over custom parsers for config file reads — handles YAML, TOML, XML with consistent syntax

Do NOT Use When

  • Data is already in a running database — query the database directly
  • File is under 10 lines — just use the Read tool and process in-context
  • Task requires complex multi-step logic with conditionals — write a Python script instead
  • JSON is simple enough to read by eye — use the Read tool, don't over-engineer
  • Working with binary or non-structured data formats

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No third-party reports yet.

Signed by skilld at 820dfc8. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 6 months ago
What it can do
Runs commands Reads files
version
1.0.0
author
ykotik
All 9 allowed tools
Bash(jq*)Bash(yq*)Bash(gron*)Bash(mlr*)Bash(xsv*)Bash(duckdb*)ReadGlobGrep

README badge

README badge for jeremylongshore/tons-of-skills-marketplace/data-processing