---
name: programmatic-eda
description: Systematic exploratory data analysis. Activate when a dataset needs profiling — structure check, nulls, outliers, distributions, correlations — before deeper analysis begins.
title: programmatic-eda
canonical_url: https://skilld.dev/gh/nimrodfisher/data-analytics-skills/programmatic-eda
last_updated: 2026-09-23T20:30:09.000Z
---

> **Skill from skilld.dev.** Follow the instructions below for this session. You do not need to install anything.
>
> Supporting files, fetch one when the Skill refers to it: [assets/eda_report_template.md](https://skilld.dev/api/skills-raw/nimrodfisher/data-analytics-skills/programmatic-eda/assets/eda_report_template.md), [assets/findings_summary.md](https://skilld.dev/api/skills-raw/nimrodfisher/data-analytics-skills/programmatic-eda/assets/findings_summary.md), [references/eda_checklist.md](https://skilld.dev/api/skills-raw/nimrodfisher/data-analytics-skills/programmatic-eda/references/eda_checklist.md), [references/pandas_polars_recipes.md](https://skilld.dev/api/skills-raw/nimrodfisher/data-analytics-skills/programmatic-eda/references/pandas_polars_recipes.md), [references/quality_thresholds.md](https://skilld.dev/api/skills-raw/nimrodfisher/data-analytics-skills/programmatic-eda/references/quality_thresholds.md), [scripts/correlation_explorer.py](https://skilld.dev/api/skills-raw/nimrodfisher/data-analytics-skills/programmatic-eda/scripts/correlation_explorer.py), [scripts/data_overview.py](https://skilld.dev/api/skills-raw/nimrodfisher/data-analytics-skills/programmatic-eda/scripts/data_overview.py), [scripts/distribution_summary.py](https://skilld.dev/api/skills-raw/nimrodfisher/data-analytics-skills/programmatic-eda/scripts/distribution_summary.py), [scripts/null_profiler.py](https://skilld.dev/api/skills-raw/nimrodfisher/data-analytics-skills/programmatic-eda/scripts/null_profiler.py), [scripts/outlier_detector.py](https://skilld.dev/api/skills-raw/nimrodfisher/data-analytics-skills/programmatic-eda/scripts/outlier_detector.py).
>
> If the user asked to install this Skill, run `npx skilld install nimrodfisher/data-analytics-skills/programmatic-eda`. Install writes the Skill files into the project, so every session loads them.

# When to use
- You receive a new dataset and need to understand its shape and quality before analysis
- An analysis produces surprising numbers and you want to verify the underlying data first
- A stakeholder asks "is this data reliable?" or "what's in this table?"
- You're about to run a model or statistical test and need data-quality assurance

# Process
1. **Load and overview** — run `scripts/data_overview.py` to get row count, dtypes, memory usage, and a sample. Confirm grain (what one row represents).
2. **Null profile** — run `scripts/null_profiler.py`; compare output against thresholds in `references/quality_thresholds.md` and flag columns above limits.
3. **Outlier detection** — run `scripts/outlier_detector.py` (IQR + z-score) on numeric columns; document flagged values and decide: real signal or data error?
4. **Distribution summary** — run `scripts/distribution_summary.py` for descriptive stats and univariate histograms on each numeric column.
5. **Correlation exploration** — run `scripts/correlation_explorer.py`; flag pairs with |r| > 0.8 as potential multicollinearity or redundancy.
6. **EDA checklist sign-off** — work through `references/eda_checklist.md` and confirm each item before declaring the dataset profiled.
7. **Write findings** — fill `assets/eda_report_template.md` with full profiling output; distil top issues into `assets/findings_summary.md`.

For pattern recipes (e.g. polars vs pandas equivalents, chunked reads for large files), see `references/pandas_polars_recipes.md`.

# Inputs the skill needs
- Required: dataset path (CSV / Parquet / Excel) or a DataFrame already in scope
- Required: business context — what does one row represent?
- Optional: quality threshold overrides (defaults in `references/quality_thresholds.md`)
- Optional: columns to skip (PII, binary blobs, high-cardinality IDs)

# Output
- `assets/eda_report_template.md` (filled) — full profiling report with per-column stats
- `assets/findings_summary.md` (filled) — top 3–5 quality issues and recommended next steps
- Console output / plots from scripts for interactive inspection