Index of every page on this site: llms.txt
All skills
k-dense-ai avatar

/datalad

@ec59a29
Run it yourself

Retrieves, versions, and publishes scientific datasets with DataLad and git-annex, and captures computational provenance with datalad run, rerun, and containers-run. Use when cloning or fetching data from OpenNeuro, DANDI, datasets.datalad.org, or any DataLad dataset; when a file in a dataset reads as a broken symlink or a small pointer instead of real data; when an analysis needs a machine-readable record of how each output was produced so it can be re-executed; or when publishing a dataset to siblings such as a GitHub repository plus a storage remote. Also use to decide between DataLad and plain Git for a data-carrying repository.

Use this Skill: https://skilld.dev/gh/k-dense-ai/scientific-agent-skills/datalad

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesprovenance.md

≈3.5k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Computational provenance with DataLad

Why this is the interesting half

Version control tells you that a file changed. Provenance tells you what produced it, from what, and with which software. datalad run records the command and tracked inputs/outputs in the same commit as the change. Capturing software versions additionally requires a versioned environment specification or a tracked container; an executable name alone does not identify its installed build. datalad rerun then reads that record back and re-executes it, so "is this reproducible" becomes a command rather than an argument.

Principles: STAMPED, and the YODA layout it grew out of

YODA ("YODAs Organigram on Data Analysis") is the convention DataLad analyses were originally built on, and it is still what datalad create -c yoda configures. Its three principles (one thing one dataset, record where the data came from, record what was done to it) describe what a well-formed analysis looks like, but they were inspirational rather than operational: there is nothing in them to check a dataset against.

STAMPED (https://stamped-principles.org) is the operationalized successor, formalized by some of the original YODA authors. It states seven properties of a reproducible research object, each backed by normative MUST/SHOULD/MAY requirements:

Property Core requirement
Self-contained Everything essential to replicate the computation is reachable within a single top-level research object
Tracked Persistent content identification and provenance are recorded for every component and every modification, including the versions involved
Actionable The object carries enough instruction to reproduce all results, specified as executable specifications rather than prose
Modular Components are organized as independently versioned modules, included directly or linked as subdatasets
Portable Procedures depend on no undocumented host state; environments are explicitly specified and version controlled
Ephemeral Results are produced in disposable environments built only from the object's own contents
Distributable Every referenced module and component is persistently retrievable by others

Actionable is the property datalad run exists to satisfy. A README describing how a figure was produced is documentation; a run record is an executable specification, and datalad rerun is what makes it executable. The same commit satisfies Tracked, because the command, its inputs, its outputs, and the versions they were taken at are recorded next to the change rather than in a separate log that can drift away from it. Modular maps onto subdatasets, Portable onto tracked images and runtime specifications, Ephemeral onto disposable execution environments, and Distributable onto siblings and RIA stores (see publishing.md).

Two companion resources make this checkable rather than aspirational: https://checklist.stamped-principles.org walks the requirements as a MUST/SHOULD/MAY checklist, and https://examples.stamped-principles.org collects worked patterns, including ones built directly on datalad run and datalad rerun.

The YODA layout in practice

Apply the layout at creation time:

datalad create -c yoda "my_analysis"

That produces:

.
├── .gitattributes
├── CHANGELOG.md
├── code
│   ├── .gitattributes
│   └── README.md
└── README.md

The configuration matters more than the directories. Everything in code/, plus README.md and CHANGELOG.md, is tracked by Git rather than git-annex, so scripts stay directly readable and diffable in a clone that has fetched no data at all. Input data is then added as a subdataset:

datalad clone -d . https://github.com/OpenNeuroDatasets/ds000001.git inputs/raw

The -d . is what registers the clone as a subdataset of the analysis rather than leaving an unrelated repository sitting inside it.

datalad run

datalad run [-h] [-d DATASET] [-i PATH] [-o PATH] [--expand {inputs|outputs|both}]
    [--assume-ready {inputs|outputs|both}] [--explicit] [-m MESSAGE]
    [--sidecar {yes|no}] [--dry-run {basic|command}]
    [--on-cmd-failure {error|save|all}] [-J NJOBS]
    [--version] ...
Option Documented behaviour Practical consequence
-i/--input PATH "A dependency for the run. Before running the command, the content for this relative path will be retrieved." The command does not fail on an unfetched pointer, and rerun knows what to fetch elsewhere
-o/--output PATH "Prepare this relative path to be an output file of the command." Unlocks or removes the target so git-annex write protection does not block the write
--explicit "Consider the specification of inputs and outputs to be explicit. Don't warn if the repository is dirty." Lets a run proceed in a dirty dataset, and saves only the declared outputs
--assume-ready {inputs|outputs|both} "Assume that inputs do not need to be retrieved and/or outputs do not need to unlocked or removed." Skips preparation for speed; only safe when you have already done it
--expand {inputs|outputs|both} "Expand globs when storing inputs and/or outputs in the commit message." Records the concrete file list rather than the glob, which is what you want when the glob's meaning could change
--dry-run {basic|command} "Do not run the command; just display details about the command execution." Check placeholder expansion before committing anything
--sidecar {yes|no} Store the run record in a separate file rather than in the commit message Keeps long records out of git log output

Placeholders available in the command string: {pwd} (current working directory), {dspath} (dataset path), {tmpdir} (a temporary directory), {inputs} and {outputs} (the values of the corresponding flags), and {inputs[0]} for indexed access.

Globs are permitted in --input and --output, and multiple flags may be given:

datalad run -m "second-level model" \
  -i "derivatives/sub-*/func/*_bold.nii.gz" \
  -i "code/model.py" \
  -o "results/group_map.nii.gz" \
  --expand inputs \
  "python code/model.py {outputs}"

The run record

The commit message carries a machine-readable JSON block between the markers === Do not change lines below === and ^^^ Do not change lines above ^^^. It records the command, the dataset ID, the exit status, and the input and output specifications. The handbook is explicit that this section is "less for the human user" and exists "for DataLad, in particular for the datalad rerun command". Editing it by hand, including during an interactive rebase, breaks rerun silently.

Two behaviours that surprise people:

  • A run producing no change to the dataset produces no commit at all, exactly as a datalad save with nothing to save does. An empty history entry is not evidence the run failed to execute, only that it changed nothing.
  • datalad run refuses to start in a dirty dataset. This is the point of the command: a record built on an unknown starting state does not establish anything. Save or discard first, or state the scope with --explicit.

Failed command provenance

DataLad 1.6.5 exposes --on-cmd-failure error|save|all. The default error stops without saving; save records changed outputs and returns success; all saves changed outputs and preserves the failing exit code. Inspect the run record exit field when evaluating results: a successful DataLad invocation need not mean the scientific command succeeded. On replay, a recorded nonzero code can be accepted when repeated, and a zero exit is accepted even if the original failed. A run producing no tracked changes still has no commit. These options are distinct from global result handling via --on-failure.

datalad rerun

datalad rerun [-h] [--since SINCE] [-d DATASET] [-b NAME] [-m MESSAGE] [--onto base]
    [--script FILE] [--report] [--assume-ready {inputs|outputs|both}] [--explicit]
    [--on-cmd-failure {error|save|all}] [-J NJOBS] [--version] [REVISION]
  • REVISION selects which recorded command to replay and defaults to HEAD.
  • --since SINCE replays a range: "the commands from all commits that are reachable from revision but not SINCE will be re-executed (in other words, the commands in git log SINCE..REVISION)". This is how a multi-step pipeline is replayed in order.
  • --onto base gives the "start point for rerunning the commands. If not specified, commands are executed at HEAD." Use --onto '' to start at the parent of the first run commit selected for replay. This resets the top-level dataset only; installed subdataset worktrees do not automatically move to their recorded commits.
  • -b/--branch NAME creates and checks out a branch before replaying.
  • --report displays what would be done without executing, which is the safe first call.
  • --script FILE extracts the commands to a file instead of running them, with - for stdout. This is how a DataLad history becomes a plain shell script for a reviewer or a cluster submission.

The reproducibility check worth building into a project:

git tag original-results HEAD
# Replace FIRST_RUN with the first run commit hash; the ^ includes that run.
datalad rerun --report --since FIRST_RUN^ original-results
datalad rerun --onto= -b repro-check --since FIRST_RUN^ original-results
git diff original-results HEAD -- results/

This comparison checks tracked output identities, including annex keys. It does not by itself establish numerical or scientific validity. Use a disposable clone for a fresh-environment check, restore recorded subdataset revisions, and inspect tolerances, randomness, and software versions separately. The FIRST_RUN example is a template; the local regression test substitutes a real commit and checks the regenerated bytes.

Containers

datalad-container (reviewed release 1.2.6) records the software environment alongside the command.

datalad containers-add [-h] [-u URL] [-d DATASET] [--call-fmt FORMAT]
    [-i IMAGE] [--update] [--extra-input FILE] [--version] NAME

Container source options (source-verified, runtime examples illustrative):

  • A local SIF file plus --call-fmt 'apptainer exec {img} {cmd}' avoids assuming a singularity executable exists when only Apptainer is installed.
  • docker://debian:stable-slim builds a Singularity image. In 1.2.6 the released containers-add implementation invokes singularity build and its guessed call format uses singularity exec; it does not automatically substitute apptainer for this path. Build the SIF externally with Apptainer, then register the local file.
  • dhub://debian:stable-slim uses Docker, saves the pulled image into the dataset, and configures a Docker adapter with the current directory mounted at /tmp.
  • shub:// remains recognized by the extension, but its legacy Singularity Hub service was not verified in this review. Prefer a retrievable local image or an actively maintained registry source.

Use an immutable registry digest when building an image, retain the resulting image, and record its architecture. A mutable tag alone does not pin software. Configure required binds/environment/GPU options in the tracked call format and declare overlays with --extra-input. Container tracking alone does not isolate host mounts or ensure cross-platform numerical equivalence.

datalad containers-run [-h] [-n NAME] [-d DATASET] [-i PATH] [-o PATH] [-m MESSAGE]
    [--expand {inputs|outputs|both}] [--explicit] [--sidecar {yes|no}] [--version] ...

-n/--container-name selects "the name of or a path to a known container to use for execution, in case multiple containers are configured". With exactly one container configured it may be omitted. During execution the environment variable DATALAD_CONTAINER_NAME holds the name of the container in use, which is available to the command itself.

The image is tracked in the dataset like any other file, so it is annexed content: a collaborator gets it with datalad get and the provenance record points at a specific image rather than at a tag someone may have re-pushed.

Exporting provenance to a standard form

DataLad's run records are DataLad's own format. Converting them to an interoperable representation is an open area rather than a solved one, and this is worth stating plainly rather than implying a pipeline exists:

Until one of those is settled, the durable artifact is the DataLad history itself plus datalad rerun --script, which produces a plain, reviewable command sequence that does not depend on DataLad to read.

Further reading

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts17d3 checks · Risk SAFE
  • Gen Agent Trust Hub17d

    The skill provides a comprehensive interface for DataLad, a data management tool designed for scientific datasets. It supports dataset discovery, cloning from public repositories, large file management via git-annex, and automated provenance tracking by executing shell commands. All capabilities, including network access and command execution, are documented and essential to the skill's primary function.

  • Socket17d

    No alerts

  • Snyk17d

    Risk: LOW · No issues

Signed by skilld at ec59a29. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 4 days ago.

Activeupdated 2 months ago
All 1 allowed tools
Read Write Edit Bash
Other metadata
compatibility
Requires Python 3.10+, DataLad 1.6.5, Git, and git-annex 10.x. Tested with git-annex 10.20260901 and datalad-container 1.2.6 on macOS ARM64. Containers additionally require Singularity, Apptainer, or Docker. Remote data access needs network access and may need provider credentials. Local filesystem workflows work offline.
metadata
{
  "version": "1.2",
  "last-reviewed": "2026-09-30",
  "skill-author": "Dylan Pulver"
}

README badge

README badge for k-dense-ai/scientific-agent-skills/datalad