DOCA DPA High-Level Tracer — Tasks

Where to start: The verbs that carry real workflow content are ## install (route to host-side DOCA install + DPA env), ## configure (mode + JSON config + capture window), ## run (the capture → decode → render pipeline), ## test (iterative tuning of mode / window / overhead), ## debug (layered diagnosis), and ## use (hand the captured trace back to the doca-dpa debug loop). The two routing-stub verbs (build, modify) are kept because the agent's task-verb contract is uniform across the bundle, and each carries a meaningful pointer to where the user's question actually belongs.

This file is loaded by SKILL.md after CAPABILITIES.md. It walks the agent through the documented invocations of doca_dpa_hl_tracer, the capture → decode → render workflow, and the hand-off back to the doca-dpa debug ladder.

install

doca_dpa_hl_tracer is shipped pre-built as part of every DOCA install that includes the DOCA DPA Tools optional component, under /opt/mellanox/doca/tools/. The operator-side install path:

Confirm DOCA is installed with the DPA Tools component. If the binary is missing, route to doca-setup ## install to install or repair the host-side DOCA package selection; confirm the version per doca-version TASKS.md ## configure.
Confirm the DPACC compiler is installed and version-matched to the DOCA install. Per the doca-dpa overlay, DPA-side images are built by DPACC and the host-side doca-dpa library is paired with a matching DPACC version; the tracer decodes against ELFs produced by that DPACC.
Confirm a BlueField with a DPA processor is physically present and visible to the host through DOCA. Walk doca-dpa TASKS.md ## configure step 1 to enumerate the device and confirm DPA support. If the BlueField is not visible or the DPA is not exposed, tracing is meaningless.
Confirm a DPA-side application image exists on disk. The tracer decodes against the same ELF the host-side doca-dpa lifecycle loaded into the doca_dpa_app context. The user must know where their DPA-side ELF lives (the build that DPACC produced).
Confirm the operator has the privileges the public DOCA DPA Tools guide requires. Cycle-accurate capture and high-priority threads may require elevated privileges; the public guide is the authoritative source for the exact permission set.

If the binary is not at the standard path, the fix is to install / repair the host-side DOCA package selection that includes DPA Tools, not to patch the file in place.

configure

The tracer's configuration is the invocation flags + the JSON config file. Steps the agent walks the user through, in order:

Gate — confirm DPA-side tracing is the right surface. Walk the when-is-this-the-right-surface table in CAPABILITIES.md ## Capabilities and modes. If the bug is host-side, the DPACC build, raw cycle profiling, or device visibility, route to the right skill first; do not configure the tracer.
Pick capture mode. --mode TRACE (full per-event, higher overhead) vs --mode CRIT (critical-events only, lower overhead). Default "start here" is CRIT unless the bug demands per-event detail. --sub-mode is valid only with --mode TRACE; omit it for CRIT. The TRACE-only sub-mode variants accepted by the installed binary are authoritative on --help; the agent does not invent strings. --device <mlx5_*> is mandatory (dpa_hl_tracer_utils.cpp calls doca_argp_param_set_mandatory(dev_param)); every invocation in ## run and the Command appendix below ships it. An agent producing a bare doca_dpa_hl_tracer --mode <X> command will fail argp validation before the tool reaches the DPA.
Size the JSON config. Stage a config file with:
- limits_config.log_file_max_size_in_bytes and bin_file_max_size_in_bytes — sized to the expected event rate × capture window.
- limits_config.file_size_limit_policy — pick stop-on-limit (0) for "don't lose old events" or truncate-and-continue (1) for "keep the most recent events". The two fail differently; pick explicitly.
- threads_config.receiver_thread.thread_priority / thread_core — a real-time priority + a pinned core for the receiver thread so the capture is not at the mercy of the scheduler. The shipped example pins this to a high priority.
- threads_config.binary_writer_thread, file_writer_thread, printer_thread — priorities + cores for the writers and the stdout printer; -1 means default. Pin the chosen JSON config to disk; it is part of the captured artifact set per CAPABILITIES.md ## Observability.
Plan the capture window. Decide explicitly when the tracer will start, when it will stop, and which workload phase needs to be inside the window per CAPABILITIES.md ## Capabilities and modes capture-window invariant. Events outside the window are silently lost.
Snapshot the DPA-side ELF. Pin the DPA-side ELF path and capture a SHA of the file. The decode step requires this exact ELF; a rebuild between capture and decode produces noise per the ELF-must-match-image rule.
Confirm the host-side doca-dpa lifecycle has already brought the DPA workload up at least once. Per doca-dpa TASKS.md ## run, the workload must be running for events to fire; the tracer is observation, not a workload driver.

For the canonical DOCA universal lifecycle on the host-side workload this tracer observes, see doca-programming-guide TASKS.md ## configure and doca-dpa TASKS.md ## configure. This skill is concerned with the operator-side configuration of the tracer invocation.

build

doca_dpa_hl_tracer is shipped pre-built as part of every DOCA install that includes the DPA Tools optional component (/opt/mellanox/doca/tools/doca_dpa_hl_tracer). There is no source tree the external user is expected to compile, no build flags, no meson or make workflow for the tracer itself.

Routing for nearby "build" questions:

"The binary isn't there — do I need to build it?" → no. Route to doca-setup ## install. The fix is to install (or re-install) the DOCA package selection that includes the DPA Tools component.
"I want to build my own DPA-side application that emits trace events." → not a tracer question. Route to doca-dpa TASKS.md ## build for the cross-side build pattern + the public DPACC guide via doca-public-knowledge-map for the DPACC-side build flags that enable the tracer-instrumentation hooks.
"I want to extend the tracer with new event classes." → out of scope here; this skill is for external operators consuming the shipped tracer, not for contributors extending it.

The ## What this skill deliberately does not ship block in SKILL.md explicitly forbids adding a build recipe for the tracer; revisit that policy before changing this section.

modify

Do not modify the shipped doca_dpa_hl_tracer binary. It is an NVIDIA-shipped CLI; there is no documented public way to change its capture behaviour, event taxonomy, or output format, and none should be invented.

What the agent does modify, every time, is the tracer invocation and the JSON config — the --mode, the --sub-mode, the config file's limits_config and threads_config sections, the capture window. That is the configuration loop in ## configure above and the iteration loop in ## test below; treat modify the invocation + JSON config, not the binary as the operating mode.

Routing for nearby "modify" questions:

"The trace overhead is too high — can I disable some events?" → adjust --mode, and for TRACE only --sub-mode, per ## configure step 2; raise the receiver-thread priority per step 3. If the event taxonomy itself is too coarse / too fine, the answer is "this is the documented surface; for other granularity, use the raw cycle profiler" — route via doca-public-knowledge-map ## DOCA tools.
"Can I change the trace output format?" → no; the binary trace format is documented and the human- readable log is its decoded view. If the user needs a different post-processing format, the right answer is "write a parser against the documented format on your installed version" — and even that scripting is out of scope per SKILL.md ## What this skill deliberately does not ship.
"I want a different measurement than the tracer reports." → re-examine the when-is-this-the-right-surface gate in CAPABILITIES.md ## Capabilities and modes; if the question is genuinely outside the DPA programming events surface (e.g. raw cycle counts, cache behaviour), this is not the right tool.

run

The capture → decode → render pipeline — every tracing session goes through it, no exceptions. The full invocation surface lives in the public DOCA DPA Tools page; this section names the shape of the flow, not verbatim command lines (per CAPABILITIES.md ## Safety policy "do not invent flags").

Confirm prerequisites. Per ## install and ## configure: binary present, version paired with doca-dpa library + DPACC, BlueField visible and DPA exposed, DPA-side ELF identified, host-side doca-dpa lifecycle has brought the workload up.
Capture. For CRIT, run doca_dpa_hl_tracer --device <mlx5_*> --mode CRIT --config-file <json> --output-file <bin-path> --log-file <log-path>. For TRACE, add the installed binary's documented --sub-mode <variant>. Never pass --sub-mode with CRIT. The tracer attaches to the named device, opens the JSON config, and writes the binary trace + log file while the capture window is open. Re-confirm exact flag names against --help on the installed binary.
Stop the capture explicitly. Either let it run until the file-size limit closes it (per the file_size_limit_policy) or signal it to stop at the end of the chosen window. Note which way the capture ended — the two failure modes (stop-on-limit vs truncate-and-continue) look different in the trace.
Verify the ELF, then decode. Recompute the SHA of <dpa-elf> and compare it with the SHA recorded at capture time before invoking the parser. If they differ, abort decode, preserve <bin-path> unchanged, and locate the exact capture-time ELF; rebuilding or substituting an ELF cannot repair the trace. Only after the hashes match, run doca_dpa_hl_tracer --input-file <bin-path> --parse-file <parsed-output> --elf-file <dpa-elf>. The ELF must be the exact build the doca_dpa_app context loaded at capture time per the ELF-must-match-image rule in CAPABILITIES.md ## Capabilities and modes. Decoding against a different ELF produces noise and is prohibited by this pre-decode gate.
Read the rendered output. The human-readable log plus the parsed output are the agent's read-side surface. Compare event ordering against the DPA-side kernel source; quote events verbatim, do not paraphrase.
Inspect the tool's own stderr. Per CAPABILITIES.md ## Observability, stderr carries DOCA log output the agent uses to distinguish install / device-binding / image-not- instrumented failures from real "no events" answers.

When recording the run for downstream consumers, write down: the DOCA version, the DPACC compiler version, the BlueField identity + firmware version, the DPA-side ELF path + SHA, the JSON config used, the mode and, for TRACE only, sub-mode, the capture wall-clock window, and the produced binary trace + log files. The downstream ## test and ## debug workflows depend on those fields. Classify this artifact set before sharing. Keep the unmodified raw trace, ELF/config tuple, logs, and metadata in restricted storage; produce a separate redacted copy that removes workload-sensitive event arguments, RDMA WR fields, comm-call pointers, paths, and identifiers as the operator's policy requires.

test

doca_dpa_hl_tracer is a diagnostic capture tool, so its ## test verb is about testing the trace itself — confirming the captured events are the right events, captured at sound overhead, with a sound capture window — not unit-testing the tracer binary.

## test is an iterative loop, not a one-shot pass. A capture that completes is not the same as a capture that produced defensible evidence; each iteration tightens one axis of capture soundness (mode vs overhead, window vs missed events, file-size limit vs truncation, ELF match vs decode noise, DPA-side workload up vs idle) and loops back to ## run.

The eval-loop overlay (rows apply to every trace session):

Iteration trigger	What it looks like	What changes next iteration
Decoded trace is empty	Could be install (binary failed to attach), device-binding, image not instrumented, capture window missed the workload, or workload was idle	Walk the error taxonomy in `CAPABILITIES.md ## Error taxonomy` layers 1–4 in order; do not jump.
`TRACE`-mode capture shifted measured kernel timing materially	Overhead-saturated; per-event interval near the receiver-thread scheduling floor	Drop to `CRIT` mode; raise receiver-thread priority in the JSON config; shrink the capture window.
`bin_file` hit `bin_file_max_size_in_bytes` mid-run	Capture truncated by `file_size_limit_policy=0` (stop-on-limit)	Either raise the limit, or switch to `file_size_limit_policy=1` (truncate-and-continue) if losing the oldest events is acceptable.
Pre-decode ELF SHA differs from capture-time SHA	ELF mismatch — the candidate `--elf-file` is not the build that produced the trace	Abort decode, preserve the raw binary trace, and locate the exact capture-time ELF per `CAPABILITIES.md ## Capabilities and modes`. Do not rebuild or decode against the mismatch.
Same workload, same config, traces look different across runs	Workload non-determinism, mode/sub-mode drift, or capture window straddling different phases	Pin the workload's input, the capture window, and the JSON config; re-run; if still divergent, the workload's non-determinism is the real answer the user came for.
Event ordering disagrees with the kernel source	Could be real (a sync mis-issue, a comm-call ordering bug) or apparent (the renderer's ordering is not source order)	Re-read the public DOCA DPA Tools page's documented event-ordering rules; correlate captured events with the kernel's expected DPA programming events sequence.
Capture completes but the kernel-entry event is missing	Layer 3 (image-not-instrumented) of the error taxonomy	Confirm DPACC build flags enabled instrumentation; re-build the DPA image if not.

The agent's rule: every change to the capture parameters re-opens the loop. Re-running with a tweaked mode and quoting the previous event-counts without re-checking overhead / window / ELF match is exactly the failure mode this loop replaces.

Baseline-capture rule. When the goal of the tracer session is a baseline (vs an ad-hoc question), the captured artifact must include the metadata tuple per CAPABILITIES.md ## Observability — (DOCA version, DPACC version, ELF path + SHA, mode + TRACE-only sub-mode when applicable, JSON config, capture window, BlueField identity

firmware) — alongside the binary trace, the log file, and the tool's stderr. Without all of these the baseline cannot be regression-tested later.

Loop termination: stop iterating once two consecutive runs do not change the picture — the answer is now "this is what the DPA-side workload does at the high-level event layer on this image + this BlueField". Escalate cross-version comparisons to doca-version TASKS.md ## test or doca-dpa TASKS.md ## debug with the captured artifacts as evidence.

This skill does NOT ship a "test fixture" or pre-recorded expected output. The expected output is workload-, DPA- image-, BlueField-, and DOCA-version-specific; pinning one would mislead operators on a different setup.

debug

When doca_dpa_hl_tracer fails to capture, the captured trace is empty / unreadable, or the decoded events do not match the kernel source, walk the layered error taxonomy in CAPABILITIES.md ## Error taxonomy in order. The shape of the diagnosis:

Install. Confirm the binary at /opt/mellanox/doca/tools/doca_dpa_hl_tracer exists, is executable, and its loader-dependent shared libs are present. Route to doca-setup ## install if not.
Device-binding. Confirm the BlueField the tracer is attaching to is visible to DOCA per doca-dpa TASKS.md ## configure step 1 and that the DPA processor is exposed. The tracer cannot attach if either is missing.
Image-not-instrumented. Confirm the DPA-side image's DPACC build flags enabled the tracer-instrumentation hooks per the public DPACC guide via doca-public-knowledge-map.
Capture-window. Confirm the capture window actually overlapped the workload phase under investigation and that the workload had not already ended or remained idle. Re-walk ## configure step 4; widen or re-time the window.
Decode. Confirm the ELF passed at decode time is the same build that produced the trace. SHA the ELF at capture and immediately before decode; mismatch means abort decode, preserve the raw trace, and re-locate the exact capture-time ELF.
Overhead-saturated. If TRACE mode produced a suspiciously uniform per-event interval, drop to CRIT and re-capture; if the workload's wall-clock shifted materially under tracing, raise the receiver-thread priority + pin its core in the JSON config.
Version. Walk doca-version TASKS.md ## debug end-to-end; the tracer ↔ doca-dpa library ↔ DPACC triple is the common version mismatch failure.
Cross-cutting. Cause is below DOCA — driver, firmware, BlueField mode, IOMMU, NUMA. Hand off to doca-debug ## debug and doca-setup ## debug.

In every case: quote what the tracer reported. Do not paraphrase event names, do not reorder fields, do not summarize the event stream into a single number.

use

The captured trace is evidence for a host-side DPA debug session. The agent's hand-off:

Pair the captured trace with the (DOCA version, DPACC version, ELF SHA, mode, JSON config, capture window, BlueField + firmware) tuple per CAPABILITIES.md ## Observability.
Walk doca-dpa TASKS.md ## debug with the captured trace as the runtime evidence. The host-side library skill owns the "what to change next" decision; the tracer owns the "what happened on the DPA" signal.
Cross-reference with the cross-cutting debug ladder in doca-debug ## debug when the captured events point at a layer below DOCA (driver, firmware, NUMA).
Retain the captured artifact set for later regression hunts; a trace without the metadata tuple is unreplicable.

The agent's rule: the tracer captures; the doca-dpa library skill acts. Conflating the two is the most common DPA-debug first-touch error.

Deferred task verbs

The verbs below are not doca_dpa_hl_tracer work and should be routed out before the agent does any of them under this skill's name.

DPA-side programming (writing the DPA kernel, using doca-dpa-comms, using doca-dpa-verbs) → route via doca-public-knowledge-map to the public DOCA DPA, DPACC, DPA-Comms, and DPA-Verbs guides + the shipped DPA samples. This tool traces those events; it does not redefine them.
Raw cycle profiling of the DPA processor → different surface, different tool. Route via doca-public-knowledge-map ## DOCA tools.
Host-side doca-dpa API debugging → doca-dpa TASKS.md ## debug. The tracer's host-side error surface routes back here when the cause is host-side, not DPA-side.
Production observability of DPA workloads → doca-telemetry. The tracer is a diagnostic capture surface, not a continuous production observability surface.

Command appendix

doca_dpa_hl_tracer-specific invocation classes the verbs above reach for. Every row is a CLASS — the agent must not invent flags beyond --help on the installed binary and the public DOCA DPA Tools page.

Infra-aware preamble (every row below). Per the bundle's detect → prefer → fall back → report contract documented in doca-structured-tools-contract ## The agent behavior contract, the agent should:

Probe for the matching structured helper FIRST (doca-env --json for version + devices + DPA availability in one shot; doca-capability-snapshot for per-device DPA capability flags).
If the probe succeeds, the structured tool's output is the authoritative answer.
If the probe fails, fall back to the manual command in the row.
The schemas the structured tools emit are defined in doca-structured-tools-contract ## Schemas.

Purpose (class)	Invocation (shape)	Owning step	Reads as healthy when …
Discover the documented flag surface	`doca_dpa_hl_tracer --help` + the public DOCA DPA Tools page	`## configure` step 2; `## debug` layer 1	Prints the documented inventory of `--device`, `--mode`, `--sub-mode`, `--config-file`, `--input-file`, `--output-file`, `--parse-file`, `--elf-file`, `--log-file`.
Capture a `CRIT`-mode baseline	`doca_dpa_hl_tracer --device <mlx5_*> --mode CRIT --config-file <json> --output-file <bin>`	`## run` step 2 (start-here mode)	Binary trace file grows during the capture window; stderr is quiet; on stop, the file size is bounded by `bin_file_max_size_in_bytes`.
Widen to a `TRACE`-mode capture for fine-grained perf	Same as above with `--mode TRACE` and the relevant TRACE-only `--sub-mode`	`## test` iteration	Captured event stream contains the full per-event detail; overhead is acceptable for the answer the user needs.
Decode a captured trace against the matching ELF	First compare the candidate ELF SHA with the capture-time SHA; only on equality run `doca_dpa_hl_tracer --input-file <bin> --parse-file <parsed> --elf-file <dpa-elf>`	`## run` step 4	Hashes match before decode; parsed output names the documented DPA programming events and symbols resolve cleanly. A mismatch aborts without modifying the raw trace.
Render a quick read of a captured trace	Inspect the human-readable `--log-file` alongside the binary trace	`## run` step 5; `## test` iteration	The log file shows decoded events with sensible timestamps.

Three cross-cutting rules for this appendix:

Never invent a flag, event name, mode token, or JSON key. --help on the installed binary and the public DOCA DPA Tools page are the joint contract.
Pair the trace with the (DOCA + DPACC + ELF SHA + mode + JSON config + window + BlueField + firmware) tuple. Every row above presumes the tuple was captured.
Cross-link instead of duplicate. Cross-cutting commands (pkg-config --modversion doca-dpa, doca_caps --list-devs, dmesg, mlxconfig -d <bdf> q) live in doca-debug ## debug and doca-setup TASKS.md ## debug; this appendix names only doca_dpa_hl_tracer-specific invocation classes.

Cross-cutting

A few rules that apply across every verb in this file, restated here so they are visible at the point of action and not buried in SKILL.md:

The public DOCA DPA Tools page plus the installed --help are the joint source of truth. When they disagree, the installed --help wins for the user's actual run.
Tracing is observation, not workload. The DPA workload must already be up via doca-dpa; the tracer does not drive the kernel.
Capture is bounded. Pick the file-size limit policy explicitly; the two failure modes (stop-on-limit vs truncate-and-continue) differ.
Quote the (DOCA + DPACC + ELF SHA + mode + JSON config + window + BlueField + firmware) tuple. A trace artifact without the tuple is unreplicable.
This skill assumes a healthy DOCA install with the DPA Tools component, a paired DPACC, a BlueField with a visible DPA, and the doca-dpa host-side lifecycle already started. If any of those is in doubt, route to doca-setup or doca-dpa before running anything else here.

/doca-dpa-hl-tracer

TASKS.md