/doca-dpa-hl-tracer

DOCA DPA High-Level Tracer — Capabilities

Where to start: doca_dpa_hl_tracer is a host-side CLI that captures, decodes, and renders DPA-side execution traces at the DPA programming events layer rather than as raw cycle counts. The pattern overview below names the recurring doca_dpa_hl_tracer-class questions. Pick the pattern first, then drill into the H2 that owns the substance. For the how of executing each pattern, jump to TASKS.md.

This file is loaded by SKILL.md. It documents what the tracer captures, which DPA programming events it exposes, what its capture-mode + JSON config surface looks like, what the trace-overhead-vs-fidelity tradeoff is, how it pairs with the doca-dpa library and the DPACC compiler, its layered error and observability surfaces, and the safety posture that bounds when tracing is appropriate. For step-by-step invocations and the capture → decode → render workflow, see TASKS.md.

Pattern overview

Every doca_dpa_hl_tracer-class question this skill teaches resolves into one of SIX patterns. The patterns are CLASSES — they apply across every DPA-side workload the tool can trace, not just one.

Two cross-cutting rules that apply to every pattern above:

• Tracing is observation, not workload. The tracer captures what the DPA kernel does; it does not cause the workload to run. The DPA-side workload must be brought up by doca-dpa first — start the host-side doca_dpa context, load the DPA application image, run at least one launch. A tracer started against an idle DPA produces an empty capture and the failure is by construction, not a bug.
• Capture is bounded, not unbounded. Trace overhead and trace file size both grow with capture duration. The agent's rule: pick the smallest defensible mode + window for the question being asked; widen only when the bug demands it.

Capabilities and modes

doca_dpa_hl_tracer is shipped as a single CLI binary at /opt/mellanox/doca/tools/doca_dpa_hl_tracer on every DOCA install that includes the DOCA DPA Tools optional component. It runs on the host and observes a DPA-side workload already running on a BlueField the host can reach through DOCA. There is no daemon component on the DPA side; the DPA-side instrumentation is part of the image DPACC produced.

When is DPA-side high-level tracing the right surface? A small gate the agent walks before configuring the tracer:

DPA programming event taxonomy. The tracer exposes the documented DPA programming events surface — i.e. events the DPA programming model itself defines, not the raw instruction stream. The exact event names + IDs are versioned with the doca-dpa library and authoritative on --help of the installed binary. The event classes the agent reasons about:

The agent's rule for any "my kernel is doing the wrong thing" report: name which event-class signal is inconsistent with the kernel source, and quote the captured events verbatim — do not paraphrase event names.

Capture modes. Two documented operating modes:

--sub-mode is valid only with --mode TRACE. The documented strings are DBUF and COMCH, selecting how TRACE data is carried off the DPA; they are not narrower TRACE / CRIT capture variants. A CRIT invocation must omit --sub-mode. Re-confirm the exact strings against --help on the installed version; the agent does not invent them.

JSON config layout. Capture parameters live in a JSON config file (passed via --config-file). The documented keys (re-confirm against the public DOCA DPA Tools page and --help on the installed version):

The capture is bounded by the file-size pair: with file_size_limit_policy=0 (stop-on-limit) the capture ends at the smaller limit, with file_size_limit_policy=1 (truncate-and-continue) the older events are evicted and the capture continues — the two policies fail in different ways and the operator must choose explicitly.

Capture-window invariant. The tracer captures events that fire during the window between --output-file write start and capture end. Events that fire outside the window are silently lost. The agent's rule for a sweep test: pre- plan the window to span the relevant workload phase, not just "a few seconds".

ELF-must-match-image rule. Decode (--parse-file, --elf-file) requires the same DPA-side ELF that doca-dpa loaded into the doca_dpa_app context. A decode against a different ELF build produces noise / wrong-symbol output even when the binary trace is intact. The agent's rule: snapshot the ELF path and SHA at capture time, then recompute the SHA immediately before decode. If they differ, abort decode, preserve the binary trace unchanged, and locate the exact capture-time ELF. Rebuilding an ELF or decoding anyway cannot repair the mismatch.

Version compatibility

For the canonical DOCA version-detection chain, the four-way match rule, NGC container semantics, and the headers-win-over-docs rule, see doca-version. The body lives there; this skill does not duplicate it.

The doca_dpa_hl_tracer-specific overlay is:

• Tracer ↔ doca-dpa library ↔ DPACC must match. The tracer is the operator-side companion to the host-side doca-dpa library and the DPA-side image is produced by the DPACC compiler. All three must come from a matched DOCA release band per the doca-version CAPABILITIES.md ## Version compatibility four-way rule, plus the DPA-specific DOCA must match DPACC overlay doca-dpa carries.
• The DPA-side image must be the same image the tracer decodes against. The captured binary trace file references event IDs and offsets that resolve against the specific ELF DPACC produced. Decoding against a different ELF build (e.g. an older / newer build of the same source) yields wrong-symbol output silently. The agent attaches the ELF path + SHA to the captured artifact.
• Capture-mode tokens and JSON keys are versioned. TRACE / CRIT tokens, TRACE-only --sub-mode strings, and the JSON key set evolve release to release; re-verify against --help on the installed binary and the public DOCA DPA Tools page rather than against this skill's prose.
• Public DOCA DPA Tools page is the source of truth for the tool's command-line surface. When that page disagrees with this skill (e.g. a flag landed in a release this skill was not written against), the live guide wins.

Error taxonomy

doca_dpa_hl_tracer's error surface spans the host install, the DPA / device binding, the DPA-side image's instrumentation, the capture window, decode-against-ELF, and the cross-cutting partial-install / mixed-version layer. The agent distinguishes these layers in escalating order; jumping layers wastes the user's time on the wrong fix.

1. Install. The binary is missing from /opt/mellanox/doca/tools/, the DOCA DPA Tools optional component is not installed, the binary is not executable, or its loader-dependent shared libs are missing. Routing: route to doca-setup ## install to install / repair the host-side DOCA package selection; confirm the installed version per doca-version TASKS.md ## configure.
2. Device-binding. The tracer cannot bind the BlueField device (PCIe address / IB name not found / not accessible). Cause: the BlueField is not visible to DOCA on this host, the device's DPA processor is not exposed, or the underlying driver stack (mlx5_core, IB stack) is not loaded. Routing: first confirm the device is on the DOCA capability snapshot per doca-dpa TASKS.md ## configure step 1 and doca-setup ## test; only re-attempt the tracer once the device is visible.
3. Image-not-instrumented. The tracer binds but the DPA-side image emits no events. Cause: the DPA-side application image is not the documented instrumented build (DPACC build flags did not enable the tracer-instrumentation hooks). Routing: confirm via doca-dpa TASKS.md ## run that the instrumented image was loaded; re-read the public DPACC guide via doca-public-knowledge-map for the documented instrumentation build options.
4. Capture-window. Capture started after the relevant events fired (including a workload that ended before capture began), ended before they fired, observed an idle workload, or saturated the file-size limit before completing. Cause: window too short, file-size limits too small for the chosen mode, file_size_limit_policy=0 (stop-on-limit) truncated the capture. Routing: re-walk TASKS.md ## configure step 3 (size the window + limits); re-run with adjusted bounds.
5. Decode. The binary trace file is intact but the parser produces noise / wrong-symbol output. Cause: the ELF passed to --elf-file is the wrong build, was rebuilt after capture, or is missing the debug-info sections the tracer's parser uses. Routing: recompute the ELF SHA before invoking decode and compare it with the capture-time SHA (per ## Capabilities and modes ELF-must-match-image rule). On mismatch, abort decode and locate the exact capture-time ELF; do not rebuild, substitute, or decode against the mismatched file.
6. Overhead-saturated. The capture completed but the measured behaviour is dominated by tracer overhead, not the workload. Symptoms: per-event timing in the trace is suspiciously uniform; the workload's wall-clock latency went up by a noticeable factor compared to without the tracer; the per-event interval is at the noise floor of the receiver thread's scheduling quantum. Routing: drop from TRACE to CRIT mode; re-run with a smaller capture window; consider raising the receiver-thread priority in the JSON config.
7. Version. Cross-cutting partial-install / mixed-version per doca-version CAPABILITIES.md ## Version compatibility. Symptoms: tracer version disagrees with pkg-config --modversion doca-dpa; DPACC compiler version is not aligned with the DOCA install. Routing: walk doca-version TASKS.md ## debug before any further investigation.
8. Cross-cutting. Cause is below DOCA — driver, firmware, BlueField mode, IOMMU, NUMA. Routing: hand off to doca-debug ## debug and doca-setup ## debug.

doca_dpa_hl_tracer does not itself participate in the cross-library DOCA_ERROR_* taxonomy because it is a CLI driving instrumentation, not a doca_* library call. The cross-library taxonomy is owned by doca-programming-guide CAPABILITIES.md ## Error taxonomy.

Observability

doca_dpa_hl_tracer's observability surface is the captured trace itself, plus the tool's own stderr / log file. Specifically:

• Binary trace file (bin_file). The primary machine-readable surface. Capped by bin_file_max_size_in_bytes in the JSON config; the agent's rule for any baseline that will be re-read later is capture the bin_file, not just the human-readable rendering.
• Human-readable log file (log_file). The decoded / rendered view of the events. Capped by log_file_max_size_in_bytes. Useful for a quick read during the capture-decode loop; the bin_file is the durable artifact.
• Tool's own stderr. The tracer emits DOCA log output to stderr (the DOCA_LOG_BACKEND stack) — the agent captures it alongside the trace files; an empty trace with a meaningful stderr is the most common install / capture-window failure mode.
• JSON config echoed at start. The agent's rule: retain the JSON config used for each capture as part of the captured artifact set, so the (mode + JSON config + ELF) triple that produced the trace is reconstructable later.
• Capture-time tuple. The minimum metadata to attach to any captured trace artifact: (DOCA version, DPACC compiler version, ELF path + SHA, mode + TRACE-only sub-mode when applicable, JSON config, capture wall-clock window, BlueField identity + firmware version). Without this, two captures in a fleet drawer cannot be ranked or attributed.

For the cross-cutting env-side observability primitives (PCIe scans, devlink, mlxconfig), see doca-setup CAPABILITIES.md ## Observability. For the program-side observability surface (DOCA log levels, DOCA_LOG_LEVEL, --sdk-log-level), see doca-programming-guide CAPABILITIES.md ## Observability.

Safety policy

Overlay on the bundle-wide hardware-safety meta-policy. The rules below are this skill's per-artifact overlay on the cross-cutting rules in doca-hardware-safety CAPABILITIES.md ## Safety policy (specifically ### Per-artifact overlay pattern). When the two layers disagree, the stricter wins; when either layer says STOP, the agent stops.

doca_dpa_hl_tracer is a diagnostic observation tool — it does not modify firmware, change BlueField mode, or patch DPA images. Its safety surface is instead about the honesty of the captured signal and the boundedness of the capture itself. The rules:

• Tracing is not a production observability surface. The tool is documented for diagnostic capture; it is not a substitute for production telemetry (route there via doca-telemetry). The agent does not recommend running the tracer continuously against a production workload.
• The tracer perturbs what it observes. TRACE mode in particular can shift kernel timing by a measurable factor in tight loops. Any conclusion drawn from a trace must account for the overhead-vs- fidelity tradeoff in ## Capabilities and modes. Quoting a captured latency without noting the mode is misleading.
• Capture is bounded; choose the failure mode explicitly. file_size_limit_policy=0 (stop-on-limit) silently truncates at the limit; file_size_limit_policy=1 (truncate-and-continue) evicts older events. Both fail in different ways and the operator must pick.
• Pair the trace with the ELF + config. A trace artifact without the (ELF path + SHA, JSON config, mode) triple is unreplicable. The agent's rule for every captured trace: retain the triple alongside the binary trace and log files.
• Captured traces can leak workload information. Event arguments, RDMA WR fields, and comm-call payload pointers may show up in the trace. Before sharing, classify the binary trace, decoded output, logs, stderr, config, and metadata under the operator's data-handling policy; redact workload-sensitive fields from a derived shareable copy. Preserve the unmodified raw trace and matching ELF/config tuple in restricted storage. Never overwrite the restricted raw artifact with the redacted derivative.
• Do not invent event names, mode tokens, or JSON keys. The documented surface lives on the public DOCA DPA Tools page and in --help on the installed binary; prose-derived names are the most common hallucination failure for this skill.

Public-source pointer

The canonical public source for doca_dpa_hl_tracer is the DOCA DPA Tools umbrella on docs.nvidia.com, reachable through doca-public-knowledge-map ## DOCA tools. The companion programming-model surface lives in the public DOCA DPA, DPACC, DPA-Comms, and DPA-Verbs guides on the same site. Do not invent flags, event names, mode tokens, or JSON keys beyond what those public sources document, and re-verify against --help on the installed binary.