NV Import Vision Model Report -- Step 8

Generate benchmark report with charts, HTML, and PDF from completed benchmarks.

The model directory is: $ARGUMENTS

⛔ STRICT HTML+PDF RULE — NO EXCEPTIONS, NO DEVIATIONS

HTML and PDF MUST be generated via the canonical pipeline script. Do NOT write your own HTML generator.

The ONLY permitted way to generate the HTML + PDF:
python3 .claude/skills/deepstream-import-vision-model/scripts/report/md-to-html-pdf.py \
  models/$MODEL_NAME/reports/benchmark_report.md \
  .claude/skills/deepstream-import-vision-model/scripts/report/report-style.css \
  models/$MODEL_NAME/reports/ \
  $MODEL_NAME
This produces:

models/$MODEL_NAME/reports/benchmark_report.html — styled with report_style.css, charts embedded as base64

models/$MODEL_NAME/reports/benchmark_report_${MODEL_NAME}.pdf — via wkhtmltopdf

FORBIDDEN — never do any of these:

Write your own generate_html.py or any custom markdown-to-HTML converter script

Call wkhtmltopdf directly — use md-to-html-pdf.py which already calls it correctly

Use md-to-pdf.sh — GFM+Mermaid design doc tool only, wrong CSS

Use pandoc, pdflatex, or any other converter

The report_style.css provides the ONLY correct CSS (dark navy headers #283593, alternating rows #e8eaf6, dark code blocks #263238). Any other CSS produces wrong-looking reports.

8a: Report Structure — 12 Mandatory Sections

The report must contain exactly these 12 sections in order:

Model Configuration — model name, source (HF repo / NGC), architecture, ONNX source, input/output shapes, classes, custom parser name, cluster mode, precision, engine profile
System Configuration — GPU (name + VRAM), Driver, CUDA, TensorRT, DeepStream, OS, Python, PyTorch, ONNX versions
Preprocessing — net-scale-factor, offsets, color format, normalization details (with reference to the preprocessing table in deepstream-import-vision-model/SKILL.md)
Engine Build Summary — source format, conversion path, engine filename (with max_bs postfix), engine size (MB), FP16 flag, builder_optimization_level if non-default, timing cache path
trtexec Results — two runs (BS=1 and BS=MAX_BS) with: QPS, Images/s, GPU Compute mean/P99 (ms). Do NOT include H2D/D2H latency or Host Latency. Show PEAK_GPU_STREAMS derivation:
```
PEAK_GPU_STREAMS = floor(QPS_at_MAX_BS × MAX_BS / 30)
                 = floor(imgs_per_sec_at_MAX_BS / 30)
```
PEAK_GPU_STREAMS Derivation — explicit calculation block showing formula, inputs, and result. If a second engine was built, show both PEAK_GPU_STREAMS computations.
Single-Stream Validation — KITTI frame count, frames with detections, top-10 detected classes (from KITTI dump), validation result (PASS/FAIL)
DeepStream Benchmark Results — two runs:
- DS Run 1 (Calibration at PEAK_GPU_STREAMS): streams, batch, FPS/stream, total img/s, real-time (YES/NO)
- DS Run 2 (Validation at RT_STREAMS): streams, batch, FPS/stream, total img/s, real-time (YES)
trtexec vs DeepStream Comparison — 3-column table: trtexec | DS Run 1 | DS Run 2, rows: engine, batch/streams, total imgs/s, FPS/stream, real-time ≥30fps, DS Efficiency %
Efficiency Analysis — efficiency formula, Run 1 and Run 2 percentages, breakdown of the gap (NVDEC + mux + GStreamer overhead), GPU-bound vs pipeline-bound verdict

Pipeline Timing — per-step wall-clock duration and total:

Step	Description	Duration
1-3	HF Model Acquire (download + inspect ONNX)	{time}s
4	Engine build	{time}s
5	trtexec BS=1 + BS=MAX_BS	{time}s
6	Parser + config + visual validation + KITTI	{time}s
7 Run 1	DS Calibration (PEAK_GPU_STREAMS streams)	{time}s
7 Run 2	DS Validation (RT_STREAMS streams)	{time}s
8	Report generation	{time}s
Total	End-to-end	{total}s

Reference Commands — exact reproducible commands:
- trtexec engine build (full command with all flags and paths)
- trtexec benchmark BS=1 and BS=MAX_BS
- DeepStream single-stream validation (gst-launch-1.0 with filesink + OSD)
- DeepStream multi-stream benchmark (deepstream-app with enable-perf-measurement=1 via ds-perf-run.sh, PEAK_GPU_STREAMS and RT_STREAMS variants)
- nvinfer config key fields (as an ini code block)
- Custom parser build command (make with DEEPSTREAM_DIR and CUDA_VER)
- Use actual absolute paths from the model directory, never placeholders

Pre-flight: Extract Variables from Benchmark Logs

Before generating any output, derive all variables by reading completed benchmark files. These variables are used by every section below.

STEP8_START=$(date +%s.%N)

MODEL_DIR="${ARGUMENTS%/}"
MODEL_NAME=$(basename "$MODEL_DIR")

# Engine + MAX_BS + MODEL_FILENAME from the shared resolver (scripts/model/resolve-engine.sh).
eval "$(bash .claude/skills/deepstream-import-vision-model/scripts/model/resolve-engine.sh "$MODEL_NAME")" || exit 1

# Extract input name and spatial dims from ONNX (needed for reference commands in the report)
ONNX_FILE=$(ls models/$MODEL_NAME/model/*.onnx 2>/dev/null | grep -v '_dynamic' | head -1)
if [ -n "$ONNX_FILE" ]; then
  INSPECT_OUT=$(python3 .claude/skills/deepstream-import-vision-model/scripts/model/inspect-onnx.py "$ONNX_FILE" 2>/dev/null)
  INPUT_NAME=$(echo "$INSPECT_OUT" | grep -oP 'input_name:\s*\K\S+')
  H=$(echo "$INSPECT_OUT" | grep -oP 'height:\s*\K[0-9]+')
  W=$(echo "$INSPECT_OUT" | grep -oP 'width:\s*\K[0-9]+')
fi
INPUT_NAME=${INPUT_NAME:-"images"}  # fallback
H=${H:-"640"}; W=${W:-"640"}       # fallback — update if model uses different resolution

# Parse trtexec BS=1 log — fixed filename trtexec_b1.log (no timestamp, no wildcard needed)
TRTEXEC_LOG_BS1="models/$MODEL_NAME/benchmarks/b1/trtexec_b1.log"
[ -f "$TRTEXEC_LOG_BS1" ] || { echo "ERROR: $TRTEXEC_LOG_BS1 not found — run Steps 4-5 first (references/engine-build.md)"; exit 1; }
QPS_BS1=$(grep -oP 'Throughput:\s*\K[0-9.]+' "$TRTEXEC_LOG_BS1" | tail -1)
GPU_MEAN_BS1=$(grep -oP 'GPU Compute Time:.*mean = \K[0-9.]+' "$TRTEXEC_LOG_BS1" | tail -1)

# Parse trtexec BS=MAX_BS log — fixed filename trtexec_b${MAX_BS}.log
TRTEXEC_LOG_BSMAX="models/$MODEL_NAME/benchmarks/b${MAX_BS}/trtexec_b${MAX_BS}.log"
[ -f "$TRTEXEC_LOG_BSMAX" ] || { echo "ERROR: $TRTEXEC_LOG_BSMAX not found — run Steps 4-5 first (references/engine-build.md)"; exit 1; }
QPS_BS_MAX=$(grep -oP 'Throughput:\s*\K[0-9.]+' "$TRTEXEC_LOG_BSMAX" | tail -1)
GPU_MEAN_BS_MAX=$(grep -oP 'GPU Compute Time:.*mean = \K[0-9.]+' "$TRTEXEC_LOG_BSMAX" | tail -1)
GPU_P99_BS_MAX=$(grep -oP 'GPU Compute Time:.*percentile\(99%\) = \K[0-9.]+' "$TRTEXEC_LOG_BSMAX" | tail -1)
[ -z "$QPS_BS_MAX" ] && { echo "ERROR: Could not parse Throughput from $TRTEXEC_LOG_BSMAX — log may be empty or malformed"; exit 1; }
[ -z "$MAX_BS" ] && { echo "ERROR: Could not parse batch size from engine filename: $ENGINE"; exit 1; }

read IMGS_PER_SEC PEAK_GPU_STREAMS < <(python3 -c "
import math
imgs = float('$QPS_BS_MAX') * $MAX_BS
print(round(imgs, 2), int(math.floor(imgs / 30)))
")

# Parse DeepStream Run 1 and Run 2 FPS from logs written by pipeline-run
# Fixed filename pattern: benchmarks/ds/ds_s{N}_run1.log and ds_s{N}_run2.log
# Use glob to find them (N varies per model) then extract N from filename
DS_LOG_RUN1=$(ls models/$MODEL_NAME/benchmarks/ds/ds_s*_run1.log 2>/dev/null | head -1)
# Step 7's convergence loop can leave SEVERAL run2 logs (e.g. s17 -> s11 failed -> s9 passed).
# Lexicographic `head -1` picks ds_s11 over ds_s9, reporting the NON-real-time run as the
# validated result. Convergence steps downward, so the converged log is the newest — pick by mtime.
# (`sort -V | tail -1` is NOT correct here: it re-selects the highest N, i.e. the failing run.)
DS_LOG_RUN2=$(ls -t models/$MODEL_NAME/benchmarks/ds/ds_s*_run2.log 2>/dev/null | head -1)
[ -z "$DS_LOG_RUN1" ] && { echo "ERROR: No DS Run 1 log found at benchmarks/ds/ds_s*_run1.log — run Steps 6-7 first (references/pipeline-run.md)"; exit 1; }
[ -z "$DS_LOG_RUN2" ] && { echo "ERROR: No DS Run 2 log found at benchmarks/ds/ds_s*_run2.log — run Steps 6-7 first (references/pipeline-run.md)"; exit 1; }

N_RUN1=$(basename "$DS_LOG_RUN1" | grep -oP 'ds_s\K[0-9]+(?=_run1)')
N_RUN2=$(basename "$DS_LOG_RUN2" | grep -oP 'ds_s\K[0-9]+(?=_run2)')
[[ "$N_RUN1" =~ ^[0-9]+$ ]] || { echo "ERROR: Could not parse stream count from $(basename "$DS_LOG_RUN1") — expected filename pattern ds_s<N>_run1.log"; exit 1; }
[[ "$N_RUN2" =~ ^[0-9]+$ ]] || { echo "ERROR: Could not parse stream count from $(basename "$DS_LOG_RUN2") — expected filename pattern ds_s<N>_run2.log"; exit 1; }
RT_STREAMS=$N_RUN2

# deepstream-app **PERF: format is `**PERF: fps_run0 (fps_avg0)  fps_run1 (fps_avg1)  ...`
# Capture stream-0 instantaneous FPS (\K after `**PERF:`) — 1 value per line — so
# tail -10 always covers exactly 10 measurement windows regardless of stream count.
# Multiply by stream count for total throughput.
FPS_RAW_RUN1=$(grep -oP '\*\*PERF:\s*\K[0-9.]+' "$DS_LOG_RUN1" | tail -10 | python3 -c "
import sys; vals=[float(l) for l in sys.stdin if l.strip()]; print(round(sum(vals)/len(vals),2) if vals else 0)")
FPS_RAW_RUN2=$(grep -oP '\*\*PERF:\s*\K[0-9.]+' "$DS_LOG_RUN2" | tail -10 | python3 -c "
import sys; vals=[float(l) for l in sys.stdin if l.strip()]; print(round(sum(vals)/len(vals),2) if vals else 0)")

TOTAL_FPS_RUN1=$(python3 -c "print(round(float('$FPS_RAW_RUN1') * $N_RUN1, 2))")
TOTAL_FPS_RUN2=$(python3 -c "print(round(float('$FPS_RAW_RUN2') * $N_RUN2, 2))")

echo "=== Report Variables ==="
echo "MODEL_NAME=$MODEL_NAME  MAX_BS=$MAX_BS"
echo "BS=1:       QPS=$QPS_BS1  GPU mean=${GPU_MEAN_BS1}ms"
echo "BS=$MAX_BS: QPS=$QPS_BS_MAX  imgs/s=$IMGS_PER_SEC  PEAK_GPU_STREAMS=$PEAK_GPU_STREAMS"
echo "DS Run 1:   FPS/stream=$FPS_RAW_RUN1  streams=$N_RUN1  total=$TOTAL_FPS_RUN1 img/s"
echo "DS Run 2:   FPS/stream=$FPS_RAW_RUN2  streams=$N_RUN2  total=$TOTAL_FPS_RUN2 img/s  RT_STREAMS=$RT_STREAMS"

Then immediately write benchmark_data.json before generating charts (so charts can load it if needed):

mkdir -p models/$MODEL_NAME/reports
python3 << 'EOF'
import json, os

def to_num(v, cast=float):
    """Return cast(v) or None if v is empty/invalid — prevents malformed JSON."""
    try:
        return cast(v) if v and str(v).strip() else None
    except (ValueError, TypeError):
        return None

data = {
    "model_name":       os.environ.get("MODEL_NAME", ""),
    "engine":           os.environ.get("ENGINE", ""),
    "max_bs":           to_num(os.environ.get("MAX_BS"), int),
    "trtexec": {
        "bs1":   {
            "qps":         to_num(os.environ.get("QPS_BS1")),
            "gpu_mean_ms": to_num(os.environ.get("GPU_MEAN_BS1"))
        },
        "bsmax": {
            "qps":         to_num(os.environ.get("QPS_BS_MAX")),
            "gpu_mean_ms": to_num(os.environ.get("GPU_MEAN_BS_MAX")),
            "p99_ms":      to_num(os.environ.get("GPU_P99_BS_MAX")),
            "imgs_per_sec": to_num(os.environ.get("IMGS_PER_SEC"))
        }
    },
    "peak_gpu_streams": to_num(os.environ.get("PEAK_GPU_STREAMS"), int),
    "deepstream": {
        "run1": {
            "streams":        to_num(os.environ.get("N_RUN1"), int),
            "total_fps":      to_num(os.environ.get("TOTAL_FPS_RUN1")),
            "fps_per_stream": to_num(os.environ.get("FPS_RAW_RUN1"))
        },
        "run2": {
            "streams":        to_num(os.environ.get("N_RUN2"), int),
            "total_fps":      to_num(os.environ.get("TOTAL_FPS_RUN2")),
            "fps_per_stream": to_num(os.environ.get("FPS_RAW_RUN2"))
        }
    }
}
out_path = os.path.join("models", os.environ.get("MODEL_NAME", "unknown"),
                        "reports", "benchmark_data.json")
with open(out_path, "w") as f:
    json.dump(data, f, indent=2)
print("benchmark_data.json written")
EOF

<< 'EOF' (quoted) prevents bash expansion — Python reads all variables via os.environ.get(), applies to_num() for safe numeric conversion (returns None instead of producing malformed JSON when a variable is unset), then uses json.dump to guarantee valid output.

8c-1: Chart Generation (MANDATORY)

All Python scripts in this step run inside the shared venv at build/.venv_optimum (which holds matplotlib, numpy, markdown, and onnxruntime). Activate it once before running any report scripts:

source build/.venv_optimum/bin/activate

Generate exactly 5 charts using matplotlib in models/{model_name}/reports/charts/. Use the script at .claude/skills/deepstream-import-vision-model/scripts/report/generate-benchmark-charts.py or generate manually. Chart names are fixed — do not rename them.

Filename	Content	Chart type
`chart_trtexec_bs1_vs_bsmax.png`	Bar chart: QPS at BS=1 vs BS=MAX_BS (side by side)	Grouped bar
`chart_trtexec_throughput.png`	GPU-only images/sec at MAX_BS, with PEAK_GPU_STREAMS annotation (dashed line at y=PEAK_GPU_STREAMS×30)	Single bar or line
`chart_ds_streams_vs_fps.png`	Line chart: X=stream count (PEAK_GPU_STREAMS, RT_STREAMS), Y=FPS/stream. Red dashed line at 30fps threshold.	Line + markers
`chart_trt_vs_ds.png`	Grouped bars: trtexec total imgs/s \| DS Run 1 total imgs/s \| DS Run 2 total imgs/s	Grouped bar
`chart_efficiency.png`	DS efficiency %: 2 bars (Run 1 efficiency, Run 2 efficiency), dashed line at 100%	Bar

Do NOT generate H2D/D2H transfer overhead charts.

Chart style requirements:

Figure size: figsize=(10, 6), DPI: 150
Title: two-line format via two_line_title(model_name, subtitle) — model name on line 1, chart description on line 2 (prevents long titles from clipping outside figure bounds)
Axis labels: 13px; Bar value labels: bold, 12-13px, positioned above bars
Grid: axis='y', alpha=0.3; plt.tight_layout() before save
Use matplotlib.use('Agg') (no display needed)

8c-1b: Markdown Report (MANDATORY)

Generate benchmark_report.md before the HTML. This file must contain all 12 sections filled with actual values — no placeholders allowed.

First, gather system info not already captured in pre-flight:

GPU_INFO=$(nvidia-smi --query-gpu=name,memory.total --format=csv,noheader | head -1)
GPU_NAME=$(echo "$GPU_INFO" | cut -d, -f1 | xargs)
GPU_VRAM=$(echo "$GPU_INFO" | cut -d, -f2 | xargs)
DRIVER_VER=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader | head -1 | xargs)
CUDA_VER=$(nvcc --version 2>/dev/null | grep -oP 'release \K[0-9.]+' || echo "N/A")
TRT_VER=$(trtexec 2>&1 | head -3 | grep -oP 'TensorRT v\K[0-9.]+' || echo "N/A")
DS_VER=$(deepstream-app --version-all 2>/dev/null | grep -oP 'DeepStreamSDK \K[0-9.]+' || echo "N/A")
ENGINE_SIZE_MB=$(du -m "$ENGINE" | cut -f1)
IMGS_PER_SEC_BS1=$(python3 -c "print(round(float('$QPS_BS1') * 1, 2))")
GPU_P99_BS1=$(grep -oP 'GPU Compute Time:.*percentile\(99%\) = \K[0-9.]+' "$TRTEXEC_LOG_BS1" | tail -1)
GPU_P99_BS1=${GPU_P99_BS1:-"N/A"}  # fallback if log too short to have P99
EFFICIENCY_RUN1=$(python3 -c "print(round(float('$TOTAL_FPS_RUN1') / float('$IMGS_PER_SEC') * 100, 1))")
EFFICIENCY_RUN2=$(python3 -c "print(round(float('$TOTAL_FPS_RUN2') / float('$IMGS_PER_SEC') * 100, 1))")
RT_LABEL_RUN1=$(python3 -c "print('YES' if float('$FPS_RAW_RUN1') >= 30 else 'NO')")
RT_LABEL_RUN2=$(python3 -c "print('YES' if float('$FPS_RAW_RUN2') >= 30 else 'NO')")

Then write the markdown (use unquoted << MDEOF so bash expands variables):

cat > models/$MODEL_NAME/reports/benchmark_report.md << MDEOF
# ${MODEL_NAME} Benchmark Report

Generated: $(date '+%Y-%m-%d %H:%M:%S')

---

## 1. Model Configuration

| Parameter | Value |
|-----------|-------|
| **Model Name** | ${MODEL_NAME} |
| **Source** | (fill from Steps 1-3 log) |
| **Architecture** | (fill from config.json model_type) |
| **ONNX Source** | models/${MODEL_NAME}/model/ |
| **Precision** | FP16 |
| **Engine File** | $(basename $ENGINE) |
| **Engine Profile** | min=1x3x640x640  opt=${MAX_BS}x3x640x640  max=${MAX_BS}x3x640x640 |
| **Custom Parser** | libnvdsinfer_${MODEL_NAME}_parser.so |
| **Cluster Mode** | (fill from nvinfer config) |

## 2. System Configuration

| Parameter | Value |
|-----------|-------|
| **GPU** | ${GPU_NAME} |
| **VRAM** | ${GPU_VRAM} |
| **Driver** | ${DRIVER_VER} |
| **CUDA** | ${CUDA_VER} |
| **TensorRT** | ${TRT_VER} |
| **DeepStream** | ${DS_VER} |

## 3. Preprocessing

| Parameter | Value |
|-----------|-------|
| **net-scale-factor** | (fill from nvinfer config) |
| **offsets** | (fill from nvinfer config) |
| **Color Format** | (fill from nvinfer config) |
| **Input Resolution** | 640×640 |

## 4. Engine Build Summary

| Parameter | Value |
|-----------|-------|
| **Source Format** | ONNX |
| **Engine File** | $(basename $ENGINE) |
| **Engine Size** | ${ENGINE_SIZE_MB} MB |
| **FP16** | Enabled |
| **MAX Batch Size** | ${MAX_BS} |
| **Workspace** | 32768 MiB |
| **Timing Cache** | models/${MODEL_NAME}/benchmarks/engines/timing.cache |

## 5. trtexec Results

| Metric | BS=1 | BS=${MAX_BS} |
|--------|------|------|
| **QPS (queries/s)** | ${QPS_BS1} | ${QPS_BS_MAX} |
| **Images/s** | ${IMGS_PER_SEC_BS1} | ${IMGS_PER_SEC} |
| **GPU Compute Mean (ms)** | ${GPU_MEAN_BS1} | ${GPU_MEAN_BS_MAX} |
| **GPU Compute P99 (ms)** | ${GPU_P99_BS1} | ${GPU_P99_BS_MAX} |

> Note: H2D/D2H latency excluded — trtexec run with \`--noDataTransfers\` to match DeepStream (GPU-to-GPU data flow, no host transfers).

![trtexec BS=1 vs BS=${MAX_BS}](charts/chart_trtexec_bs1_vs_bsmax.png)

## 6. PEAK_GPU_STREAMS Derivation

\`\`\`
PEAK_GPU_STREAMS = floor(imgs_per_sec_at_MAX_BS / 30)
                = floor(${IMGS_PER_SEC} / 30)
                = ${PEAK_GPU_STREAMS} streams
\`\`\`

![trtexec throughput at BS=${MAX_BS}](charts/chart_trtexec_throughput.png)

## 7. Single-Stream Validation

| Parameter | Value |
|-----------|-------|
| **Video Source** | sample_720p.mp4 (1280×720) |
| **KITTI Output Dir** | models/${MODEL_NAME}/samples/kitti_output/ |
| **Total Frames** | (fill from kitti dump) |
| **Frames with Detections** | (fill from kitti dump) |
| **Detection Rate** | (fill — must be ≥ 90%) |
| **Visual Capture Mode** | (fill: `nvv4l2h264enc MP4` OR `theoraenc OGV (NVENC unavailable)` OR `skipped (no encoder available)`) |
| **Visual Capture Artifact** | (fill: `samples/${MODEL_NAME}_output.mp4` for NVENC path; `samples/${MODEL_NAME}_output.ogv` for theoraenc fallback; `N/A` if skipped) |
| **Validation Result** | PASS |

> **Encoder reporting rule (MANDATORY):** The Visual Capture Mode field MUST be exactly one of:
> - `nvv4l2h264enc MP4` — NVENC succeeded; artifact is `.mp4`
> - `theoraenc OGV (NVENC unavailable)` — if `DS_SINGLE_STREAM_MODE=theoraenc-fallback`; use `.ogv` path from `DS_SINGLE_STREAM_OUTPUT=`
> - `skipped (no encoder available)` — if `DS_SINGLE_STREAM_MODE=skipped`; no artifact file
> `x264enc` and `openh264enc` are prohibited and must never appear in this field.

## 8. DeepStream Benchmark Results

### DS Run 1 — Calibration at PEAK_GPU_STREAMS (${N_RUN1} streams)

| Metric | Value |
|--------|-------|
| **Streams** | ${N_RUN1} |
| **Batch Size** | ${N_RUN1} |
| **FPS / Stream** | ${FPS_RAW_RUN1} |
| **Total Images/s** | ${TOTAL_FPS_RUN1} |
| **Real-Time (≥30 fps/stream)** | ${RT_LABEL_RUN1} |

### DS Run 2 — Validation at RT_STREAMS (${N_RUN2} streams)

| Metric | Value |
|--------|-------|
| **Streams** | ${N_RUN2} |
| **Batch Size** | ${N_RUN2} |
| **FPS / Stream** | ${FPS_RAW_RUN2} |
| **Total Images/s** | ${TOTAL_FPS_RUN2} |
| **Real-Time (≥30 fps/stream)** | ${RT_LABEL_RUN2} |

![DeepStream FPS/stream vs stream count](charts/chart_ds_streams_vs_fps.png)

## 9. trtexec vs DeepStream Comparison

| Metric | trtexec BS=${MAX_BS} | DS Run 1 (${N_RUN1} streams) | DS Run 2 (${N_RUN2} streams) |
|--------|---------------------|------------------------------|------------------------------|
| **Engine** | $(basename $ENGINE) | $(basename $ENGINE) | $(basename $ENGINE) |
| **Batch / Streams** | BS=${MAX_BS} | ${N_RUN1} streams | ${N_RUN2} streams |
| **Total imgs/s** | ${IMGS_PER_SEC} | ${TOTAL_FPS_RUN1} | ${TOTAL_FPS_RUN2} |
| **FPS / stream** | $(python3 -c "print(round(float('$IMGS_PER_SEC')/${MAX_BS},1))") | ${FPS_RAW_RUN1} | ${FPS_RAW_RUN2} |
| **Real-Time ≥30fps** | YES | ${RT_LABEL_RUN1} | ${RT_LABEL_RUN2} |
| **DS Efficiency %** | — | ${EFFICIENCY_RUN1}% | ${EFFICIENCY_RUN2}% |

![trtexec vs DeepStream total throughput](charts/chart_trt_vs_ds.png)

## 10. Efficiency Analysis

\`\`\`
DS Efficiency = DS_total_imgs_per_sec / trtexec_imgs_per_sec × 100
Run 1: ${TOTAL_FPS_RUN1} / ${IMGS_PER_SEC} × 100 = ${EFFICIENCY_RUN1}%
Run 2: ${TOTAL_FPS_RUN2} / ${IMGS_PER_SEC} × 100 = ${EFFICIENCY_RUN2}%
\`\`\`

Efficiency gap breakdown: NVDEC decode overhead (~5-10%), GStreamer mux/queue overhead (~5-10%), CPU scheduler jitter (~2-5%).

Interpretation notes for the numbers above:

- **Well-balanced pipeline**: GPU=99-100%, NVDEC=99-100%, CPU=30-40% with no single core pinned. The ~50% DS/trtexec gap at this utilization is physically irreducible — it's the cost of real decode + memory transfers that trtexec skips with \`--noDataTransfers\`.
- **DS efficiency above 100% is expected for ViT / transformer models**: the TRT compiler backend (opt-level 4) often produces bimodal GPU latency with two alternating execution paths (e.g., 1.5ms and 4.0ms modes for OWL-ViT). trtexec reports high variance and a conservative median; DeepStream's pipelined scheduling smooths the bimodal pattern and can achieve 100-110% of the trtexec baseline. This is not a measurement error.
- **1080p tends to saturate NVDEC** while GPU has headroom. The pipeline is pinned to 720p (\`sample_720p.mp4\`) specifically to keep benchmarks comparable across models.

![DeepStream efficiency vs trtexec baseline](charts/chart_efficiency.png)

## 11. Pipeline Timing

| Step | Description | Duration |
|------|-------------|----------|
| 1-3 | HF Model Acquire (download + inspect ONNX) | (fill from step timing) |
| 4 | Engine build | (fill from step timing) |
| 5 | trtexec BS=1 + BS=${MAX_BS} | (fill from step timing) |
| 6 | Parser + config + visual validation + KITTI | (fill from step timing) |
| 7 Run 1 | DS Calibration (${N_RUN1} streams) | (fill from step timing) |
| 7 Run 2 | DS Validation (${N_RUN2} streams) | (fill from step timing) |
| 8 | Report generation | (fill) |
| **Total** | **End-to-end** | **(fill)** |

## 12. Reference Commands

### Engine Build
\`\`\`bash
trtexec --onnx=models/${MODEL_NAME}/model/${MODEL_FILENAME}.onnx \\
  --saveEngine=models/${MODEL_NAME}/benchmarks/engines/${MODEL_FILENAME}_dynamic_b${MAX_BS}.engine \\
  --minShapes=${INPUT_NAME}:1x3x${H}x${W} \\
  --optShapes=${INPUT_NAME}:${MAX_BS}x3x${H}x${W} \\
  --maxShapes=${INPUT_NAME}:${MAX_BS}x3x${H}x${W} \\
  --fp16 --memPoolSize=workspace:32768M \\
  --timingCacheFile=models/${MODEL_NAME}/benchmarks/engines/timing.cache
\`\`\`

### trtexec Benchmark
\`\`\`bash
# BS=1
trtexec --loadEngine=$(basename $ENGINE) --shapes=${INPUT_NAME}:1x3x${H}x${W} \\
  --noDataTransfers --warmUp=1000 --duration=10

# BS=${MAX_BS}
trtexec --loadEngine=$(basename $ENGINE) --shapes=${INPUT_NAME}:${MAX_BS}x3x${H}x${W} \\
  --noDataTransfers --warmUp=1000 --duration=10
\`\`\`

### DeepStream Single-Stream Validation
\`\`\`bash
# See models/${MODEL_NAME}/scripts/ for full gst-launch-1.0 command
\`\`\`

### DeepStream Multi-Stream Benchmark
\`\`\`bash
# DS Run 1: ${N_RUN1} streams — see models/${MODEL_NAME}/scripts/
# DS Run 2: ${N_RUN2} streams — see models/${MODEL_NAME}/scripts/
\`\`\`

### Custom Parser Build
\`\`\`bash
cd models/${MODEL_NAME}/parser && make DEEPSTREAM_DIR=/opt/nvidia/deepstream/deepstream CUDA_VER=13.2
\`\`\`
MDEOF
echo "benchmark_report.md written: $(wc -l < models/$MODEL_NAME/reports/benchmark_report.md) lines"

Note on "fill" fields: Fields marked (fill from ...) must be replaced with actual values from the step logs before finalizing. Search the step output logs for the exact values and substitute them. Do not leave any (fill ...) placeholder in the final report.

8c-2 + 8c-3: HTML + PDF Report (MANDATORY — ONE COMMAND)

Before generating HTML+PDF, verify all 5 charts exist:

CHART_DIR="models/$MODEL_NAME/reports/charts"
MISSING_CHARTS=0
for CHART in chart_trtexec_bs1_vs_bsmax.png chart_trtexec_throughput.png \
             chart_ds_streams_vs_fps.png chart_trt_vs_ds.png chart_efficiency.png; do
  [ ! -f "$CHART_DIR/$CHART" ] && { echo "ERROR: Missing $CHART_DIR/$CHART"; MISSING_CHARTS=$((MISSING_CHARTS+1)); }
done
[ "$MISSING_CHARTS" -gt 0 ] && { echo "ERROR: $MISSING_CHARTS chart(s) missing — re-run 8c-1"; exit 1; }
echo "All 5 charts verified OK"

Then run the canonical pipeline script — this generates BOTH the HTML and PDF correctly:

# setup.sh apt-installs wkhtmltopdf INSIDE the container, so it does NOT survive the ephemeral
# `--rm` bootstrap container. Every later `docker run --rm` starts without it and the PDF step
# silently degrades to HTML-only. Ensure it is present in THIS invocation before rendering.
command -v wkhtmltopdf >/dev/null 2>&1 || { apt-get update -qq && apt-get install -y -qq wkhtmltopdf; }

python3 .claude/skills/deepstream-import-vision-model/scripts/report/md-to-html-pdf.py \
  models/$MODEL_NAME/reports/benchmark_report.md \
  .claude/skills/deepstream-import-vision-model/scripts/report/report-style.css \
  models/$MODEL_NAME/reports/ \
  $MODEL_NAME

This script uses report_style.css (navy #283593 headers, #e8eaf6 rows, #263238 code blocks), embeds charts as base64 data URIs, calls wkhtmltopdf internally, and outputs benchmark_report.html + benchmark_report_{model_name}.pdf.

NAMING RULES:

HTML: always benchmark_report.html (no model name suffix)

PDF: always benchmark_report_{model_name}.pdf (model name postfix required)

Verify PDF size is >500 KB (confirms charts embedded). Run all python commands with the shared venv active (source build/.venv_optimum/bin/activate); markdown and matplotlib are already installed there.

8c-4: Final Report Checklist and Timing

After generating markdown, HTML, and PDF, record step timing:

STEP8_END=$(date +%s.%N)
STEP8_DURATION=$(python3 -c "print(round($STEP8_END - $STEP8_START, 2))")   # bc is not in the container; python3 always is
echo "[Step 8] Report generation completed in ${STEP8_DURATION}s"

Before marking the report as complete, verify ALL of these exist:

reports/benchmark_report.md — markdown source (12 sections)
reports/benchmark_report.html — styled HTML (charts/ alongside)
reports/benchmark_report_{model_name}.pdf — PDF >500 KB (confirms charts embedded)
reports/benchmark_data.json — raw benchmark numbers
reports/charts/ — all 5 PNGs: chart_trtexec_bs1_vs_bsmax.png, chart_trtexec_throughput.png, chart_ds_streams_vs_fps.png, chart_trt_vs_ds.png, chart_efficiency.png
Charts: fixed filenames above — never rename or add model name suffix to charts

/deepstream-import-vision-model

referencesreport-generation.md

NV Import Vision Model Report -- Step 8

⛔ STRICT HTML+PDF RULE — NO EXCEPTIONS, NO DEVIATIONS