/deepstream-import-vision-model

DS Run Pipeline -- Steps 6-7

Integrate a TensorRT model into DeepStream with parser, validation, and multi-stream benchmarks.

The model directory is: $ARGUMENTS

Pre-flight: Extract Variables

[ -z "$ARGUMENTS" ] && { echo "ERROR: No model directory provided. Usage: /deepstream-import-vision-model models/<model_name>/"; exit 1; }
MODEL_DIR="${ARGUMENTS%/}"
MODEL_NAME=$(basename "$MODEL_DIR")

# Find ONNX file (exclude _dynamic variants created during export)
ONNX_FILE=$(ls models/$MODEL_NAME/model/*.onnx 2>/dev/null | grep -v '_dynamic' | head -1)
[ -z "$ONNX_FILE" ] && { echo "ERROR: No ONNX file found in models/$MODEL_NAME/model/ — run Steps 1-3 first (references/model-acquire.md)"; exit 1; }
MODEL_FILENAME=$(basename "$ONNX_FILE" .onnx)

# Engine + MAX_BS from the shared resolver (see scripts/model/resolve-engine.sh).
eval "$(bash .claude/skills/deepstream-import-vision-model/scripts/model/resolve-engine.sh "$MODEL_NAME")" || exit 1

# Read PEAK_GPU_STREAMS from trtexec Step 5b log — fixed filename, no timestamp, no wildcard
TRTEXEC_LOG="models/$MODEL_NAME/benchmarks/b${MAX_BS}/trtexec_b${MAX_BS}.log"
[ -f "$TRTEXEC_LOG" ] || { echo "ERROR: trtexec log not found at $TRTEXEC_LOG — run Steps 4-5 first (references/engine-build.md)"; exit 1; }
QPS_BS_MAX=$(grep -oP 'Throughput:\s*\K[0-9.]+' "$TRTEXEC_LOG" | tail -1)
read IMGS_PER_SEC PEAK_GPU_STREAMS < <(python3 -c "
import math
imgs = float('$QPS_BS_MAX') * $MAX_BS
print(round(imgs, 2), int(math.floor(imgs / 30)))
")

# Read spatial dimensions from ONNX inspection
INSPECT_OUT=$(python3 .claude/skills/deepstream-import-vision-model/scripts/model/inspect-onnx.py "$ONNX_FILE")
INPUT_NAME=$(echo "$INSPECT_OUT" | grep -oP 'input_name:\s*\K\S+')
H=$(echo "$INSPECT_OUT"          | grep -oP 'height:\s*\K[0-9]+')
W=$(echo "$INSPECT_OUT"          | grep -oP 'width:\s*\K[0-9]+')
[ -z "$INPUT_NAME" ] && { echo "ERROR: could not parse INPUT_NAME from inspect output"; exit 1; }
[ -z "$H" ]          && { echo "ERROR: could not parse H — dynamic spatial dims? Set H manually"; exit 1; }
[ -z "$W" ]          && { echo "ERROR: could not parse W — dynamic spatial dims? Set W manually"; exit 1; }

# Detect installed CUDA version for parser compilation
CUDA_VER=$(ls /usr/local/ 2>/dev/null | grep -oP '^cuda-\K[0-9]+\.[0-9]+$' | sort -V | tail -1)
[ -z "$CUDA_VER" ] && CUDA_VER=13.2
echo "CUDA_VER=$CUDA_VER"

# Count labels
[ -f "models/$MODEL_NAME/config/labels.txt" ] || { echo "ERROR: labels.txt not found — run Steps 1-3 first (references/model-acquire.md)"; exit 1; }
NUM_LABELS=$(wc -l < models/$MODEL_NAME/config/labels.txt)

# Parser function suffix: PascalCase of MODEL_NAME, sanitized for C++ identifiers
# e.g. yolov8n→Yolov8n  rtdetr-l→RtdetrL  grounding-dino-base→GroundingDinoBase
PARSER_FUNC_SUFFIX=$(python3 -c "
import re
parts = re.sub(r'[^a-zA-Z0-9]', ' ', '$MODEL_NAME').split()
print(''.join(p.capitalize() for p in parts))
")
# Sanitize MODEL_NAME for use in C++ source/library filenames — mirrors PARSER_FUNC_SUFFIX logic.
# e.g. rtdetr-l → rtdetr_l  grounding-dino-base → grounding_dino_base
# printf, NOT echo: echo appends a newline that `tr -c` turns into a trailing '_', yielding
# e.g. `rtdetr_r50vd_` and a custom-lib-path that never matches the Makefile's TARGET_LIB.
MODEL_NAME_SAFE=$(printf '%s' "$MODEL_NAME" | tr -c 'A-Za-z0-9' '_')

# Video source — default is sample_720p.mp4 (MANDATORY). Never autonomously substitute
# sample_1080p_h264.mp4 or any other file. DS_VIDEO may only be set when the user explicitly
# provides a custom video path; it is not a licence to pick a different resolution.
VIDEO="${DS_VIDEO:-/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4}"
[ -f "$VIDEO" ] || {
  echo "ERROR: Video file not found: $VIDEO"
  echo "  Fix 1: Set DS_VIDEO=/path/to/sample_720p.mp4 before running"
  echo "  Fix 2: Install DeepStream samples (DeepStream 9.1 samples): apt-get install deepstream-9.1-samples"
  exit 1
}

echo "Model:            $MODEL_NAME"
echo "ONNX:             $ONNX_FILE  (input=$INPUT_NAME, ${H}x${W})"
echo "Engine:           $ENGINE  (MAX_BS=$MAX_BS)"
echo "PEAK_GPU_STREAMS: $PEAK_GPU_STREAMS  (floor($IMGS_PER_SEC img/s / 30))"
echo "Labels:           $NUM_LABELS classes"

All subsequent commands use these variables — never hardcoded paths or template placeholders.

Step 6: DeepStream Integration

STEP6_START=$(date +%s.%N)

6a: Inspect Model Output Format

Verify output tensor shapes and value ranges before writing the parser:

python3 -c "
import onnxruntime as ort, numpy as np
sess = ort.InferenceSession('$ONNX_FILE')
inp = sess.get_inputs()[0]
out = sess.get_outputs()
print(f'Input: {inp.name} shape={inp.shape}')
for o in out: print(f'Output: {o.name} shape={o.shape}')
dummy = np.random.randn(*[d if isinstance(d,int) else 1 for d in inp.shape]).astype(np.float32)
result = sess.run(None, {inp.name: dummy})
for i,r in enumerate(result): print(f'Output[{i}] range: [{r.min():.4f}, {r.max():.4f}]')
"

CRITICAL: Determine the correct net-scale-factor from the output ranges and model family:

Wrong scale factor = zero detections. Always verify with KITTI dump (Step 6g) before benchmarks.

6b: Write Custom Bounding Box Parser

Create models/$MODEL_NAME/parser/nvdsinfer_custombboxparser_${MODEL_NAME_SAFE}.cpp:

extern "C"
bool NvDsInferParseCustom${PARSER_FUNC_SUFFIX}(
    std::vector<NvDsInferLayerInfo> const &outputLayersInfo,
    NvDsInferNetworkInfo const &networkInfo,
    NvDsInferParseDetectionParams const &detectionParams,
    std::vector<NvDsInferObjectDetectionInfo> &objectList);

CHECK_CUSTOM_PARSE_FUNC_PROTOTYPE(NvDsInferParseCustom${PARSER_FUNC_SUFFIX});

Parser implementation rules:

• Include nvdsinfer_custom_impl.h and use NvDsInferObjectDetectionInfo (classId, left, top, width, height, detectionConfidence)
• Decode model-specific output format into pixel-space bounding boxes:
  • YOLOX-style [N, num_anchors, 5+C]: decode grid offsets, exp(w/h), objectness×class_score
  • SSD-style [N, num_dets, 6]: extract class, confidence, normalized → pixel coords
  • YOLO with BatchedNMS: parse keepCount, bboxes, scores, classes from 4 output layers
• Clip all coordinates to [0, networkInfo.width-1] and [0, networkInfo.height-1]
• Use detectionParams.perClassPreclusterThreshold for confidence filtering
• NMS: Dense heads → cluster-mode=2 (DeepStream NMS). Fused TRT NMS → cluster-mode=4
• Sanity check for undecoded output: if bbox values land in [0, 3], the parser is reading grid-space offsets. Most models need (raw + grid_offset) * stride for cx/cy and exp(raw) * stride for w/h. Verify raw output ranges with Python/ONNX Runtime before writing the parser.
• Reference: /opt/nvidia/deepstream/deepstream/sources/libs/nvdsinfer_customparser/nvdsinfer_custombboxparser.cpp; Header: sources/includes/nvdsinfer_custom_impl.h

Model-family parser patterns

• DETR / Conditional DETR: outputs logits [B, num_queries, num_classes+1] and pred_boxes [B, num_queries, 4]. Boxes are (cx, cy, w, h) normalized to [0,1] — convert to (left, top, width, height) in pixels. Use softmax (not sigmoid) on logits. Background class is the LAST index (e.g., index 91 for a 92-class DETR, despite config.json showing "0": "N/A"). Skip the background class when iterating. DETR uses Hungarian matching — NMS is not needed; set cluster-mode=4 (not nms-iou-threshold=0.0, which is a legacy key).
• OWL-ViT / CLIP-based zero-shot detectors: outputs logits [B, num_patches, num_classes] and pred_boxes [B, num_patches, 4]. Sigmoid activation (per-class independent scoring, not softmax). Boxes are (cx, cy, w, h) normalized [0,1]. Use cluster-mode=2 (NMS with IoU threshold). CLIP preprocessing: net-scale-factor=0.01459, offsets=122.77;116.75;104.09. Confidence threshold 0.10 works well for general detection; lower to 0.05 for recall-focused tasks.
• HF RT-DETR preprocessing quirk: RTDetrImageProcessor may have do_normalize=false even though image_mean/image_std fields exist. When do_normalize=false, the model expects [0,1] scaled input — set net-scale-factor=1/255 with no offsets. The ONNX export does NOT bake normalization into the first Conv layer. Verify with ONNX Runtime on a real frame before debugging nvinfer.

NGC TAO models — use the built-in parser library

NVIDIA NGC TAO models (trafficcamnet, peoplenet, TrafficCamNet Transformer Lite, etc.) ship with TAO-specific parsers pre-compiled into a system library:

• Library path: /opt/nvidia/deepstream/deepstream/lib/libnvds_infercustomparser.so — NOT libnvds_infercustomparser_tao.so (even if the NGC YAML config suggests it).
• Custom parse function names: NvDsInferParseCustomDDETRTAO, NvDsInferParseCustomRTDETRTAO, etc.
• No custom parser compilation needed — point custom-lib-path at the system library and parse-bbox-func-name at the TAO function.
• KITTI dump from deepstream-app may emit zero-valued bbox coordinates for DETR/RT-DETR parsers even when detections are correct. Verify visually with JPEG frame extraction instead.

network-type vs model-type — use network-type=0

• model-type is a legacy/unknown key — nvinfer ignores it with a warning.
• network-type=0 (Detector) is required to invoke parse-bbox-func-name.
• network-type=100 (Other) does NOT invoke the custom bbox parser — it requires output-tensor-meta=1 for external post-processing.
• Symptom of the wrong key: custom parse function is never called (zero detections, no parser debug output) — check that network-type=0 is set.

6c: Create Makefile

Write models/$MODEL_NAME/parser/Makefile using Python to guarantee literal TAB characters in recipe lines (heredoc in bash can produce spaces, which break make):

python3 - << EOF
model = '$MODEL_NAME'
model_safe = '$MODEL_NAME_SAFE'
content = (
    "DEEPSTREAM_DIR ?= /opt/nvidia/deepstream/deepstream\n"
    "CUDA_VER ?= 12.8\n"
    "CC := g++\n"
    "CFLAGS := -Wall -std=c++11 -shared -fPIC\n"
    "CFLAGS += -I\$(DEEPSTREAM_DIR)/sources/includes -I/usr/local/cuda-\$(CUDA_VER)/include\n"
    "LIBS := -lnvinfer\n"
    "LFLAGS := -Wl,--start-group \$(LIBS) -Wl,--end-group\n"
    f"SRCFILES := nvdsinfer_custombboxparser_{model_safe}.cpp\n"
    f"TARGET_LIB := libnvdsinfer_{model_safe}_parser.so\n"
    "\n"
    "all: \$(TARGET_LIB)\n"
    "\$(TARGET_LIB): \$(SRCFILES)\n"
    "\t\$(CC) -o \$@ \$^ \$(CFLAGS) \$(LFLAGS)\n"   # TAB required by make
    "clean:\n"
    "\trm -rf \$(TARGET_LIB)\n"                       # TAB required by make
)
with open(f'models/{model}/parser/Makefile', 'w') as f:
    f.write(content)
print(f"Makefile written: models/{model}/parser/Makefile")
EOF

6d: Build Parser Library

make -C models/$MODEL_NAME/parser \
  DEEPSTREAM_DIR=/opt/nvidia/deepstream/deepstream \
  CUDA_VER=$CUDA_VER

# Verify the symbol is exported
nm -D models/$MODEL_NAME/parser/libnvdsinfer_${MODEL_NAME_SAFE}_parser.so | grep NvDsInferParseCustom

6e: Create nvinfer Config File

cat > models/$MODEL_NAME/config/config_infer_primary_${MODEL_NAME}.txt << EOF
[property]
gpu-id=0
net-scale-factor=0.00392156862745098
model-color-format=0
onnx-file=../model/${MODEL_FILENAME}.onnx
model-engine-file=../benchmarks/engines/${MODEL_FILENAME}_dynamic_b${MAX_BS}.engine
labelfile-path=labels.txt
batch-size=1
network-mode=2
num-detected-classes=${NUM_LABELS}
process-mode=1
interval=0
gie-unique-id=1
network-type=0
custom-lib-path=../parser/libnvdsinfer_${MODEL_NAME_SAFE}_parser.so
parse-bbox-func-name=NvDsInferParseCustom${PARSER_FUNC_SUFFIX}
# 2=DeepStream NMS (dense heads: YOLO, SSD). Use 4 if engine has fused NMS output
cluster-mode=2
infer-dims=3;${H};${W}
maintain-aspect-ratio=1

[class-attrs-all]
topk=200
nms-iou-threshold=0.45
pre-cluster-threshold=0.25
EOF

Path note: All paths are relative to the config/ directory where this file lives. net-scale-factor defaults to 1/255 — update to 1.0 if the model expects 0–255 input (verify via Step 6a).

Verify label count matches:

echo "labels.txt: $NUM_LABELS classes -> num-detected-classes=$NUM_LABELS"

6f: Single-Stream Visual Validation

ENCODER RULE: Primary encoder is nvv4l2h264enc (NVENC via V4L2) → .mp4. x264enc and openh264enc are prohibited. On systems where /dev/v4l2-nvenc is unavailable, the approved fallback is theoraenc + oggmux (LGPL; both ship in gst-plugins-base) → .ogv. If theoraenc/oggmux are absent, video creation is skipped. Use .claude/skills/deepstream-import-vision-model/scripts/deepstream/ds-single-stream.sh which handles this automatically and emits a DS_SINGLE_STREAM_MODE= marker the report parser reads.

Primary (NVENC available):

mkdir -p models/$MODEL_NAME/samples

GST_DEBUG=1 gst-launch-1.0 \
  filesrc location=$VIDEO ! \
  qtdemux ! queue leaky=downstream ! h264parse ! queue ! nvv4l2decoder ! queue ! \
  m.sink_0 nvstreammux name=m batch-size=1 width=1280 height=720 ! queue ! \
  nvinfer config-file-path=models/$MODEL_NAME/config/config_infer_primary_${MODEL_NAME}.txt ! queue ! \
  nvvideoconvert ! 'video/x-raw(memory:NVMM),format=RGBA' ! \
  nvdsosd ! nvvideoconvert ! 'video/x-raw(memory:NVMM),format=NV12' ! \
  nvv4l2h264enc ! h264parse ! mp4mux ! \
  filesink location=models/$MODEL_NAME/samples/${MODEL_NAME}_output.mp4 sync=0

Fallback (NVENC unavailable — /dev/v4l2-nvenc missing, theoraenc/oggmux present):

Output extension switches from .mp4 to .ogv (Ogg/Theora container). theoraenc consumes planar I420, not NV12.

GST_DEBUG=1 gst-launch-1.0 \
  filesrc location=$VIDEO ! \
  qtdemux ! queue leaky=downstream ! h264parse ! queue ! nvv4l2decoder ! queue ! \
  m.sink_0 nvstreammux name=m batch-size=1 width=1280 height=720 ! queue ! \
  nvinfer config-file-path=models/$MODEL_NAME/config/config_infer_primary_${MODEL_NAME}.txt ! queue ! \
  nvvideoconvert ! nvdsosd ! nvvideoconvert ! \
  "video/x-raw, format=I420" ! theoraenc quality=48 ! oggmux ! \
  filesink location=models/$MODEL_NAME/samples/${MODEL_NAME}_output.ogv sync=0

Extract a frame to visually confirm bounding boxes — auto-detect which output file exists:

SAMPLE_OUT=$(ls models/$MODEL_NAME/samples/${MODEL_NAME}_output.{mp4,ogv} 2>/dev/null | head -1)

case "$SAMPLE_OUT" in
  *.mp4)
    gst-launch-1.0 \
      filesrc location="$SAMPLE_OUT" ! \
      qtdemux ! h264parse ! nvv4l2decoder ! videoconvert ! "video/x-raw,format=RGB" ! \
      jpegenc quality=95 ! \
      multifilesink location=models/$MODEL_NAME/samples/frame_%04d.jpg max-files=3
    ;;
  *.ogv)
    gst-launch-1.0 \
      filesrc location="$SAMPLE_OUT" ! \
      oggdemux ! theoradec ! videoconvert ! "video/x-raw,format=RGB" ! \
      jpegenc quality=95 ! \
      multifilesink location=models/$MODEL_NAME/samples/frame_%04d.jpg max-files=3
    ;;
esac

If no detections appear, the most common cause is wrong net-scale-factor — update the config and re-run.

6g: KITTI Dump — Verify Detections Programmatically

Run a KITTI dump to confirm detections exist before multi-stream benchmarks.

Note: gie-kitti-output-dir is a deepstream-app [application] property — it is not read by nvinfer directly. Appending it to the nvinfer config and running a gst-launch-1.0 ... nvinfer ... pipeline silently produces zero KITTI files. Use the ds-kitti-dump.sh helper, which wraps deepstream-app with the correct [application] section.

mkdir -p models/$MODEL_NAME/samples/kitti_output

bash .claude/skills/deepstream-import-vision-model/scripts/deepstream/ds-kitti-dump.sh \
  models/$MODEL_NAME/config/config_infer_primary_${MODEL_NAME}.txt \
  models/$MODEL_NAME/samples/kitti_output \
  100 \
  "$VIDEO"

# Summarise detection results
KITTI_FILES=$(ls models/$MODEL_NAME/samples/kitti_output/*.txt 2>/dev/null | wc -l)
echo "KITTI frames written: $KITTI_FILES"
echo "Top detected classes:"
cat models/$MODEL_NAME/samples/kitti_output/*.txt 2>/dev/null \
  | awk '{print $1}' | sort | uniq -c | sort -rn | head -10

Validation gate: If KITTI_FILES == 0 or all files are empty, detections are broken. Do NOT proceed to Step 7.

# MANDATORY hard stop — do not comment out or remove this check
if [ "$KITTI_FILES" -eq 0 ]; then
  echo "ERROR: KITTI validation FAILED — zero detection files written."
  echo "Fix net-scale-factor, parser output format, or config before retrying."
  echo "Do NOT proceed to Step 7 benchmarks with broken detections."
  exit 1
fi
FRAMES_WITH_DETECTIONS=$(grep -rl '.' models/$MODEL_NAME/samples/kitti_output/ 2>/dev/null | wc -l)
DETECTION_RATE=$(python3 -c "print(round($FRAMES_WITH_DETECTIONS/$KITTI_FILES*100,1))")
echo "Detection rate: $FRAMES_WITH_DETECTIONS / $KITTI_FILES frames = ${DETECTION_RATE}%"
if python3 -c "exit(0 if $FRAMES_WITH_DETECTIONS/$KITTI_FILES >= 0.9 else 1)"; then
  echo "KITTI validation PASSED (>= 90% frames with detections)"
else
  echo "ERROR: Detection rate ${DETECTION_RATE}% < 90% threshold. Fix parser before proceeding."
  exit 1
fi

STEP6_END=$(date +%s.%N)
STEP6_DURATION=$(python3 -c "print(round($STEP6_END - $STEP6_START, 2))")   # bc is not in the container; python3 always is
echo "[Step 6] completed in ${STEP6_DURATION}s"

DeepStream Troubleshooting

Step 7: Multi-Stream DeepStream Benchmark

7b: Create DS Benchmark Config

Create one nvinfer config for all DS benchmark runs. batch-size is overridden at runtime via the nvinfer GStreamer element property.

This config is derived from the Step 6e config rather than re-authored — every [property] and [class-attrs-all] value is identical. Only the relative paths and batch-size differ, because this copy lives one directory deeper (benchmarks/ds/ instead of config/). Deriving it keeps the two from drifting apart:

mkdir -p models/$MODEL_NAME/benchmarks/ds

sed -e 's#^onnx-file=\.\./#onnx-file=../../#' \
    -e 's#^model-engine-file=\.\./benchmarks/engines/#model-engine-file=../engines/#' \
    -e 's#^labelfile-path=labels\.txt#labelfile-path=../../config/labels.txt#' \
    -e "s#^batch-size=1\$#batch-size=${MAX_BS}#" \
    -e 's#^custom-lib-path=\.\./#custom-lib-path=../../#' \
    models/$MODEL_NAME/config/config_infer_primary_${MODEL_NAME}.txt \
  > models/$MODEL_NAME/benchmarks/ds/config_infer_ds_${MODEL_NAME}.txt

The five derived keys, for reference:

Path note: Paths are relative to benchmarks/ds/ where this config lives.

Queue Placement Rules (MANDATORY)

Every pipeline stage must be separated by queue elements. Use leaky=downstream after qtdemux to drop excess frames under GPU saturation; all other queues use no leaky setting (threading only). Always set batched-push-timeout=-1 on nvstreammux. Never include nvmultistreamtiler, nvdsosd, or extra nvvideoconvert in benchmark runs — only use for single-stream visual validation (Step 6f).

7c: Two-Run DS Benchmark

Only 2 DS pipeline runs characterise DS overhead vs trtexec.

Both runs go through deepstream-app with [application] enable-perf-measurement=1 (wrapped by .claude/skills/deepstream-import-vision-model/scripts/deepstream/ds-perf-run.sh). FPS is parsed from the canonical **PERF: lines DeepStream emits at the configured measurement interval. This replaces the older gst-launch-1.0 ... ! fpsdisplaysink path so the runtime no longer depends on gstreamer1.0-plugins-bad.

PERF line format: **PERF: <fps_run> (<fps_avg>) — one float per active source. The helper script averages the per-stream instantaneous FPS across the last few measurement windows; the parser below mirrors that contract.

DS Run 1 — Calibration at PEAK_GPU_STREAMS streams:

CRITICAL: Use $PEAK_GPU_STREAMS directly. Do NOT pre-apply any efficiency discount (no ×0.6, ×0.7, etc.). Run 1 measures the real overhead — do not guess it.

Log filenames are fixed — no timestamp variation. Always ds_s${N}_run1.log and ds_s${N}_run2.log in benchmarks/ds/. The report-generation skill reads these exact paths.

# Hard constraint: num_streams <= engine max batch size — always
N=$(python3 -c "print(min($PEAK_GPU_STREAMS, $MAX_BS))")
LOG_RUN1="models/$MODEL_NAME/benchmarks/ds/ds_s${N}_run1.log"

STEP7_RUN1_START=$(date +%s.%N)
bash .claude/skills/deepstream-import-vision-model/scripts/deepstream/ds-perf-run.sh \
  models/$MODEL_NAME/benchmarks/ds/config_infer_ds_${MODEL_NAME}.txt \
  "$N" \
  "$LOG_RUN1" \
  "$VIDEO"

FPS_RUN1=$(grep -oP '\*\*PERF:\s*\K[0-9.]+' "$LOG_RUN1" | tail -10 | python3 -c "
import sys; vals=[float(l) for l in sys.stdin if l.strip()]; print(round(sum(vals)/len(vals),2) if vals else 0)")
python3 -c "exit(0 if float('$FPS_RUN1') > 0 else 1)" || \
  { echo "ERROR: FPS parsing failed for Run 1 — check $LOG_RUN1"; exit 1; }

TOTAL_FPS_RUN1=$(python3 -c "print(round(float('$FPS_RUN1') * $N, 2))")
RT_STREAMS=$(python3 -c "import math; print(min(int(math.floor(float('$TOTAL_FPS_RUN1') / 30)), $MAX_BS))")
echo "DS Run 1: $N streams | FPS/stream=$FPS_RUN1 | total=$TOTAL_FPS_RUN1 img/s | RT_STREAMS=$RT_STREAMS"
STEP7_RUN1_END=$(date +%s.%N)
STEP7_RUN1_DURATION=$(python3 -c "print(round($STEP7_RUN1_END - $STEP7_RUN1_START, 2))")   # bc is not in the container; python3 always is
echo "[Step 7 Run 1] completed in ${STEP7_RUN1_DURATION}s"

DS Run 2 — Validation at RT_STREAMS:

N=$RT_STREAMS
LOG_RUN2="models/$MODEL_NAME/benchmarks/ds/ds_s${N}_run2.log"

STEP7_RUN2_START=$(date +%s.%N)
bash .claude/skills/deepstream-import-vision-model/scripts/deepstream/ds-perf-run.sh \
  models/$MODEL_NAME/benchmarks/ds/config_infer_ds_${MODEL_NAME}.txt \
  "$N" \
  "$LOG_RUN2" \
  "$VIDEO"

FPS_RUN2=$(grep -oP '\*\*PERF:\s*\K[0-9.]+' "$LOG_RUN2" | tail -10 | python3 -c "
import sys; vals=[float(l) for l in sys.stdin if l.strip()]; print(round(sum(vals)/len(vals),2) if vals else 0)")
python3 -c "exit(0 if float('$FPS_RUN2') > 0 else 1)" || \
  { echo "ERROR: FPS parsing failed for Run 2 — check $LOG_RUN2"; exit 1; }

TOTAL_FPS_RUN2=$(python3 -c "print(round(float('$FPS_RUN2') * $N, 2))")
RT_CONFIRMED=$(python3 -c "print('YES' if float('$FPS_RUN2') >= 30 else 'NO')")
echo "DS Run 2: $N streams | FPS/stream=$FPS_RUN2 | total=$TOTAL_FPS_RUN2 img/s | Real-time: $RT_CONFIRMED"
STEP7_RUN2_END=$(date +%s.%N)
STEP7_RUN2_DURATION=$(python3 -c "print(round($STEP7_RUN2_END - $STEP7_RUN2_START, 2))")   # bc is not in the container; python3 always is
echo "[Step 7 Run 2] completed in ${STEP7_RUN2_DURATION}s"

NVDEC saturation on fast nano models: very fast models (YOLO-nano family, etc.) can saturate NVDEC before GPU. Symptom: DS aggregate FPS plateaus at the same value regardless of stream count (e.g., 6,976 at 128 streams, 7,060 at 200 streams). In this case, PEAK_GPU_STREAMS from trtexec is an overestimate — Run 1 at that count will show fps/stream well below 30. The RT_STREAMS = floor(TOTAL_FPS_RUN1 / 30) formula above produces the correct NVDEC-limited ceiling. Do not pre-apply an efficiency factor to PEAK_GPU_STREAMS to compensate — the 2-run method measures overhead, it does not guess it.

If Run 2 is still not real-time (FPS/stream < 30): converge to the true ceiling — do NOT halve.

Why not halve: halving overshoots badly. If Run 2 at 38 streams measures 29.6 fps/stream (total 1124.8 img/s — only 1.3% under real-time), the real ceiling is ~37 streams, not 19. Halving to 19 discards ~half the GPU's real capacity and reports a misleadingly low number. Instead, recompute the target from Run 2's measured total throughput (floor(TOTAL_FPS_RUN2 / 30)), which is strictly below the count that just failed, then step down by 1 until real-time. This lands on the true ceiling in 1–2 short retries.

# Bounded convergence: recompute from measured throughput, then decrement by 1.
RETRIES=0
while [ "$RT_CONFIRMED" = "NO" ] && [ "$RETRIES" -lt 5 ]; do
  # First correction: jump to floor(measured_total/30); afterwards step down by 1.
  NEXT=$(python3 -c "import math; print(int(math.floor(float('$TOTAL_FPS_RUN2') / 30)))")
  [ "$NEXT" -ge "$N" ] && NEXT=$((N - 1))   # guarantee strict progress below the failing count
  RT_STREAMS=$(python3 -c "print(max(1, $NEXT))")
  N=$RT_STREAMS
  echo "Run 2 not real-time (fps/stream=$FPS_RUN2) — converging to $N streams"
  LOG_RUN2="models/$MODEL_NAME/benchmarks/ds/ds_s${N}_run2.log"
  bash .claude/skills/deepstream-import-vision-model/scripts/deepstream/ds-perf-run.sh \
    models/$MODEL_NAME/benchmarks/ds/config_infer_ds_${MODEL_NAME}.txt \
    "$N" \
    "$LOG_RUN2" \
    "$VIDEO"
  FPS_RUN2=$(grep -oP '\*\*PERF:\s*\K[0-9.]+' "$LOG_RUN2" | tail -10 | python3 -c "
import sys; vals=[float(l) for l in sys.stdin if l.strip()]; print(round(sum(vals)/len(vals),2) if vals else 0)")
  TOTAL_FPS_RUN2=$(python3 -c "print(round(float('$FPS_RUN2') * $N, 2))")
  RT_CONFIRMED=$(python3 -c "print('YES' if float('$FPS_RUN2') >= 30 else 'NO')")
  RETRIES=$((RETRIES + 1))
  echo "Retry $RETRIES: $N streams | FPS/stream=$FPS_RUN2 | Real-time: $RT_CONFIRMED"
  [ "$N" -le 1 ] && break
done

CONSTRAINT: num_streams <= engine_max_bs always. Already enforced above via min(RT_STREAMS, MAX_BS).

TRTEXEC_QPS=$(grep -oP 'Throughput:\s*\K[0-9.]+' "$TRTEXEC_LOG" | tail -1)
TRTEXEC_IMGS=$(python3 -c "print(round(float('$TRTEXEC_QPS') * $MAX_BS, 2))")
DS_EFF_RUN1=$(python3 -c "print(round(float('$TOTAL_FPS_RUN1') / float('$TRTEXEC_IMGS') * 100, 1))")
DS_EFF_RUN2=$(python3 -c "print(round(float('$TOTAL_FPS_RUN2') / float('$TRTEXEC_IMGS') * 100, 1))")

Timing and Output Summary

TOTAL_67_DURATION=$(python3 -c "print(round($STEP6_DURATION + $STEP7_RUN1_DURATION + $STEP7_RUN2_DURATION, 2))")   # bc is not in the container; python3 always is

When complete, print:

=== DeepStream Integration Complete ===
Model: $MODEL_NAME | Engine: $ENGINE
trtexec: $TRTEXEC_IMGS img/s @ BS=$MAX_BS
DS Run 1 (PEAK): $PEAK_GPU_STREAMS streams | $FPS_RUN1 fps/s | eff $DS_EFF_RUN1%
DS Run 2 (RT):   $RT_STREAMS streams | $FPS_RUN2 fps/s | RT: $RT_CONFIRMED | eff $DS_EFF_RUN2%
Timing: Step6=${STEP6_DURATION}s Run1=${STEP7_RUN1_DURATION}s Run2=${STEP7_RUN2_DURATION}s Total=${TOTAL_67_DURATION}s
Ready for: Step 8 — read references/report-generation.md models/$MODEL_NAME/