This roadmap adds server-side GPU preprocessing for YOLO inference. Each numbered step is one reviewable change with its own verification. Option 2 must not start until the Option 1 exit gate passes.
Option 1 is implemented and passed the local GPU acceptance gate on 2026-07-21: all three Triton models loaded, DALI tensor parity passed, HTTP and gRPC detection parity passed, invalid input remained controlled, and latency/GPU evidence was recorded. See the workflow and evidence. Promotion into a GPU CI environment remains a deployment follow-up. Phase 2 has not started.
Tritonic encoded image bytes
|
v
Triton ensemble model
|
+-- Option 1: DALI preprocessing model
| JPEG decode -> letterbox -> normalize -> HWC-to-CHW
|
`-- Option 2: C++/CUDA preprocessing backend
nvJPEG decode -> fused CUDA preprocessing kernel
|
v
TensorRT YOLO model -> existing neuriplo-tasks postprocessing -> rendering
An ensemble is exposed to Tritonic as a normal Triton model. Tritonic does not
need EnsembleRequest, EnsembleResponse, or inferEnsemble() types. The
client-side change is an encoded-image input path plus separate inner-model
metadata for neuriplo-tasks output postprocessing.
The Python backend preprocessing example is the repository-layout and ensemble-wiring reference, not the Option 1 implementation. Option 1 uses the Triton DALI backend for GPU preprocessing. The public contract follows NVIDIA's ensemble model documentation.
- One TensorRT YOLO model supported by
--model_type=yolo: YOLO11s, FP32 NCHW, 640x640 is the reference. - JPEG still images, batch size 1, HTTP and gRPC.
- Ensemble input:
IMAGE,TYPE_UINT8, shape[-1]. - Preprocessor output exactly matches the TensorRT input contract.
- TensorRT outputs pass through unchanged. Existing
neuriplo-tasksNMS, coordinate restoration, and rendering remain client-side. - Video, PNG, batching, segmentation, pose, server NMS, and shared-memory optimization are out of scope until the Option 1 exit gate passes.
Files: deploy/object_detection/yolo/ensemble/README.md
Record TensorRT input/output names, shapes, datatypes, channel order, letterbox rule, interpolation, normalization, and padding value.
Verify: Values match Triton metadata and the corresponding neuriplo-tasks
YOLO preprocessor.
Files: tests/data/ensemble/yolo/, integration-test manifest
Select small, wide, and tall JPEG fixtures. Save the current client-preprocessed YOLO tensors and final detections with explicit numeric tolerances.
Verify: The existing path reproduces the fixtures twice consecutively.
Files: integration-tests/yolo-ensemble/run.py
Measure preprocessing, request, server-compute, and end-to-end latency plus throughput. Keep correctness assertions separate from performance reporting.
Verify: Detection mismatches fail; success emits a machine-readable report.
Files: deploy/object_detection/yolo/ensemble/dali/model_repository/
Add configs and version directories for yolo_dali_preprocess, yolo_trt, and
yolo_dali_ensemble. Do not commit a TensorRT engine.
Verify: A layout test confirms every configured directory exists.
Files: deploy/object_detection/yolo/ensemble/dali/generate_pipeline.py
Build an inference-only pipeline with named external_source, GPU image decode,
exact YOLO letterbox, channel conversion, scaling/normalization, and CHW output.
Serialize the DALI artifact used by the repository.
Verify: Two clean generations produce a loadable pipeline with the declared input and output names.
Files: yolo_dali_preprocess/config.pbtxt
Declare variable-length TYPE_UINT8 input and the exact output consumed by
yolo_trt. Match the config input name to DALI external_source.
Verify: Triton strict model configuration loads the model without autocomplete.
Files: yolo_trt/config.pbtxt, setup script
Pin tensor names, profiles, max_batch_size, and instance placement. Add a setup
command that installs a user-supplied engine.
Verify: Direct yolo_trt inference accepts the golden preprocessed tensor.
Files: yolo_dali_ensemble/config.pbtxt
Map ensemble input to DALI, DALI output to TensorRT, and TensorRT outputs to the ensemble. Keep the ensemble version directory empty.
Verify: All three models are ready and one encoded-image request returns the declared YOLO tensors.
Files: include/tritonic/infra/config.hpp, src/main/ConfigManager.cpp, tests
Add --input_mode=preprocessed|encoded-image, default preprocessed. Reject
encoded mode for chat, temporal tasks, and unsupported multi-input tasks.
Verify: Parser tests cover default, valid, and invalid combinations.
Files: configuration, src/main/App.cpp, tests
Add --task_model=<name> for encoded mode. Query --model (ensemble) for request
I/O and --task_model (inner TensorRT model) to construct the task. Validate
ensemble outputs against the task-model outputs before inference.
Verify: Mock tests cover compatible metadata and fail fast on missing or incompatible metadata.
Files: focused input builder, unit tests
Read original JPEG bytes without decode/re-encode, set the variable-length
UINT8 request shape, and use the existing ITriton::infer() call.
Verify: Tests prove byte preservation and correct shapes for all fixtures.
Files: src/main/App.cpp, App tests
Bypass task_->preprocess() only in encoded mode. Convert ensemble outputs and
call the existing YOLO postprocessor with original image dimensions. Preserve
rendering and output files.
Verify: Existing path tests remain unchanged; a mock test proves one bypass per encoded request.
Compare the DALI component output with the CPU golden tensors. Diagnose letterbox rounding, interpolation, RGB order, padding, scaling, and layout before changing tolerances.
Verify: Every fixture passes documented absolute and relative tolerances.
Run both paths and compare count, class, confidence, and restored coordinates.
Verify: Both protocols pass; invalid bytes produce controlled errors without crashing Triton or Tritonic.
Files: ensemble README.md, root README.md
Pin compatible Triton/DALI images and document generation, engine installation, server startup, Tritonic commands, output, and cleanup. Verify every link.
Verify: A clean checkout succeeds without undocumented files or commands.
Option 1 is working only when:
- DALI, TensorRT, and ensemble models are ready.
- HTTP and gRPC encoded-JPEG tests pass.
- CPU and DALI preprocessing tensors meet recorded tolerances.
- The original client-preprocessed path still passes.
- Benchmark evidence covers both paths, GPU utilization, and latency. A speedup is not required for correctness acceptance, but regressions are recorded.
- The pinned workflow succeeds from a clean checkout.
Do not begin Phase 2 before this gate has CI or reproducible GPU acceptance evidence.
Choose padded uniform input or one request per sample with dynamic/ragged batching; do not leave the transport implicit.
Verify: Batch sizes 1, 2, maximum, and a short final batch match single-image results for mixed aspect ratios.
Choose and benchmark client JPEG encoding, raw frame tensors, or a video-aware decoder. Do not silently JPEG-encode frames.
Verify: Frame order, dimensions, colors, and detections match a fixed clip.
Add PNG after decoder tests, then qualify other YOLO detection exports one at a time. Segmentation, pose, and end-to-end NMS are separate contracts.
Verify: Each addition has metadata and golden-result entries.
Option 2 is a Triton custom C++ backend, not CUDA inside the Tritonic client. It preserves the DALI preprocessor I/O so the ensemble and client do not fork.
Files: deploy/object_detection/yolo/ensemble/cuda/CONTRACT.md
Copy the proven DALI names, shapes, datatypes, errors, and tolerances. Specify JPEG support and CUDA device/stream ownership.
Verify: One contract test targets either preprocessor by model name only.
Files: backends/yolo_preprocess/, CMake and container files
Implement lifecycle, validation, response allocation, and structured errors without image processing.
Verify: It loads, reports ready, rejects malformed requests, and passes lifecycle tests.
Decode into device memory on the backend CUDA stream. Reuse per-instance handles and buffers; reject corrupt or unsupported inputs.
Verify: Tests cover grayscale, RGB, wide, tall, truncated, and invalid JPEGs.
Implement letterbox resize, interpolation, channel conversion, padding, normalization, and HWC-to-CHW. Optimize/fuse only after parity.
Verify: Output matches DALI golden tensors and compute-sanitizer memcheck passes.
Process samples on the provided stream, reuse bounded instance buffers, handle partial failures, and avoid global device synchronization.
Verify: Batch and concurrency tests show no leaks or cross-request corruption.
Files: deploy/object_detection/yolo/ensemble/cuda/model_repository/
Add yolo_cuda_preprocess and yolo_cuda_ensemble with the same public I/O.
Pin compiler, CUDA, Triton backend SDK, nvJPEG, and server versions.
Verify: A clean container build loads every component model.
Run Option 1 tensor and detection parity tests against CUDA over HTTP and gRPC.
Verify: No CUDA-specific weakening of tests or tolerances.
Measure preprocessing, allocations, GPU utilization, end-to-end latency, and throughput across sizes, batches, and concurrency after warmup.
Verify: Results include hardware, driver, container, and commit identifiers.
Add size/overflow limits, cancellation, unload/reload, multi-instance, multi-GPU, stress, observability, and troubleshooting coverage.
Verify: Stress, sanitizer, malformed-input, reload, and clean-checkout gates pass.
- CUDA passes the unchanged Option 1 correctness suite.
- Sanitizer and stress tests have no memory, stream, or concurrency failures.
- Container build and deployment are reproducible.
- DALI-versus-CUDA evidence is published with environment identifiers.
- Documentation states the default implementation and measured reason.
GPU postprocessing is worthwhile when profiling shows CPU decode/NMS or transfer of raw YOLO heads is material. Prefer a TensorRT end-to-end export with supported NMS first; use a Triton C++/CUDA postprocessing backend only when that cannot implement the required YOLO output contract.
Record raw-output bytes, device-to-host and network transfer, CPU postprocessing latency, total latency, throughput, batch size, and concurrency.
Verify: Publish profiles for representative small and large YOLO outputs. Proceed only with a recorded target such as latency, throughput, or bandwidth.
Export or build an end-to-end TensorRT model that performs YOLO decode and NMS using APIs/plugins supported by the pinned TensorRT version. Expose only final boxes, scores, classes, and valid-detection count.
Verify: Final detections match the Phase 0 golden results and the engine loads without unpinned or deprecated plugin dependencies.
Reuse the Phase 2 backend infrastructure for decode, thresholding, sorting, and NMS on the Triton CUDA stream. Bound candidates and workspace explicitly.
Verify: Tensor parity, compute-sanitizer, worst-case candidate, batch, and concurrency tests pass without weakening tolerances.
Add preprocessing -> TensorRT -> GPU postprocessing ensemble variants. Add an
explicit final-detections output mode in Tritonic that bypasses
neuriplo-tasks NMS but preserves rendering and original-image coordinates.
Verify: Metadata validation rejects raw/final output mismatches before infer; HTTP and gRPC results match the existing client-postprocessed path.
Benchmark client CPU postprocessing, TensorRT-integrated postprocessing, and the CUDA backend with identical inputs and server settings.
Verify: Retain GPU postprocessing only when its end-to-end benefit justifies the extra model-specific maintenance. Document the selected default and fallback.
Profile prototype and mask-coefficient transfer plus CPU mask generation. On GPU, run detection decode/NMS first, gather coefficients only for retained detections, then perform prototype multiplication, sigmoid, crop, and resize. Do not materialize masks for every pre-NMS candidate.
Define a bounded output contract before implementation: boxes/classes/scores plus
one of fixed-size UINT8 masks, cropped instance masks, contours, or RLE. Include
transfer and client rendering cost when choosing the representation.
Verify: Detection and mask parity match the existing
neuriplo-tasks segmentation path across empty, single-object, many-object,
overlapping-object, and batch cases. The benchmark must include output bytes and
end-to-end latency, not only GPU kernel time.
Ensemble support is complete after Option 1 passes its exit gate. Options 2 and 3 are separately gated optimizations requiring real NVIDIA GPU evidence; code review or compilation alone is not completion.