TritonIC - C++ Inference Client

TritonIC is a C++ application that supports two complementary inference modes:

• Triton backend — computer vision tasks (object detection, segmentation, classification, optical flow, pose, depth) via the NVIDIA Triton Inference Server over HTTP or gRPC.
• Chat backend — text and multimodal generation via any OpenAI-compatible /v1/chat/completions endpoint (Ollama, llama.cpp, SGLang, vLLM, OpenAI, etc.).

The two backends are complementary: use Triton for real-time computer vision at high throughput; use the Chat backend for VLMs and LLMs without a Triton server.

🚧 Status: Under Development — expect frequent updates.

Table of Contents

• Project Structure
• Architecture
• Tested Models
  • Object Detection · Instance Segmentation · Classification
  • Optical Flow · Open Vocabulary Detection · Pose Estimation
  • Video Classification · Depth Estimation · Image Understanding (VLM)
• Build Client Libraries
• Dependencies
• Development Setup
  • Pre-commit Hooks
  • Build and Compile
• Tasks
• Notes
• Deploying Models
  • Model Repository Structure
  • Starting Triton Server
• Running Inference
  • Command-line inference
  • Shared Memory Support
  • Ensemble models
  • Docker scripts
  • Chat backend
• Docker Support
• Kubernetes Deployment
• Demo
• References
• Feedback

Full documentation index: docs/README.md

Project Structure

tritonic/
├── src/
│   ├── main/                     # Entry point (client.cpp), App, Logger, ConfigManager
│   ├── triton/                   # Triton client (Triton.hpp/.cpp, forwarding headers)
│   ├── chat/                     # OpenAI-compatible backend (ChatBackend, ChatSession)
│   └── common/                   # Shared forwarding headers
├── include/
│   ├── tritonic/                 # Canonical namespaced headers
│   │   ├── core/                 # types.hpp, interfaces.hpp
│   │   ├── triton/               # model_info.hpp, itriton.hpp, triton_backend.hpp
│   │   ├── chat/                 # ichat_backend.hpp
│   │   └── infra/                # logger.hpp, config.hpp, config_manager.hpp
│   └── *.hpp                     # Backward-compat forwarding headers
├── deploy/                       # Model export scripts (per task type)
├── scripts/                      # Docker, setup, and utility scripts
├── config/                       # Configuration files
├── docs/                         # Documentation and guides
├── labels/                       # Label files (COCO, ImageNet, etc.)
├── data/                         # Data files (images, videos, outputs)
└── tests/                        # Unit and integration tests

CMake Fetched Dependencies:

• neuriplo-tasks (v0.7.0) — model pre/post processing and task management

Architecture

TritonIC selects an inference backend at startup via --backend:

--backend	Requires	Best for
triton (default)	NVIDIA Triton server	CV tasks — detection, segmentation, classification, optical flow, pose, depth
chat	Any OpenAI-compatible server	LLMs, VLMs, multimodal chat

The two modes are not competing — Triton handles binary tensor workloads at real-time throughput, while the Chat backend handles text/image generation over a REST API. Choose based on your model and server.

Both implement the common tritonic::core::IInferenceBackend interface (Strategy pattern), enabling clean dependency injection and unit testing without live servers.

Note: "backend" here refers to tritonic's server selection (--backend=triton vs --backend=chat). This is distinct from the Triton server's own framework backends (TensorRT, ONNX Runtime, etc.), which are configured server-side.

For full code structure and namespace layout see AGENTS.md.

The YOLO DALI ensemble can accept original JPEG bytes and run decode, letterboxing, normalization, and TensorRT inference on the server. See the reproducible GPU-preprocessing workflow and its atomic implementation roadmap.

Tested Models

Object Detection

• YOLOv5
• YOLOv6
• YOLOv7
• YOLOv8/YOLO11/YOLO26
• YOLOv9
• YOLOv10
• YOLOv12
• YOLO-NAS
• RT-DETR
• RT-DETRv2
• RT-DETRv4
• D-FINE
• DEIM
• DEIMv2
• RF-DETR

Instance Segmentation

• YOLOv5
• YOLOv8/YOLO11/YOLO26
• YOLOv10
• YOLOv12
• RF-DETR-Seg

Classification

• Torchvision Models
• TensorFlow-Keras Models
• Hugging Face Vision Transformers (ViT)

Optical Flow

• RAFT

Open Vocabulary Detection

• OWLv2
• OWL-ViT
• Grounding DINO

Pose Estimation

• YOLOv5 Pose
• YOLOv8/YOLO11/YOLO26 Pose
• ViTPose
• RF-DETR Keypoints (single-stage person keypoints, 17 COCO keypoints)
• YOLOv5 Pose
• YOLOv8/YOLO11/YOLO26 Pose

Video Classification

• VideoMAE
• ViViT
• TimeSformer

Depth Estimation

• Depth Anything V2

Image Understanding (VLM)

• Gemma 4 and compatible vision-language models via llama.cpp (image captioning, visual Q&A)
• LLaVA, LLaMA3-V, and other multimodal models via OpenAI-compatible endpoints

Build Client Libraries

To build the client libraries, refer to the official Triton Inference Server client libraries.

Alternative: Extract Client Libraries from Docker

For convenience, you can extract the pre-built Triton client libraries from the official NVIDIA Triton Server SDK image using Docker:

# Run the extraction script
./docker/scripts/extract_triton_libs.sh

This script will:

1. Create a temporary Docker container from the nvcr.io/nvidia/tritonserver:25.06-py3-sdk image
2. Extract the Triton client libraries from /workspace/install
3. Copy additional Triton server headers and libraries if available
4. Save everything to ./triton_client_libs/ directory

After extraction, set the environment variable:

export TritonClientBuild_DIR=$(pwd)/triton_client_libs/install

The extracted directory structure will contain:

• install/ - Triton client build artifacts
• triton_server_include/ - Triton server headers
• triton_server_lib/ - Triton server libraries
• workspace/ - Additional workspace files

Dependencies

Ensure the following dependencies are installed:

1. Nvidia Triton Inference Server:

docker pull nvcr.io/nvidia/tritonserver:25.06-py3

2. Triton client libraries: Tested on Release r25.06
3. Protobuf and gRPC++: Versions compatible with Triton
4. RapidJSON:

apt install rapidjson-dev

5. libcurl:

apt install libcurl4-openssl-dev

6. OpenCV 4: Tested version: 4.7.0

apt install libopencv-dev

Development Setup

Pre-commit Hooks (Recommended)

To maintain code quality and consistency, install pre-commit hooks:

# Run the setup script
./scripts/setup/pre_commit_setup.sh

# Or install manually
pip install pre-commit
pre-commit install

Build and Compile

1. Set the environment variable TritonClientBuild_DIR or update the CMakeLists.txt with the path to your installed Triton client libraries.

2. Create a build directory:

mkdir build

3. Navigate to the build directory:

cd build

4. Run CMake to configure the build:

cmake -DCMAKE_BUILD_TYPE=Release ..

Optional flags:

• -DSHOW_FRAME: Enable to display processed frames after inference
• -DWRITE_FRAME: Enable to write processed frames to disk

5. Build the application:

cmake --build .

Tasks

Export Instructions

• Object Detection
• Classification
• Instance Segmentation
• Optical Flow
• Open Vocabulary Detection
• Pose Estimation
• Video Classification
• Depth Estimation

Other tasks are in TODO list.

Notes

Ensure the model export versions match those supported by your Triton release. Check Triton releases here.

Deploying Models

To deploy models, set up a model repository following the Triton Model Repository schema. The config.pbtxt file is optional unless you're using the OpenVino backend, implementing an Ensemble pipeline, or passing custom inference parameters.

Model Repository Structure

<model_repository>/
    <model_name>/
        config.pbtxt
        <model_version>/
            <model_binary>

Starting Triton Server

Use the provided script for easy setup:

# Start Triton server with GPU support
./docker/scripts/docker_triton_run.sh /path/to/model_repository 25.06 gpu

# Start with CPU only
./docker/scripts/docker_triton_run.sh /path/to/model_repository 25.06 cpu

Or manually with Docker:

docker run --gpus=1 --rm \
  -p 8000:8000 -p 8001:8001 -p 8002:8002 \
  -v /full/path/to/model_repository:/models \
  nvcr.io/nvidia/tritonserver:<xx.yy>-py3 tritonserver \
  --model-repository=/models

Omit the --gpus flag if using the CPU version.

Running Inference

Command-Line Inference on Video or Image

./tritonic \
    --source=/path/to/source.format \
    --model_type=<model_type> \
    --model=<model_name_folder_on_triton> \
    --labelsFile=/path/to/labels/coco.names \
    --protocol=<http or grpc> \
    --serverAddress=<triton-ip> \
    --port=<8000 for http, 8001 for grpc> \

For dynamic input sizes:

    --input_sizes="c,h,w"

Shared Memory Support

Tritonic supports shared memory to improve inference performance by reducing data copying between the client and Triton server. Two types of shared memory are available:

System (POSIX) Shared Memory

Uses CPU-based shared memory for efficient data transfer:

./tritonic \
    --source=/path/to/source.format \
    --model=<model_name> \
    --shared_memory_type=system \
    ...

CUDA Shared Memory

Uses GPU memory directly for zero-copy inference (requires GPU support):

./tritonic \
    --source=/path/to/source.format \
    --model=<model_name> \
    --shared_memory_type=cuda \
    --cuda_device_id=0 \
    ...

Configuration Options:

• --shared_memory_type or -smt: Shared memory type (none, system, or cuda). Default: none
• --cuda_device_id or -cdi: CUDA device ID when using CUDA shared memory. Default: 0

Ensemble models

Triton ensembles move preprocessing, and optionally postprocessing, onto the GPU: the client sends the encoded JPEG and the server runs DALI decode/resize/normalize, TensorRT inference, and — for the gpu_pre_gpu_* variants — a custom CUDA postprocess operator.

Ensemble	Task	Deployment guide
yolo_dali_ensemble	Detection, GPU preprocess only	YOLO ensemble
yolo26det_*	Detection, GPU pre/post	YOLO26 detection ensemble
yolo26seg_*	Instance segmentation, GPU pre/post	YOLO26 segmentation ensemble
yolo11seg_*	Instance segmentation, GPU pre/post	YOLO11 segmentation ensemble

Instance-segmentation tasks return raster masks by default. Select convex-hull polygon rings with --segmentation_output=polygon; exteriors and holes are returned in image coordinates. --postprocess_mode=gpu supports both representations through separate encoded-image ensembles, so mask-only requests do not run polygon extraction. The yolo26seg examples below apply unchanged to yolo11seg by substituting the model names.

GPU mask output (default):

./build/tritonic \
  --source=data/images/bus.jpg \
  --model_type=yolo26seg \
  --model=yolo26seg_gpu_pre_gpu_mask_post \
  --task_model=yolo26seg_trt \
  --labelsFile=labels/coco.txt \
  --input_mode=encoded-image \
  --postprocess_mode=gpu

GPU polygon output:

./build/tritonic \
  --source=data/images/bus.jpg \
  --model_type=yolo26seg \
  --model=yolo26seg_gpu_pre_gpu_post \
  --task_model=yolo26seg_trt \
  --labelsFile=labels/coco.txt \
  --input_mode=encoded-image \
  --postprocess_mode=gpu \
  --segmentation_output=polygon

The same polygon ensemble accepts video sources. Tritonic JPEG-encodes each decoded frame in memory for DALI preprocessing and writes the rendered result to <video-directory>/output/processed_<model-name>.avi:

./build/tritonic \
  --source=/path/to/input.mp4 \
  --model_type=yolo26seg \
  --model=yolo26seg_gpu_pre_gpu_post \
  --task_model=yolo26seg_trt \
  --labelsFile=labels/coco.txt \
  --input_mode=encoded-image \
  --postprocess_mode=gpu \
  --segmentation_output=polygon

Deployment instructions and both tensor ABIs are documented in the YOLO26 segmentation ensemble guide. The reproducible CPU/GPU benchmark and semantic parity gate are in the YOLO26-seg benchmark.

Running with the Docker scripts

Wrapper scripts run TritonIC in a container without a local build. The full placeholder and model-type-tag reference is in Running Inference with the Docker Scripts.

Chat Backend (OpenAI-compatible)

Skip Triton entirely and query any OpenAI-compatible server — Ollama, llama.cpp, SGLang, vLLM, OpenAI, Together AI, OpenRouter, Z.AI:

./build/tritonic --backend=chat --chat_url=http://localhost:11434/v1/chat/completions \
  --model=gemma3:4b --prompt="Describe this image" --source=data/images/bus.jpg

Full flag reference in Chat Backend; the test procedure is in Chat Backend Testing.

Docker Support

For detailed instructions on installing Docker and the NVIDIA Container Toolkit, refer to the Docker Setup Document.

Build

docker build --rm -t tritonic -f docker/Dockerfile .

Run Container

docker run --rm \
  -v /path/to/host/data:/app/data \
  tritonic \
  --network host \
  --source=<path_to_source_on_container> \
  --model_type=<model_type> \
  --model=<model_name_folder_on_triton> \
  --labelsFile=<path_to_labels_on_container> \
  --protocol=<http or grpc> \
  --serverAddress=<triton-ip> \
  --port=<8000 for http, 8001 for grpc>

Kubernetes Deployment

For Kubernetes setup and deployment details, see:

• Kubernetes Deployment Guide
• K8s Scripts Usage

Quick start:

./k8s/scripts/check_and_deploy_triton.sh

This script performs:

1. kubectl installation check (and install if missing)
2. Kubernetes cluster liveness check — automatically starts or installs a local cluster (minikube → kind → k3s) if none is reachable; installs kind via Docker if no tool is present
3. NVIDIA GPU availability check inside cluster — installs nvidia-container-toolkit and the NVIDIA device plugin automatically if the host has a GPU
4. Triton deployment status check and reconciliation against the current manifests
5. Triton deploy or update (GPU or CPU manifest)
6. External Triton endpoint summary for the NodePort service

Default external access on minikube:

• HTTP: http://$(minikube ip):30800
• gRPC: $(minikube ip):30801
• Metrics: http://$(minikube ip):30802/metrics

Notes:

• The deployment uses the Triton 25.12-py3 image by default for Kubernetes.
• On minikube, if the Triton image is already present in host Docker, the deploy script loads it into the node before rollout to avoid long registry pulls.
• GPU deployments use the Recreate strategy so updates work on single-node, single-GPU clusters.

Demo

Real-time inference test (GPU RTX 3060):

• YOLOv7-tiny exported to ONNX: Demo Video
• YOLO11s exported to onnx: Demo Video
• RAFT Optical Flow Large(exported to traced torchscript): Demo Video

References

• Triton Inference Server Client Example
• Triton User Guide
• Triton Tutorials
• ONNX Models
• Torchvision Models
• Tensorflow Model Garden

Feedback

Any feedback is greatly appreciated. If you have any suggestions, bug reports, or questions, don't hesitate to open an issue. Contributions, corrections, and suggestions are welcome to keep this repository relevant and useful.