error running add_sub example

[Arnold1]

hi,

i run into the following error when running model_analyzer:
[Model Analyzer] Model add_sub_config_default load failed: [StatusCode.INTERNAL] failed to load 'add_sub', failed to poll from model repository

it fails for both cases:

• --model-repository /workspace/examples/quick-start
• --model-repository /workspace/examples/quick-start/add_sub

I followed this tutorial: https://github.com/triton-inference-server/model_analyzer/blob/r24.12/docs/quick_start.md
Steps to reproduce the issue:

mkdir model_analyzer_test && cd model_analyzer_test

git init && git remote add -f origin https://github.com/triton-inference-server/model_analyzer.git

git config core.sparseCheckout true && \
echo 'examples' >> .git/info/sparse-checkout && \
git pull origin main

docker pull nvcr.io/nvidia/tritonserver:24.12-py3-sdk

docker run -it --gpus all \
      -v /var/run/docker.sock:/var/run/docker.sock \
      -v $(pwd)/examples/quick-start:/workspace/examples/quick-start \
      --net=host nvcr.io/nvidia/tritonserver:24.12-py3-sdk

# we are inside the docker now

mkdir ./output

model-analyzer profile \
    --model-repository /workspace/examples/quick-start \
    --profile-models add_sub --triton-launch-mode=docker \
    --output-model-repository-path output/add_sub_output \
    --export-path profile_results


model-analyzer profile     --model-repository /workspace/examples/quick-start     --profile-models add_sub --triton-launch-mode=docker     --output-model-repository-path output/add_sub_output     --export-path profile_results
[Model Analyzer] Call to cuInit results in CUDA_ERROR_NO_DEVICE
[Model Analyzer] Starting a Triton Server using docker
[Model Analyzer] Loaded checkpoint from file /workspace/checkpoints/0.ckpt
[Model Analyzer] GPU devices match checkpoint - skipping server metric acquisition
[Model Analyzer] 
[Model Analyzer] Starting automatic brute search
[Model Analyzer] 
[Model Analyzer] Creating model config: add_sub_config_default
[Model Analyzer] 
[Model Analyzer] Model add_sub_config_default load failed: [StatusCode.INTERNAL] failed to load 'add_sub', failed to poll from model repository
[Model Analyzer] Saved checkpoint to /workspace/checkpoints/1.ckpt
[Model Analyzer] Creating model config: add_sub_config_0
[Model Analyzer]   Setting instance_group to [{'count': 1, 'kind': 'KIND_CPU'}]
[Model Analyzer]   Setting max_batch_size to 1
[Model Analyzer]   Enabling dynamic_batching
[Model Analyzer] 
[Model Analyzer] Model add_sub_config_0 load failed: [StatusCode.INTERNAL] failed to load 'add_sub', failed to poll from model repository
[Model Analyzer] No changes made to analyzer data, no checkpoint saved.
Traceback (most recent call last):
  File "/usr/local/bin/model-analyzer", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/model_analyzer/entrypoint.py", line 278, in main
    analyzer.profile(
  File "/usr/local/lib/python3.12/dist-packages/model_analyzer/analyzer.py", line 131, in profile
    self._profile_models()
  File "/usr/local/lib/python3.12/dist-packages/model_analyzer/analyzer.py", line 251, in _profile_models
    self._model_manager.run_models(models=[model])
  File "/usr/local/lib/python3.12/dist-packages/model_analyzer/model_manager.py", line 157, in run_models
    self._stop_ma_if_no_valid_measurement_threshold_reached()
  File "/usr/local/lib/python3.12/dist-packages/model_analyzer/model_manager.py", line 251, in _stop_ma_if_no_valid_measurement_threshold_reached
    raise TritonModelAnalyzerException(
model_analyzer.model_analyzer_exceptions.TritonModelAnalyzerException: The first 2 attempts to acquire measurements have failed. Please examine the Tritonserver/PA error logs to determine what has gone wrong.

docker logs of the triton inference server:

docker logs ac3cbc35ddec

=============================
== Triton Inference Server ==
=============================

NVIDIA Release 24.11 (build 124543091)
Triton Server Version 2.52.0

Copyright (c) 2018-2024, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

WARNING: The NVIDIA Driver was not detected.  GPU functionality will not be available.
   Use the NVIDIA Container Toolkit to start this container with GPU support; see
   https://docs.nvidia.com/datacenter/cloud-native/ .

Warning: '--strict-model-config' has been deprecated! Please use '--disable-auto-complete-config' instead.
W0416 04:18:12.745911 7 pinned_memory_manager.cc:273] "Unable to allocate pinned system memory, pinned memory pool will not be available: CUDA driver version is insufficient for CUDA runtime version"
I0416 04:18:12.745946 7 cuda_memory_manager.cc:117] "CUDA memory pool disabled"
E0416 04:18:12.746011 7 server.cc:241] "CudaDriverHelper has not been initialized."
I0416 04:18:12.746193 7 server.cc:604] 
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+

I0416 04:18:12.746213 7 server.cc:631] 
+---------+------+--------+
| Backend | Path | Config |
+---------+------+--------+
+---------+------+--------+

I0416 04:18:12.746222 7 server.cc:674] 
+-------+---------+--------+
| Model | Version | Status |
+-------+---------+--------+
+-------+---------+--------+

I0416 04:18:12.746391 7 metrics.cc:783] "Collecting CPU metrics"
I0416 04:18:12.746484 7 tritonserver.cc:2598] 
+----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Option                           | Value                                                                                                                                                                                                           |
+----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id                        | triton                                                                                                                                                                                                          |
| server_version                   | 2.52.0                                                                                                                                                                                                          |
| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data parameters statistics trace logging |
| model_repository_path[0]         | /workspace/examples/quick-start                                                                                                                                                                                 |
| model_control_mode               | MODE_EXPLICIT                                                                                                                                                                                                   |
| strict_model_config              | 0                                                                                                                                                                                                               |
| model_config_name                |                                                                                                                                                                                                                 |
| rate_limit                       | OFF                                                                                                                                                                                                             |
| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                                       |
| min_supported_compute_capability | 6.0                                                                                                                                                                                                             |
| strict_readiness                 | 1                                                                                                                                                                                                               |
| exit_timeout                     | 30                                                                                                                                                                                                              |
| cache_enabled                    | 0                                                                                                                                                                                                               |
+----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I0416 04:18:12.748541 7 grpc_server.cc:2558] "Started GRPCInferenceService at 0.0.0.0:8001"
I0416 04:18:12.748725 7 http_server.cc:4729] "Started HTTPService at 0.0.0.0:8000"
I0416 04:18:12.789778 7 http_server.cc:362] "Started Metrics Service at 0.0.0.0:8002"
E0416 04:18:18.666092 7 model_repository_manager.cc:1415] "failed to poll model 'add_sub': model not found in any model repository."
Signal (15) received.
I0416 04:18:18.675493 7 server.cc:305] "Waiting for in-flight requests to complete."
I0416 04:18:18.675512 7 server.cc:321] "Timeout 30: Found 0 model versions that have in-flight inferences"
I0416 04:18:18.675524 7 server.cc:336] "All models are stopped, unloading models"
I0416 04:18:18.675531 7 server.cc:345] "Timeout 30: Found 0 live models and 0 in-flight non-inference requests"

cc @mc-nv @pvijayakrish @debermudez @nv-braf @mc-nv

[nv-braf]

I believe the issue is that you've deviated from the quick start by changing the symbolic link from:
-v $(pwd)/examples/quick-start:$(pwd)/examples/quick-start \
to:
-v $(pwd)/examples/quick-start:/workspace/examples/quick-start

When running the triton server in docker mode the links must match exactly.

[Arnold1]

@nv-braf thanks you are right - that works.

i have a few follow up questions to ensemble models:
I tried using my ensemble model with model analyzer. my ensemble model has 3 steps:
pre-processing (tokenizer) -> trt inference (trt plan) -> post-processing.

• question to batching
  • to run model analyzer with batching > 0: i need to build tensorrt plan for each batch size model analyzer will use?
  • will i need to change anything in my current config.pbtxt for pre-processing, text-classification and post-processing in my model repo regarding batching (e.g. max_batch_size or dynamic batching or input / output shape)?
  • why is model analyzer using batch size = 1 (Setting max_batch_size to 1) if i set model_config_parameters: max_batch_size: [0] in the config.yaml?

data.json:

{
    "data":
        [
            {
                "text_input": ["i love machine learning"]
            }
         ]
}

• does text_input:1 make sense in perf_analyzer_flag for data.json above? ... 1 is the batch size? what happens if i have different test data - different prompt lengths - is that possible?
• will model_analyzer only send 1 request per config in this case?
• do the model analyzer logs look ok or do you see any issues?

i added validation on the data but get a shape error - do you see something wrong with data.json or the perf_analyzer_flags?
error:

5:49:20 [Model Analyzer] Running perf_analyzer failed with exit status 99:
error: Failed to init manager inputs: The variable-sized tensor "class" with model shape [-1] needs to have its shape fully defined. See the --shape option.

model analyzer config:

perf_analyzer_flags:
  input-data: data.json
  shape:
  - text_input:1
  - class:1

data.json:

{
  "data":
    [
      {
        "text_input": ["hello how are you"]
      }
    ],
  "validation_data":
    [
      {
        "class": ["class_1"]
      }
    ]
}

my model analyzer config.yaml:

triton_docker_image: tritonserver:25.02-transformers-torch #nvcr.io/nvidia/tritonserver:25.02-py3 with transformers and pytorch lib
triton_launch_mode: docker

model_repository: /teamspace/studios/this_studio/tmp/model-repository-trt

triton_server_flags:
  log_verbose: True

cpu_only_composing_models: pre-processing,post-processing

# Disable run config search
run_config_search_disable: True
#run_config_search_mode: quick

# Model profiling configuration
profile_models:
  ensemble-model:
    parameters:
      concurrency:
        start: 1
        stop: 30
        step: 5
    model_config_parameters:
      max_batch_size: [0]

perf_analyzer_flags:
  input-data: data.json
  shape:
  - text_input:1

# ALL THE FOLLOWING FIELDS ARE OPTIONAL 
# The directory to which the model analyzer will save model config variants
#output_model_repository_path: /workspace/output
#results

export_path: profile_results

# Allow model analyzer to overwrite contents of the output model repository
override_output_model_repository: true

model-analyzer with logs:

model-analyzer -v profile \
  -f config.yaml

04:34:53 [Model Analyzer] DEBUG: 
{'always_report_gpu_metrics': False,
 'batch_sizes': [1],
 'bls_composing_models': [],
 'checkpoint_directory': '/workspace/checkpoints',
 'client_max_retries': 50,
 'client_protocol': 'grpc',
 'collect_cpu_metrics': False,
 'concurrency': [1],
 'concurrency_sweep_disable': False,
 'config_file': 'config.yaml',
 'constraints': {},
 'cpu_only_composing_models': ['pre-processing', 'post-processing'],
 'dcgm_disable': False,
 'duration_seconds': 3,
 'early_exit_enable': False,
 'export_path': 'profile_results',
 'filename_model_gpu': 'metrics-model-gpu.csv',
 'filename_model_inference': 'metrics-model-inference.csv',
 'filename_server_only': 'metrics-server-only.csv',
 'genai_perf_flags': {},
 'gpu_output_fields': ['model_name',
                       'gpu_uuid',
                       'batch_size',
                       'concurrency',
                       'model_config_path',
                       'instance_group',
                       'satisfies_constraints',
                       'gpu_used_memory',
                       'gpu_utilization',
                       'gpu_power_usage'],
 'gpus': ['all'],
 'inference_output_fields': ['model_name',
                             'batch_size',
                             'concurrency',
                             'model_config_path',
                             'instance_group',
                             'max_batch_size',
                             'satisfies_constraints',
                             'perf_throughput',
                             'perf_latency_p99'],
 'latency_budget': None,
 'max_percentage_of_search_space': 10,
 'min_percentage_of_search_space': 5,
 'min_throughput': None,
 'model_repository': '/teamspace/studios/this_studio/tmp/model-repository-trt',
 'model_type': 'generic',
 'monitoring_interval': 1.0,
 'num_configs_per_model': 3,
 'num_top_model_configs': 0,
 'objectives': {'perf_throughput': 10},
 'optuna_early_exit_threshold': 10,
 'optuna_max_trials': 200,
 'optuna_min_trials': 20,
 'output_model_repository_path': '/workspace/output_model_repository',
 'override_output_model_repository': True,
 'perf_analyzer_cpu_util': 2560.0,
 'perf_analyzer_flags': {'input-data': 'data.json', 'shape': ['text_input:1']},
 'perf_analyzer_max_auto_adjusts': 10,
 'perf_analyzer_path': 'perf_analyzer',
 'perf_analyzer_timeout': 600,
 'perf_output': False,
 'perf_output_path': None,
 'plots': [{'name': 'throughput_v_latency', 'title': 'Throughput vs. Latency', 'x_axis': 'perf_latency_p99', 'y_axis': 'perf_throughput', 'monotonic': True},
           {'name': 'gpu_mem_v_latency', 'title': 'GPU Memory vs. Latency', 'x_axis': 'perf_latency_p99', 'y_axis': 'gpu_used_memory', 'monotonic': False}],
 'profile_models': [{'model_name': 'ensemble-model', 'cpu_only': False, 'objectives': {'perf_throughput': 10}, 'parameters': {'batch_sizes': [1], 'concurrency': [1, 6, 11, 16, 21, 26], 'request_rate': []}, 'weighting': 1, 'model_config_parameters': {'max_batch_size': [0]}, 'perf_analyzer_flags': {'input-data': 'data.json', 'shape': ['text_input:1']}}],
 'reload_model_disable': False,
 'request_rate': [],
 'request_rate_search_enable': False,
 'run_config_profile_models_concurrently_enable': False,
 'run_config_search_disable': True,
 'run_config_search_max_binary_search_steps': 5,
 'run_config_search_max_concurrency': 1024,
 'run_config_search_max_instance_count': 5,
 'run_config_search_max_model_batch_size': 128,
 'run_config_search_max_request_rate': 8192,
 'run_config_search_min_concurrency': 1,
 'run_config_search_min_instance_count': 1,
 'run_config_search_min_model_batch_size': 1,
 'run_config_search_min_request_rate': 16,
 'run_config_search_mode': 'brute',
 'server_output_fields': ['model_name',
                          'gpu_uuid',
                          'gpu_used_memory',
                          'gpu_utilization',
                          'gpu_power_usage'],
 'skip_detailed_reports': False,
 'skip_summary_reports': False,
 'triton_docker_args': {},
 'triton_docker_image': 'tritonserver:25.02-transformers-torch',
 'triton_docker_labels': {},
 'triton_docker_mounts': [],
 'triton_docker_shm_size': None,
 'triton_grpc_endpoint': 'localhost:8001',
 'triton_http_endpoint': 'localhost:8000',
 'triton_install_path': '/opt/tritonserver',
 'triton_launch_mode': 'docker',
 'triton_metrics_url': 'http://localhost:8002/metrics',
 'triton_output_path': None,
 'triton_server_environment': {},
 'triton_server_flags': {'log_verbose': 'True'},
 'triton_server_path': 'tritonserver',
 'use_concurrency_formula': False,
 'weighting': None}
04:34:53 [Model Analyzer] Initializing GPUDevice handles
04:34:54 [Model Analyzer] Using GPU 0 NVIDIA A10G with UUID GPU-cf2e37a6-75b4-3931-e81c-910b5ba35b8d
04:34:54 [Model Analyzer] WARNING: Overriding the output model repo path "/workspace/output_model_repository"
04:34:54 [Model Analyzer] Starting a Triton Server using docker
04:34:54 [Model Analyzer] Loaded checkpoint from file /workspace/checkpoints/2.ckpt
04:34:54 [Model Analyzer] GPU devices match checkpoint - skipping server metric acquisition
04:34:54 [Model Analyzer] 
04:34:54 [Model Analyzer] Starting quick mode search to find optimal configs
04:34:54 [Model Analyzer] 
04:34:54 [Model Analyzer] Creating model config: pre-processing_config_default
04:34:54 [Model Analyzer] 
04:34:54 [Model Analyzer] Creating model config: text-classification_config_default
04:34:54 [Model Analyzer] 
04:34:54 [Model Analyzer] Creating model config: post-processing_config_default
04:34:54 [Model Analyzer] 
04:34:54 [Model Analyzer] Creating ensemble model config: ensemble-model_config_default
04:34:58 [Model Analyzer] Existing measurement found for run config. Skipping profile
04:34:58 [Model Analyzer] DEBUG: Measurement for [0, 0, 0]: throughput = 82.5917, latency = 12.383 (best throughput: 82.5917, best_latency: 12.383)
04:34:58 [Model Analyzer] Saved checkpoint to /workspace/checkpoints/3.ckpt
04:34:58 [Model Analyzer] Found existing model config: pre-processing_config_0
04:34:58 [Model Analyzer]   Setting instance_group to [{'count': 1, 'kind': 'KIND_CPU'}]
04:34:58 [Model Analyzer] 
04:34:58 [Model Analyzer] Found existing model config: text-classification_config_0
04:34:58 [Model Analyzer]   Setting instance_group to [{'count': 1, 'kind': 'KIND_GPU'}]
04:34:58 [Model Analyzer] 
04:34:58 [Model Analyzer] Creating model config: post-processing_config_5
04:34:58 [Model Analyzer]   Setting instance_group to [{'count': 1, 'kind': 'KIND_CPU'}]
04:34:58 [Model Analyzer] 
04:34:58 [Model Analyzer] Creating ensemble model config: ensemble-model_config_8
04:34:58 [Model Analyzer]   Setting max_batch_size to 1
04:35:03 [Model Analyzer] DEBUG: Triton Server started.
04:35:25 [Model Analyzer] DEBUG: Model pre-processing_config_0 loaded
04:35:31 [Model Analyzer] DEBUG: Model text-classification_config_0 loaded
04:35:37 [Model Analyzer] DEBUG: Model post-processing_config_5 loaded
04:35:38 [Model Analyzer] DEBUG: Model ensemble-model_config_8 loaded
04:35:38 [Model Analyzer] Profiling ensemble-model_config_8: concurrency=2
04:35:38 [Model Analyzer] DEBUG: Running ['perf_analyzer', '-m', 'ensemble-model', '-b', '1', '-u', 'localhost:8001', '-i', 'grpc', '-f', 'ensemble-model-results.csv', '--verbose-csv', '--concurrency-range', '2', '--input-data', 'data.json', '--shape', 'text_input:1', '--measurement-mode', 'count_windows', '--collect-metrics', '--metrics-url', 'http://localhost:8002/metrics', '--metrics-interval', '1000']
04:35:42 [Model Analyzer] DEBUG: Reading PA results from ensemble-model-results.csv
04:35:42 [Model Analyzer] DEBUG: Measurement for [0, 0, 0]: throughput = 98.2514, latency = 20.9 (best throughput: 98.2514, best_latency: 20.9)
04:35:42 [Model Analyzer] Saved checkpoint to /workspace/checkpoints/3.ckpt
04:35:42 [Model Analyzer] DEBUG: Need more data. Measuring [0, 0, 1]
04:35:42 [Model Analyzer] Found existing model config: pre-processing_config_0
04:35:42 [Model Analyzer]   Setting instance_group to [{'count': 1, 'kind': 'KIND_CPU'}]
04:35:42 [Model Analyzer] 
04:35:42 [Model Analyzer] Found existing model config: text-classification_config_0
04:35:42 [Model Analyzer]   Setting instance_group to [{'count': 1, 'kind': 'KIND_GPU'}]
04:35:42 [Model Analyzer] 
04:35:42 [Model Analyzer] Creating model config: post-processing_config_6
04:35:42 [Model Analyzer]   Setting instance_group to [{'count': 2, 'kind': 'KIND_CPU'}]
04:35:42 [Model Analyzer] 
04:35:42 [Model Analyzer] Creating ensemble model config: ensemble-model_config_9
04:35:42 [Model Analyzer]   Setting max_batch_size to 1
04:35:51 [Model Analyzer] DEBUG: Stopped Triton Server.
04:35:51 [Model Analyzer] DEBUG: Triton Server started.
04:36:06 [Model Analyzer] DEBUG: Model pre-processing_config_0 loaded
04:36:12 [Model Analyzer] DEBUG: Model text-classification_config_0 loaded
04:36:18 [Model Analyzer] DEBUG: Model post-processing_config_6 loaded
04:36:19 [Model Analyzer] DEBUG: Model ensemble-model_config_9 loaded
04:36:19 [Model Analyzer] Profiling ensemble-model_config_9: concurrency=2
04:36:19 [Model Analyzer] DEBUG: Running ['perf_analyzer', '-m', 'ensemble-model', '-b', '1', '-u', 'localhost:8001', '-i', 'grpc', '-f', 'ensemble-model-results.csv', '--verbose-csv', '--concurrency-range', '2', '--input-data', 'data.json', '--shape', 'text_input:1', '--measurement-mode', 'count_windows', '--collect-metrics', '--metrics-url', 'http://localhost:8002/metrics', '--metrics-interval', '1000']
04:36:23 [Model Analyzer] DEBUG: Reading PA results from ensemble-model-results.csv
04:36:23 [Model Analyzer] DEBUG: Measurement for [0, 0, 1]: throughput = 98.5839, latency = 21.147 (best throughput: 98.5839, best_latency: 21.147)
04:36:23 [Model Analyzer] Saved checkpoint to /workspace/checkpoints/3.ckpt
04:36:23 [Model Analyzer] DEBUG: Need more data. Measuring [1, 1, 0]
04:36:23 [Model Analyzer] Found existing model config: pre-processing_config_1
04:36:23 [Model Analyzer]   Setting instance_group to [{'count': 2, 'kind': 'KIND_CPU'}]
04:36:23 [Model Analyzer] 
04:36:23 [Model Analyzer] Found existing model config: text-classification_config_1
04:36:23 [Model Analyzer]   Setting instance_group to [{'count': 2, 'kind': 'KIND_GPU'}]
04:36:23 [Model Analyzer] 
04:36:23 [Model Analyzer] Found existing model config: post-processing_config_5
04:36:23 [Model Analyzer]   Setting instance_group to [{'count': 1, 'kind': 'KIND_CPU'}]
04:36:23 [Model Analyzer] 
04:36:23 [Model Analyzer] Creating ensemble model config: ensemble-model_config_10
04:36:23 [Model Analyzer]   Setting max_batch_size to 1
04:36:31 [Model Analyzer] DEBUG: Stopped Triton Server.
04:36:31 [Model Analyzer] DEBUG: Triton Server started.
04:36:46 [Model Analyzer] DEBUG: Model pre-processing_config_1 loaded
04:36:55 [Model Analyzer] DEBUG: Model text-classification_config_1 loaded
04:37:00 [Model Analyzer] DEBUG: Model post-processing_config_5 loaded
04:37:01 [Model Analyzer] DEBUG: Model ensemble-model_config_10 loaded
04:37:01 [Model Analyzer] Profiling ensemble-model_config_10: concurrency=4
04:37:01 [Model Analyzer] DEBUG: Running ['perf_analyzer', '-m', 'ensemble-model', '-b', '1', '-u', 'localhost:8001', '-i', 'grpc', '-f', 'ensemble-model-results.csv', '--verbose-csv', '--concurrency-range', '4', '--input-data', 'data.json', '--shape', 'text_input:1', '--measurement-mode', 'count_windows', '--collect-metrics', '--metrics-url', 'http://localhost:8002/metrics', '--metrics-interval', '1000']
04:37:05 [Model Analyzer] DEBUG: Reading PA results from ensemble-model-results.csv
04:37:05 [Model Analyzer] DEBUG: Measurement for [1, 1, 0]: throughput = 127.862, latency = 34.471 (best throughput: 127.862, best_latency: 34.471)
04:37:05 [Model Analyzer] Saved checkpoint to /workspace/checkpoints/3.ckpt
04:37:05 [Model Analyzer] DEBUG: Need more data. Measuring [0, 2, 2]
04:37:05 [Model Analyzer] Found existing model config: pre-processing_config_0
04:37:05 [Model Analyzer]   Setting instance_group to [{'count': 1, 'kind': 'KIND_CPU'}]
04:37:05 [Model Analyzer] 
04:37:05 [Model Analyzer] Found existing model config: text-classification_config_2
04:37:05 [Model Analyzer]   Setting instance_group to [{'count': 3, 'kind': 'KIND_GPU'}]
04:37:05 [Model Analyzer] 
04:37:05 [Model Analyzer] Creating model config: post-processing_config_7
04:37:05 [Model Analyzer]   Setting instance_group to [{'count': 3, 'kind': 'KIND_CPU'}]
04:37:05 [Model Analyzer] 
04:37:05 [Model Analyzer] Creating ensemble model config: ensemble-model_config_11
04:37:05 [Model Analyzer]   Setting max_batch_size to 1
04:37:15 [Model Analyzer] DEBUG: Stopped Triton Server.
04:37:16 [Model Analyzer] DEBUG: Triton Server started.
04:37:30 [Model Analyzer] DEBUG: Model pre-processing_config_0 loaded
04:37:41 [Model Analyzer] DEBUG: Model text-classification_config_2 loaded
04:37:47 [Model Analyzer] DEBUG: Model post-processing_config_7 loaded
04:37:48 [Model Analyzer] DEBUG: Model ensemble-model_config_11 loaded
04:37:48 [Model Analyzer] Profiling ensemble-model_config_11: concurrency=2
04:37:48 [Model Analyzer] DEBUG: Running ['perf_analyzer', '-m', 'ensemble-model', '-b', '1', '-u', 'localhost:8001', '-i', 'grpc', '-f', 'ensemble-model-results.csv', '--verbose-csv', '--concurrency-range', '2', '--input-data', 'data.json', '--shape', 'text_input:1', '--measurement-mode', 'count_windows', '--collect-metrics', '--metrics-url', 'http://localhost:8002/metrics', '--metrics-interval', '1000']
04:37:52 [Model Analyzer] DEBUG: Reading PA results from ensemble-model-results.csv
04:37:52 [Model Analyzer] DEBUG: Measurement for [0, 2, 2]: throughput = 136.203, latency = 17.851 (best throughput: 136.203, best_latency: 17.851)
04:37:52 [Model Analyzer] Saved checkpoint to /workspace/checkpoints/3.ckpt
04:37:52 [Model Analyzer] DEBUG: Stepping [0, 0, 0]->[1, 1, 1]
04:37:52 [Model Analyzer] Found existing model config: pre-processing_config_1
04:37:52 [Model Analyzer]   Setting instance_group to [{'count': 2, 'kind': 'KIND_CPU'}]
04:37:52 [Model Analyzer] 
04:37:52 [Model Analyzer] Found existing model config: text-classification_config_1
04:37:52 [Model Analyzer]   Setting instance_group to [{'count': 2, 'kind': 'KIND_GPU'}]
04:37:52 [Model Analyzer] 
04:37:52 [Model Analyzer] Found existing model config: post-processing_config_6
04:37:52 [Model Analyzer]   Setting instance_group to [{'count': 2, 'kind': 'KIND_CPU'}]
04:37:52 [Model Analyzer] 
04:37:52 [Model Analyzer] Creating ensemble model config: ensemble-model_config_12
04:37:52 [Model Analyzer]   Setting max_batch_size to 1
04:38:05 [Model Analyzer] DEBUG: Stopped Triton Server.
04:38:06 [Model Analyzer] DEBUG: Triton Server started.
04:38:21 [Model Analyzer] DEBUG: Model pre-processing_config_1 loaded
04:38:29 [Model Analyzer] DEBUG: Model text-classification_config_1 loaded
04:38:35 [Model Analyzer] DEBUG: Model post-processing_config_6 loaded
04:38:36 [Model Analyzer] DEBUG: Model ensemble-model_config_12 loaded
04:38:36 [Model Analyzer] Profiling ensemble-model_config_12: concurrency=4
04:38:36 [Model Analyzer] DEBUG: Running ['perf_analyzer', '-m', 'ensemble-model', '-b', '1', '-u', 'localhost:8001', '-i', 'grpc', '-f', 'ensemble-model-results.csv', '--verbose-csv', '--concurrency-range', '4', '--input-data', 'data.json', '--shape', 'text_input:1', '--measurement-mode', 'count_windows', '--collect-metrics', '--metrics-url', 'http://localhost:8002/metrics', '--metrics-interval', '1000']
04:38:40 [Model Analyzer] DEBUG: Reading PA results from ensemble-model-results.csv
04:38:40 [Model Analyzer] DEBUG: Measurement for [1, 1, 1]: throughput = 147.837, latency = 27.841 (best throughput: 147.837, best_latency: 27.841)
04:38:40 [Model Analyzer] Saved checkpoint to /workspace/checkpoints/3.ckpt
04:38:40 [Model Analyzer] DEBUG: Stepping [1, 1, 1]->[2, 2, 2]
04:38:40 [Model Analyzer] Found existing model config: pre-processing_config_2
04:38:40 [Model Analyzer]   Setting instance_group to [{'count': 3, 'kind': 'KIND_CPU'}]
04:38:40 [Model Analyzer] 
04:38:40 [Model Analyzer] Found existing model config: text-classification_config_2
04:38:40 [Model Analyzer]   Setting instance_group to [{'count': 3, 'kind': 'KIND_GPU'}]
04:38:40 [Model Analyzer] 
04:38:40 [Model Analyzer] Found existing model config: post-processing_config_7
04:38:40 [Model Analyzer]   Setting instance_group to [{'count': 3, 'kind': 'KIND_CPU'}]
04:38:40 [Model Analyzer] 
04:38:40 [Model Analyzer] Creating ensemble model config: ensemble-model_config_13
04:38:40 [Model Analyzer]   Setting max_batch_size to 1
04:38:51 [Model Analyzer] DEBUG: Stopped Triton Server.

here are the triton config files:

• ensemble-model-config.pbtxt

name: "ensemble-model"
platform: "ensemble"
max_batch_size: 0

input [
    {
        name: "text_input"
        data_type: TYPE_STRING
        dims: [-1]
    }
]

output [
    {
        name: "class"
        data_type: TYPE_STRING
        dims: [-1]
    },
    {
        name: "probabilities"
        data_type: TYPE_FP64
        dims: [-1, 2]
    }
]

ensemble_scheduling {
    step [
        {
            model_name: "pre-processing"
            model_version: -1
            input_map {
                key: "text_input"
                value: "text_input"
            }
            output_map [
                {
                    key: "input_ids"
                    value: "input_ids"
                },
                {
                    key: "attention_mask"
                    value: "attention_mask"
                }
            ]
        },
        {
            model_name: "text-classification"
            model_version: -1
            input_map [
                {
                    key: "input_ids"
                    value: "input_ids"
                },
                {
                    key: "attention_mask"
                    value: "attention_mask"
                }
            ]
            output_map [
                {
                    key: "logits"
                    value: "tmp_logits"
                }
            ]
        },
        {
            model_name: "post-processing"
            model_version: -1
            input_map [
                {
                    key: "tmp_logits"
                    value: "tmp_logits"
                }
            ]
            output_map [
                {
                    key: "class"
                    value: "class"
                },
                {
                    key: "probabilities"
                    value: "probabilities"
                }
            ]
        }
    ]
}

• pre-processing-config.pbtxt

name: "pre-processing"
backend: "python"
max_batch_size: 0

input [
    {
        name: "text_input"
        data_type: TYPE_STRING
        dims: [-1]
    }
]

output [
  {
    name: "input_ids"
    data_type: TYPE_INT64
    dims: [1, -1]
  },
  {
    name: "attention_mask"
    data_type: TYPE_INT64
    dims: [1, -1]
  }
]

instance_group [
    {
        count: 2
        kind: KIND_CPU
    }
]

• text-classification-config.pbtxt

name: "text-classification"
platform: "tensorrt_plan"
max_batch_size: 0

input [
  {
    name: "input_ids"
    data_type: TYPE_INT64
    dims: [1, -1]
  },
  {
    name: "attention_mask"
    data_type: TYPE_INT64
    dims: [1, -1]
  }
]
output [
  {
    name: "logits"
    data_type: TYPE_FP32
    dims: [-1, 2]
  }
]

instance_group [
  {
      count: 1
      kind: KIND_GPU
  }
]

• post-processing-config.pbtxt

name: "post-processing"
backend: "python"
max_batch_size: 0

input [
    {
        name: "tmp_logits"
        data_type: TYPE_FP32
        dims: [-1, 2]
    }
]

output [
  {
      name: "class"
      data_type: TYPE_STRING
      dims: [-1]
  },
  {
      name: "probabilities"
      data_type: TYPE_FP64
      dims: [-1, 2]
  }
]

instance_group [
  {
      count: 2
      kind: KIND_CPU
  }
]