fix(local executor): use extra_docker_args instead of hard-coded --network=host (#286)

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com>
Co-authored-by: prokotg <19536019+prokotg@users.noreply.github.com>

Author: marta-sd

docs/nemo-evaluator-launcher/configuration/execution/local.md

@@ -9,6 +9,7 @@ See the complete configuration structure in the [Local Config File](../../../../
 ## Key Settings
 
 - **`output_dir`**: Directory where evaluation results will be saved (required)
+- **`extra_docker_args`**: Additional arguments to pass to the `docker run` command (optional). This flag allows advanced users to customize their setup (see [Advanced configuration](#advanced-configuration)).
 
 Tips:
 - ensure Docker is running on your local machine
@@ -20,6 +21,21 @@ Examples:
 - [Auto Export Example](https://github.com/NVIDIA-NeMo/Evaluator/tree/main/packages/nemo-evaluator-launcher/examples/local_auto_export_llama_3_1_8b_instruct.yaml) - Local execution with automatic result export
 - [Limit Samples Example](https://github.com/NVIDIA-NeMo/Evaluator/tree/main/packages/nemo-evaluator-launcher/examples/local_limit_samples.yaml) - Local execution with limited samples
 
+## Advanced configuration
+
+You can customize your local executor by specifying `extra_docker_args`.
+This parameter allows you to pass any flag to the `docker run` command that is executed by the NeMo Evaluator Launcher.
+You can use it to mount additional volumes, set environment variables or customize your network settings.
+
+For example, if you would like your job to use a specific docker network, you can specify:
+
+```yaml
+execution:
+  extra_docker_args: "--network my-custom-network"
+```
+
+Replace `my-custom-network` with `host` to access the host network.
+
 ## Reference
 
 - [Local Config File](../../../../packages/nemo-evaluator-launcher/src/nemo_evaluator_launcher/configs/execution/local.yaml)

docs/nemo-evaluator-launcher/tutorial.md

@@ -49,6 +49,10 @@ docker run --gpus all -p 8000:8000 vllm/vllm-openai:latest \
   --model meta-llama/Llama-3.1-8B-Instruct
 ```
 
+/// tip | Docker network settings
+When working with a locally-hosted endpoint and the local executor, make sure to configure your docker network settings via `extra_docker_args` parameter (see [Advanced configuration](../nemo-evaluator-launcher/configuration/execution/local.md#advanced-configuration) and [Deployment Frameworks Guide](tutorials/deployments/deployment-frameworks-guide.md)) 
+
+
 For more information, see:
 
   For detailed deployment instructions, see the [Deployment Frameworks Guide](tutorials/deployments/deployment-frameworks-guide.md).

docs/nemo-evaluator-launcher/tutorials/deployments/deployment-frameworks-guide.md

@@ -18,7 +18,7 @@ Models deployed with the frameworks listed below should work with nemo_evaluator
 
 ## Quick Setup Options
 
-# vLLM
+### vLLM
 
 vLLM is a fast and easy-to-use library for LLM inference and serving..
 
@@ -32,15 +32,15 @@ docker run --gpus all -p 8000:8000 vllm/vllm-openai:latest \
 - [vLLM Documentation](https://docs.vllm.ai/en/latest/)
 - [vLLM Docker Deployment](https://docs.vllm.ai/en/stable/deployment/docker.html)
 
-# SGLang
+### SGLang
 
 SGLang is a fast serving framework for large language models and vision language models. It makes your interaction with models faster and more controllable by co-designing the backend runtime and frontend language. The core features include:
 
 **Documentation:** 
 - [SGLang Documentation](https://docs.sglang.ai/)
 - [SGLang Docker Deployment](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#using-docker-recommended)
 
-# NeMo
+### NeMo
 
 NeMo Framework is NVIDIA's GPU accelerated, end-to-end training framework for large language models (LLMs), multi-modal models and speech models. The Export-Deploy library ("NeMo Export-Deploy") provides tools and APIs for exporting and deploying NeMo and 🤗Hugging Face models to production environments. It supports various deployment paths including TensorRT, TensorRT-LLM, and vLLM deployment through NVIDIA Triton Inference Server.
 
@@ -49,15 +49,15 @@ NeMo Framework is NVIDIA's GPU accelerated, end-to-end training framework for la
 - [NeMo Export-Deploy](https://github.com/NVIDIA-NeMo/Export-Deploy)
 - [NeMo Export-Deploy Scripts](https://github.com/NVIDIA-NeMo/Export-Deploy/tree/main/scripts)
 
-# TRT-LLM
+### TRT-LLM
 
 TRT-LLM provides optimized inference with OpenAI-compatible server through the `trtllm-serve` command.
 
 **Documentation:** 
 - [TensorRT-LLM Documentation](https://docs.nvidia.com/tensorrt-llm/index.html)
 - [TRT-LLM Server](https://nvidia.github.io/TensorRT-LLM/commands/trtllm-serve.html)
 
-# NIM (NVIDIA Inference Microservices)
+### NIM (NVIDIA Inference Microservices)
 
 NIM provides optimized inference microservices with OpenAI-compatible APIs.
 
@@ -68,4 +68,50 @@ NIM provides optimized inference microservices with OpenAI-compatible APIs.
 
 **Next Steps:**
 - [Local Evaluation of Existing Endpoint](../local-evaluation-of-existing-endpoint.md) - Learn how to run evaluations
-- [Testing Endpoint Compatibility](testing-endpoint-oai-compatibility.md) - Test your deployed endpoint with curl requests
+- [Testing Endpoint Compatibility](testing-endpoint-oai-compatibility.md) - Test your deployed endpoint with curl requests
+
+
+## Advance settings
+
+If you are deploying the model locally with Docker, you can use a dedicated docker network.
+This will provide a secure connetion between deployment and evaluation docker containers.
+
+```shell
+docker network create my-custom-network
+
+docker run --gpus all --network my-custom-network --name my-phi-container vllm/vllm-openai:latest \
+    --model microsoft/Phi-4-mini-instruct
+```
+
+Then use the same network in the evaluator config:
+
+```yaml
+defaults:
+  - execution: local
+  - deployment: none
+  - _self_
+
+execution:
+  output_dir: my_phi_test
+  extra_docker_args: "--network my-custom-network"
+
+target:
+  api_endpoint:
+    model_id: microsoft/Phi-4-mini-instruct
+    url: http://my-phi-container:8000/v1/chat/completions
+    api_key_name: null
+
+evaluation:
+  tasks:
+    - name: simple_evals.mmlu_pro
+      overrides:
+        config.params.limit_samples: 10 # TEST ONLY: Limits to 10 samples for quick testing
+        config.params.parallelism: 1
+```
+
+Alternatively you can expose ports as shown in the examples above and use the host network:
+
+```yaml
+execution:
+  extra_docker_args: "--network host"
+```

packages/nemo-evaluator-launcher/src/nemo_evaluator_launcher/configs/execution/local.yaml

@@ -15,3 +15,4 @@
 #
 type: local
 output_dir: ???
+extra_docker_args: ""

packages/nemo-evaluator-launcher/src/nemo_evaluator_launcher/executors/local/executor.py

@@ -164,10 +164,13 @@ def execute_eval(cls, cfg: DictConfig, dry_run: bool = False) -> str:
             auto_export_config = cfg.execution.get("auto_export", {})
             auto_export_destinations = auto_export_config.get("destinations", [])
 
+            extra_docker_args = cfg.execution.get("extra_docker_args", "")
+
             run_sh_content = (
                 eval_template.render(
                     evaluation_tasks=[evaluation_task],
                     auto_export_destinations=auto_export_destinations,
+                    extra_docker_args=extra_docker_args,
                 ).rstrip("\n")
                 + "\n"
             )
@@ -178,6 +181,7 @@ def execute_eval(cls, cfg: DictConfig, dry_run: bool = False) -> str:
             eval_template.render(
                 evaluation_tasks=evaluation_tasks,
                 auto_export_destinations=auto_export_destinations,
+                extra_docker_args=extra_docker_args,
             ).rstrip("\n")
             + "\n"
         )

packages/nemo-evaluator-launcher/src/nemo_evaluator_launcher/executors/local/run.template.sh

@@ -34,7 +34,7 @@ echo "$(date -u +%Y-%m-%dT%H:%M:%SZ)" > "$logs_dir/stage.pre-start"
 # Docker run with eval factory command
 (
     echo "$(date -u +%Y-%m-%dT%H:%M:%SZ)" > "$logs_dir/stage.running"
-    docker run --rm --shm-size=100g --network=host \
+    docker run --rm --shm-size=100g {{ extra_docker_args }} \
       --name {{ task.container_name }} \
       --volume "$artifacts_dir":/results \
       {% for env_var in task.env_vars -%}