From 8722d16d06d32ea4e0ae158c0c1603168caee294 Mon Sep 17 00:00:00 2001 From: Rahul Gurnani Date: Tue, 2 Sep 2025 00:14:31 +0000 Subject: [PATCH 1/4] Update helm chart README with pluginsCustomConfig flag --- config/charts/inferencepool/README.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/config/charts/inferencepool/README.md b/config/charts/inferencepool/README.md index 38204c044..7dbdfa2f2 100644 --- a/config/charts/inferencepool/README.md +++ b/config/charts/inferencepool/README.md @@ -45,6 +45,25 @@ Then apply it with: $ helm install vllm-llama3-8b-instruct ./config/charts/inferencepool -f values.yaml ``` +### Install with Custom EPP Plugins Configuration + +To set custom EPP plugin config, you can pass it as an inline yaml. For example: + +```yaml + pluginsCustomConfig: + custom-plugins.yaml: | + apiVersion: inference.networking.x-k8s.io/v1alpha1 + kind: EndpointPickerConfig + plugins: + - type: custom-scorer + parameters: + custom-threshold: 64 + schedulingProfiles: + - name: default + plugins: + - pluginRef: custom-scorer +``` + ### Install with Additional Ports To expose additional ports (e.g., for ZMQ), you can define them in the `values.yaml` file: @@ -126,6 +145,7 @@ The following table list the configurable parameters of the chart. | `inferenceExtension.affinity` | Affinity for the endpoint picker. Defaults to `{}`. | | `inferenceExtension.tolerations` | Tolerations for the endpoint picker. Defaults to `[]`. | | `inferenceExtension.flags.has-enable-leader-election` | Enable leader election for high availability. When enabled, only one EPP pod (the leader) will be ready to serve traffic. | +| `inferenceExtension.pluginsCustomConfig` | Custom config that is passed to EPP as inline yaml. | | `provider.name` | Name of the Inference Gateway implementation being used. Possible values: `gke`. Defaults to `none`. | ## Notes From 3762141279eb1a7aacf43194f5966d04cb8a4963 Mon Sep 17 00:00:00 2001 From: Rahul Gurnani Date: Thu, 4 Sep 2025 10:21:15 -0700 Subject: [PATCH 2/4] Update config/charts/inferencepool/README.md Co-authored-by: Nir Rozenbaum --- config/charts/inferencepool/README.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/config/charts/inferencepool/README.md b/config/charts/inferencepool/README.md index 7dbdfa2f2..1895c6af8 100644 --- a/config/charts/inferencepool/README.md +++ b/config/charts/inferencepool/README.md @@ -50,19 +50,19 @@ $ helm install vllm-llama3-8b-instruct ./config/charts/inferencepool -f values.y To set custom EPP plugin config, you can pass it as an inline yaml. For example: ```yaml - pluginsCustomConfig: - custom-plugins.yaml: | - apiVersion: inference.networking.x-k8s.io/v1alpha1 - kind: EndpointPickerConfig - plugins: - - type: custom-scorer - parameters: - custom-threshold: 64 - schedulingProfiles: - - name: default + inferenceExtension: + pluginsCustomConfig: + custom-plugins.yaml: | + apiVersion: inference.networking.x-k8s.io/v1alpha1 + kind: EndpointPickerConfig plugins: - - pluginRef: custom-scorer -``` + - type: custom-scorer + parameters: + custom-threshold: 64 + schedulingProfiles: + - name: default + plugins: + - pluginRef: custom-scorer ### Install with Additional Ports From 06dfc684925a42646f5459a0d91258f85641e839 Mon Sep 17 00:00:00 2001 From: Kellen Swain Date: Thu, 4 Sep 2025 15:53:37 -0700 Subject: [PATCH 3/4] Update config/charts/inferencepool/README.md --- config/charts/inferencepool/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/config/charts/inferencepool/README.md b/config/charts/inferencepool/README.md index 1895c6af8..38f929a87 100644 --- a/config/charts/inferencepool/README.md +++ b/config/charts/inferencepool/README.md @@ -62,7 +62,8 @@ To set custom EPP plugin config, you can pass it as an inline yaml. For example: schedulingProfiles: - name: default plugins: - - pluginRef: custom-scorer + - pluginRef: custom-scorer + ``` ### Install with Additional Ports From c66d169752ce2d1ad0bdd8c39ded4a6f080afa5d Mon Sep 17 00:00:00 2001 From: Kellen Swain Date: Thu, 4 Sep 2025 15:54:23 -0700 Subject: [PATCH 4/4] Update config/charts/inferencepool/README.md --- config/charts/inferencepool/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/charts/inferencepool/README.md b/config/charts/inferencepool/README.md index 38f929a87..82be6b85c 100644 --- a/config/charts/inferencepool/README.md +++ b/config/charts/inferencepool/README.md @@ -63,7 +63,7 @@ To set custom EPP plugin config, you can pass it as an inline yaml. For example: - name: default plugins: - pluginRef: custom-scorer - ``` +``` ### Install with Additional Ports