Skip to content

Commit 5b359a1

Browse files
rahulgurnaninirrozenbaumkfswain
authored
Update helm chart Readme with custom plugin config (#1516)
* Update helm chart README with pluginsCustomConfig flag * Update config/charts/inferencepool/README.md Co-authored-by: Nir Rozenbaum <[email protected]> * Update config/charts/inferencepool/README.md * Update config/charts/inferencepool/README.md --------- Co-authored-by: Nir Rozenbaum <[email protected]> Co-authored-by: Kellen Swain <[email protected]>
1 parent d77ad92 commit 5b359a1

File tree

1 file changed

+21
-0
lines changed

1 file changed

+21
-0
lines changed

config/charts/inferencepool/README.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,26 @@ Then apply it with:
4545
$ helm install vllm-llama3-8b-instruct ./config/charts/inferencepool -f values.yaml
4646
```
4747

48+
### Install with Custom EPP Plugins Configuration
49+
50+
To set custom EPP plugin config, you can pass it as an inline yaml. For example:
51+
52+
```yaml
53+
inferenceExtension:
54+
pluginsCustomConfig:
55+
custom-plugins.yaml: |
56+
apiVersion: inference.networking.x-k8s.io/v1alpha1
57+
kind: EndpointPickerConfig
58+
plugins:
59+
- type: custom-scorer
60+
parameters:
61+
custom-threshold: 64
62+
schedulingProfiles:
63+
- name: default
64+
plugins:
65+
- pluginRef: custom-scorer
66+
```
67+
4868
### Install with Additional Ports
4969
5070
To expose additional ports (e.g., for ZMQ), you can define them in the `values.yaml` file:
@@ -126,6 +146,7 @@ The following table list the configurable parameters of the chart.
126146
| `inferenceExtension.affinity` | Affinity for the endpoint picker. Defaults to `{}`. |
127147
| `inferenceExtension.tolerations` | Tolerations for the endpoint picker. Defaults to `[]`. |
128148
| `inferenceExtension.flags.has-enable-leader-election` | Enable leader election for high availability. When enabled, only one EPP pod (the leader) will be ready to serve traffic. |
149+
| `inferenceExtension.pluginsCustomConfig` | Custom config that is passed to EPP as inline yaml. |
129150
| `provider.name` | Name of the Inference Gateway implementation being used. Possible values: `gke`. Defaults to `none`. |
130151

131152
## Notes

0 commit comments

Comments
 (0)