add parameters section to Config API

[nirrozenbaum]

as of today, config API defines two main sections - plugins and scheduling-profiles.

In llm-d we've noticed that there is a use case for multiple plugins (more than one) that are using the same parameter. that means we need to define the same parameter twice in two different places, and if the values are not matching the behavior could be unexpected.

an example of a config file could look like this (hashBlockSize: 64 parameter is defined twice):

apiVersion: inference.networking.x-k8s.io/v1alpha1
kind: EndpointPickerConfig
plugins:
- type: prefix-cache-scorer
  parameters:
    hashBlockSize: 64
    maxPrefixBlocksToMatch: 256
    lruCapacityPerServer: 31250
- type: max-score-picker
  parameters:
    maxNumOfEndpoints: 1
- type: single-profile-handler
- type: my-plugin
  parameters:
    hashBlockSize: 64
schedulingProfiles:
- name: default
  plugins:
  - pluginRef: prefix-cache-scorer
  - pluginRef: max-score-picker

as an enhancement, I would like to propose adding a parameters section and the usage of parameterRef which may be identical to pluginRef. the intention is to allow simple key value pairs and not complex structs (should be similar to env var).

so one could define the above config as follows:

apiVersion: inference.networking.x-k8s.io/v1alpha1
kind: EndpointPickerConfig
parameters:
- name: hashBlockSize
  value: 64
plugins:
- type: prefix-cache-scorer
  parameters:
    parameterRef: hashBlockSize
    maxPrefixBlocksToMatch: 256
    lruCapacityPerServer: 31250
- type: max-score-picker
  parameters:
    maxNumOfEndpoints: 1
- type: single-profile-handler
- type: my-plugin
  parameters:
    parameterRef: hashBlockSize
schedulingProfiles:
- name: default
  plugins:
  - pluginRef: prefix-cache-scorer
  - pluginRef: max-score-picker

doing the above gives the look and feel of using env vars that potentially could be read from different plugins.
it reduces the risk of configuration inconsistencies when there is a need to share parameters across plugins.

cc: @shmuelk @ahg-g @elevran @kfswain

[shmuelk]

I have a couple of issue with the proposal above as it stands:

1. Very minor, as suggested, you can only have one reference in a set of parameters.
   Clearly, the parameterRef tag could become an array, but that doesn't really solve the bigger issues.
2. You have now added a "reserved key" into the plugin parameters, which untill now were basically what ever the plugin developer wanted.
3. You only support top level parameter keys. It could be argued that plugin parameter sets shouldn't be that complicated and should be flat.
4. You are forcing all uses of the "global parameter" to use the same key in their parameter set.

A couple of other options come to mind:

1. Support some sort of "text/value substitution". The text substitution could be done before the plugin sees the raw parameter text and thus would be oblivious to this.
   This theoretically enables adding keys "dynamically" and not just value.
   There is the obvious disadvantage that there will be sort of reserved string format that will cause the substitutions (perhaps ${name}).

2. Instead of the above proposal of:

   • type: plugin-type
     parameters:
     parameterRef: name
     plover: 42

   add a separate subsection:

   • type: plugin-type
     parameters:
     plover: 42
     parameterRefs:
     • key: theKey
       ref: name

Those suggestions are the upwards compatible ones. One could instead do something much more like environment variables in a K8S pod yaml, as in:

 - type: plugin-type
    parameters:
    - key: plover
       value: 42
    - key: theKey
       reference: theName

The API to the plugins wouldn't need to change, but clearly the config file format would change. We could use a new version for the config and perform automatic conversions of older configuration text.

This also has the minor disadvantage of forcing plugin configurations to be flat. But it does open more future opportunities as things come up.

[nirrozenbaum]

Thanks @shmuelk 🙏🏼.
Your last suggestion to define parameters in a syntax very close to how env vars in k8s yamls are defined sounds to me like the best option, but I don't have a strong preference on the syntax we select.

As long as we allow reusing the same parameter in multiple places, I'm ok with any syntax that makes sense.

[danehans]

A few pieces of feedback from a quick review of the API:

1. SchedulingProfiles should be a listMap keyed by Name.
2. I don't see any defaulting being used. For example, if Plugins and/or SchedulingProfiles is unset, can a default set of plugins and/or profiles be set? I would think we would have a baseline config that can be defaulted.
3. PluginRef- a xRef field implies a GVR reference. Consider using a []SupportedPluginTypes.
4. plugins.type- Should be camelCase (xref).
5. Weight- Update the godocs to describe weight.
6. Validation- Set min/max for field values. A follow-up issue should be created for additional validation using CEL. However, since this API is consumed as opaque data in a ConfigMap, EPP should have a validation layer that implements the validation requirements. It might be better for EndpointPickerConfig to be a separate resource that is immutable until dynamic reconfig is required.
7. Is schedulingProfiles required? The two in-tree examples each define a single profile. If so, the API should start with either schedulingProfiles that must define at least 1 supported plugin and remove the top-level plugins.
8. Adding a top-level parameters will introduce another list that needs to be validated against the other lists, etc.. This should be done with caution. Maybe plugins contain a list of supported overrides to the top-level parameters. Additionally, the top-level parameters list should be defaulted with default params used by the plugins.

[danehans]

Currently, several EPP features are gated through env vars. Instead, add feature gates to the EndpointPickerConfig API.

xref: #1391 (comment)