feat: add pdrole package for P/D role discovery and detection#735
Open
ev-shindin wants to merge 1 commit intollm-d:mainfrom
Open
feat: add pdrole package for P/D role discovery and detection#735ev-shindin wants to merge 1 commit intollm-d:mainfrom
ev-shindin wants to merge 1 commit intollm-d:mainfrom
Conversation
Add internal/utils/pdrole package that discovers Prefill/Decode role configuration from EPP's EndpointPickerConfig and detects each deployment's P/D role from pod template labels. Aligned with llm-d-inference-scheduler's actual behavior: - Label-only detection (no deployment name fallback). EPP's filter plugins (prefill-filter, decode-filter, by-label) route traffic based solely on pod labels, never deployment names. - prefill-filter: accepts only "prefill" labeled pods (allowsNoLabel=false) - decode-filter: accepts "decode"/"both" labeled + unlabeled pods (allowsNoLabel=true) - by-label: values appearing in both prefill and decode profiles' validValues are classified as BothValues (intersection logic) Discovery accepts a per-pool EndpointPool (not a global EPP name), enabling correct behavior when multiple EPPs have different P/D label settings. Returns PDDiscoveryResult with Disaggregated flag so callers can distinguish "no P/D plugins" (treat all as RoleBoth) from "P/D plugins found, use label config for detection".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
internal/utils/pdrolepackage that discovers P/D disaggregation configuration from the EPP'sEndpointPickerConfigand detects each deployment's P/D role from pod template labels*pool.EndpointPool), enabling correct behavior when multiple EPPs exist with different P/D label settingsPDDiscoveryResultwithDisaggregatedflag so callers can distinguish "no P/D plugins" (all pods serve both roles) from "P/D plugins found, use label config"Design decisions
Label-only detection (no deployment name fallback).
The EPP's filter plugins (
prefill-filter,decode-filter,by-labelfrom llm-d-inference-scheduler) route traffic based solely on pod labels. A deployment namedllama-prefillwithout allm-d.ai/rolelabel would still receive decode traffic from EPP (decode-filterhasallowsNoLabel=true). Name-based guessing would misclassify it.BothValues from intersection, not hardcoded.
For
by-labelcustom plugins, values appearing in both prefill and decode profiles'validValuesare classified asBothValues. This ensuresGetDeploymentPDRolereturnsRoleBoth(notRolePrefillorRoleDecode) for pods that pass both filters.Aligned with EPP's actual filter semantics:
allowsNoLabelprefill-filterllm-d.ai/role"prefill"falsedecode-filterllm-d.ai/role"decode","both"truePackage structure
types.goPDRole,PDRoleLabelConfig,PDDiscoveryResult, constantsdetect.goGetDeploymentPDRole— label-based role detectiondiscover.goDiscoverPDRoleLabelConfig— EPP config discovery chain*_test.goDiscovery chain
EndpointPickerConfig(YAML/JSON)prefill-filter/decode-filter(well-known label) orby-label(custom label from parameters)Disaggregated=false, default configIntended caller pattern (not wired yet)
Test plan
go build ./internal/utils/pdrole/...— cleango vet ./internal/utils/pdrole/...— cleango test ./internal/utils/pdrole/... -v -count=1— 43/43 specs pass