Skip to content

todo: render host-fabric access into engine Pods on RDMA and EFA pools #286

Description

@thxCode

What is left undone

On a host-fabric transport (RDMA or EFA), the operator renders everything the fabric needs —
hostNetwork, /dev/infiniband, IPC_LOCK/SYS_RESOURCE, and on EFA the host's
/opt/amazon/efa tree plus LD_LIBRARY_PATHinto the backend's member Pods only. The
engine Pods (vLLM / SGLang) that consume the pool get none of it: the injection webhook renders
the transport name (MOONCAKE_PROTOCOL / mooncake.json protocol) and nothing the fabric
needs.

On TCP that distinction costs nothing — it is plain pod networking. On RDMA or EFA, an
engine that is to move bytes over the fabric needs the same device and library access on its own
Pod, and today its transfer engine either fails to install the transport or silently falls back.

This gap predates EFARDMA has always had it; EFA inherits it. The enum's own bar ("only
TCP has been exercised end to end") is downstream of exactly this.

What deciding this looks like

  • Whether engine Pods should get fabric access at all: it grants hostNetwork and
    capabilities to tenant-adjacent workloads, a much wider blast radius than the cluster-scoped
    member DaemonSet. The member-side rule "a privilege is requested, never inferred" has no
    obvious engine-side spelling — there is no field on the engine's owning object that asks for
    it.
  • If yes, how: a per-pool or per-backend opt-in the injection webhook can see at mutate time,
    rendering the same host-fabric base (and the EFA extras) into the engine container; or a
    documented node-level answer (device plugin / CNI) the operator stays out of.
  • What the honest interim is: the docs now state the gap on both pages a reader would look
    for it (docs/kv-cache/backend.md, docs/reference/kv-cache-injection.md). If the decision is
    "members only, engines stay on TCP", that should be said as a rule, not left as an absence.

References

  • Member-side rendering (the half that exists): applyMemberFabric in
    pkg/worker/kvcache/mooncake/member_workload.go.
  • Engine-side injection (the half that only names the transport):
    pkg/worker/kvcache/inject/engine.go, pkg/worker/webhooks/worker/pod_kv_cache_resolve.go.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    todoWork a pull request knowingly left undone

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions