Enable prefix-cache awareness in active-active multi-replica scheduler deployments

[vMaroon]

Summary

active-active multi-replica scheduler support llm-d/llm-d-kv-cache#212

Summary

Implemented a pod reconciler controller that manages per-pod ZMQ subscribers for KVEvents processing, and the required logic. Also moved the kvevents to the same level of kvcache library, as should have been.

Components:

• PodReconciler: Kubernetes controller that watches pods and manages subscribers
  • This is one standalone approach, actual integration with the scheduler depends on [RFC] Pod lifecycle subscription through data layer kubernetes-sigs/gateway-api-inference-extension#2017
• SubscriberManager: Thread-safe manager for multiple ZMQ subscribers
• Updated kvevents.Pool - Now works with SubscriberManager instead of a global socket

Added integration + unit tests for all new functionality, updated documentation + examples.

The current integration manages pod discovery through looking at the available pods on every Score call. A pod that does not appear there in any request for 10 minutes is assumed dead.

A proper integration through the data-layer will be implemented once ready in IGW. Tracker: kubernetes-sigs/gateway-api-inference-extension#2017

[elevran]

/lgtm
/approve