Skip to content

Commit 147f01f

Browse files
committed
implement pod discovery for KVEvents subscription mgmt + blocking module rename artifacts
Signed-off-by: Maroon Ayoub <[email protected]>
1 parent d7fd218 commit 147f01f

File tree

17 files changed

+1047
-35
lines changed

17 files changed

+1047
-35
lines changed

Dockerfile

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ ENV PYTHON=python3.9
6161
RUN export CGO_CFLAGS="$(python3.12-config --cflags) -I/workspace/lib" && \
6262
export CGO_LDFLAGS="$(python3.12-config --ldflags --embed) -L/workspace/lib -ltokenizers -ldl -lm" && \
6363
CGO_ENABLED=1 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH:-amd64} \
64-
go build -a -o bin/kv-cache-manager examples/kv_events/online/main.go
64+
go build -a -o bin/kv-cache examples/kv_events/online/main.go
6565

6666
# Use distroless as minimal base image to package the manager binary
6767
# Refer to https://github.com/GoogleContainerTools/distroless for more details
@@ -90,8 +90,8 @@ COPY --from=builder /workspace/pkg/preprocessing/chat_completions /app/pkg/prepr
9090
ENV PYTHONPATH=/app/pkg/preprocessing/chat_completions:/usr/lib64/python3.12/site-packages
9191

9292
# Copy the compiled Go application
93-
COPY --from=builder /workspace/bin/kv-cache-manager /app/kv-cache-manager
93+
COPY --from=builder /workspace/bin/kv-cache /app/kv-cache
9494
USER 65532:65532
9595

96-
# Set the entrypoint to the kv-cache-manager binary
97-
ENTRYPOINT ["/app/kv-cache-manager"]
96+
# Set the entrypoint to the kv-cache binary
97+
ENTRYPOINT ["/app/kv-cache"]

Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
SHELL := /usr/bin/env bash
22

33
# Defaults
4-
PROJECT_NAME ?= llm-d-kv-cache-manager
4+
PROJECT_NAME ?= llm-d-kv-cache
55
DEV_VERSION ?= 0.0.1
66
PROD_VERSION ?= 0.0.0
77
IMAGE_TAG_BASE ?= ghcr.io/llm-d/$(PROJECT_NAME)

docs/configuration.md

Lines changed: 101 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -334,16 +334,114 @@ Configures the available device backends which store the KV Cache blocks. This w
334334

335335
### KV-Event Pool Configuration (`Config`)
336336

337-
Configures the ZMQ event processing pool for handling KV cache events.
337+
Configures the ZMQ event processing pool for handling KV cache events. The pool supports two modes:
338+
1. **Global Socket Mode**: Connects to a single ZMQ endpoint
339+
2. **Pod Reconciler Mode** (default): Automatically discovers and subscribes to per-pod ZMQ endpoints
338340

339341
```json
340342
{
341-
"zmqEndpoint": "tcp://*:5557",
342343
"topicFilter": "kv@",
343-
"concurrency": 4
344+
"concurrency": 16,
345+
"discoverPods": true
344346
}
345347
```
346348

349+
| Field | Type | Description | Default |
350+
|-------|-----------------------------------------------------------------------|-------------|---------|
351+
| `zmqEndpoint` | `string` | ZMQ address to connect to | `""` |
352+
| `topicFilter` | `string` | ZMQ subscription filter | `"kv@"` |
353+
| `concurrency` | `integer` | Number of parallel workers | `4` |
354+
| `discoverPods` | `boolean` | Enable Kubernetes pod reconciler for automatic per-pod subscriber management | `true` |
355+
| `podDiscoveryConfig` | [PodDiscoveryConfig](#pod-discovery-configuration-podDiscoveryConfig) | Configuration for pod reconciler (only used when `discoverPods` is true) | `null` |
356+
357+
#### Global Socket Mode Example
358+
359+
For connecting to a single ZMQ endpoint:
360+
361+
```json
362+
{
363+
"zmqEndpoint": "tcp://indexer:5557",
364+
"topicFilter": "kv@",
365+
"concurrency": 8,
366+
"discoverPods": false
367+
}
368+
```
369+
370+
#### Pod Reconciler Mode Example
371+
372+
For automatic Kubernetes pod discovery:
373+
374+
```json
375+
{
376+
"topicFilter": "kv@",
377+
"concurrency": 8,
378+
"discoverPods": true,
379+
"podDiscoveryConfig": {
380+
"podLabelSelector": "llm-d.ai/inferenceServing=true",
381+
"podNamespace": "inference",
382+
"socketPort": 5557,
383+
}
384+
}
385+
```
386+
387+
### Pod Discovery Configuration (`PodDiscoveryConfig`)
388+
389+
Configures the Kubernetes pod reconciler for automatic per-pod ZMQ subscriber management. The reconciler watches Kubernetes pods and dynamically creates/removes ZMQ subscribers based on pod lifecycle.
390+
391+
```json
392+
{
393+
"podLabelSelector": "llm-d.ai/inferenceServing=true",
394+
"podNamespace": "",
395+
"socketPort": 5557,
396+
}
397+
```
398+
399+
| Field | Type | Description | Default |
400+
|-------|------|-------------|-----------------------|
401+
| `podLabelSelector` | `string` | Label selector for filtering which pods to watch. Examples: `"app=vllm"`, `"app=vllm,tier=gpu"` | `"llm-d.ai/inferenceServing=true"` |
402+
| `podNamespace` | `string` | Namespace to watch pods in. If empty, watches all namespaces (requires cluster-wide RBAC) | `""` (all namespaces) |
403+
| `socketPort` | `integer` | Port number where vLLM pods expose their ZMQ socket | `5557` |
404+
405+
#### Pod Requirements
406+
407+
For the reconciler to create a subscriber for a pod, the pod must meet these conditions:
408+
409+
1. **Match label selector**: Pod labels must match the configured `podLabelSelector`
410+
2. **Running state**: `pod.Status.Phase == Running`
411+
3. **Has IP address**: `pod.Status.PodIP != ""`
412+
4. **Ready condition**: Pod has condition `PodReady == ConditionTrue`
413+
414+
When any of these conditions becomes false, the subscriber is automatically removed.
415+
416+
#### RBAC Requirements
417+
418+
When using the pod reconciler, ensure the service account has appropriate RBAC permissions:
419+
420+
**Namespace-scoped** (when `podNamespace` is set):
421+
```yaml
422+
apiVersion: rbac.authorization.k8s.io/v1
423+
kind: Role
424+
metadata:
425+
name: kv-cache-manager
426+
namespace: inference
427+
rules:
428+
- apiGroups: [""]
429+
resources: ["pods"]
430+
verbs: ["get", "list", "watch"]
431+
```
432+
433+
**Cluster-wide** (when `podNamespace` is empty):
434+
```yaml
435+
apiVersion: rbac.authorization.k8s.io/v1
436+
kind: ClusterRole
437+
metadata:
438+
name: kv-cache-manager
439+
rules:
440+
- apiGroups: [""]
441+
resources: ["pods"]
442+
verbs: ["get", "list", "watch"]
443+
```
444+
347445
## Event Processing Configuration Example
348446

349447
For the ZMQ event processing pool:
@@ -356,11 +454,6 @@ For the ZMQ event processing pool:
356454
}
357455
```
358456

359-
| Field | Type | Description | Default |
360-
|-------|------|-------------|---------|
361-
| `zmqEndpoint` | `string` | ZMQ address to connect to | `"tcp://*:5557"` |
362-
| `topicFilter` | `string` | ZMQ subscription filter | `"kv@"` |
363-
| `concurrency` | `integer` | Number of parallel workers | `4` |
364457

365458
## Token Processing Configuration
366459

examples/helper/events.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ import (
2121

2222
"github.com/llm-d/llm-d-kv-cache/examples/testdata"
2323
"github.com/llm-d/llm-d-kv-cache/pkg/kvcache/kvblock"
24-
"github.com/llm-d/llm-d-kv-cache/pkg/kvcache/kvevents"
24+
"github.com/llm-d/llm-d-kv-cache/pkg/kvevents"
2525
"github.com/llm-d/llm-d-kv-cache/pkg/utils"
2626
"github.com/vmihailenco/msgpack/v5"
2727
"sigs.k8s.io/controller-runtime/pkg/log"

examples/kv_cache_index_service/server/server.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ import (
2222
"github.com/llm-d/llm-d-kv-cache/examples/testdata"
2323
"github.com/llm-d/llm-d-kv-cache/pkg/kvcache"
2424
"github.com/llm-d/llm-d-kv-cache/pkg/kvcache/kvblock"
25-
"github.com/llm-d/llm-d-kv-cache/pkg/kvcache/kvevents"
25+
"github.com/llm-d/llm-d-kv-cache/pkg/kvevents"
2626
"github.com/llm-d/llm-d-kv-cache/pkg/utils"
2727
)
2828

examples/kv_events/online/main.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ import (
3131
"github.com/llm-d/llm-d-kv-cache/examples/testdata"
3232
"github.com/llm-d/llm-d-kv-cache/pkg/kvcache"
3333
"github.com/llm-d/llm-d-kv-cache/pkg/kvcache/kvblock"
34-
"github.com/llm-d/llm-d-kv-cache/pkg/kvcache/kvevents"
34+
"github.com/llm-d/llm-d-kv-cache/pkg/kvevents"
3535
preprocessing "github.com/llm-d/llm-d-kv-cache/pkg/preprocessing/chat_completions"
3636
"github.com/llm-d/llm-d-kv-cache/pkg/tokenization"
3737
"github.com/prometheus/client_golang/prometheus/promhttp"
Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
/*
2+
Copyright 2025 The llm-d Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package main
18+
19+
import (
20+
"context"
21+
"os"
22+
"os/signal"
23+
"syscall"
24+
25+
"k8s.io/apimachinery/pkg/runtime"
26+
clientgoscheme "k8s.io/client-go/kubernetes/scheme"
27+
ctrl "sigs.k8s.io/controller-runtime"
28+
"sigs.k8s.io/controller-runtime/pkg/log"
29+
"sigs.k8s.io/controller-runtime/pkg/log/zap"
30+
31+
"github.com/llm-d/llm-d-kv-cache/pkg/kvcache/kvblock"
32+
"github.com/llm-d/llm-d-kv-cache/pkg/kvevents"
33+
"github.com/llm-d/llm-d-kv-cache/pkg/kvevents/controller"
34+
)
35+
36+
func main() {
37+
ctx, cancel := context.WithCancel(context.Background())
38+
defer cancel()
39+
40+
// Setup graceful shutdown
41+
sigChan := make(chan os.Signal, 1)
42+
signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
43+
go func() {
44+
<-sigChan
45+
cancel()
46+
}()
47+
48+
// Setup logger
49+
ctrl.SetLogger(zap.New(zap.UseDevMode(true)))
50+
logger := log.FromContext(ctx)
51+
52+
if err := run(ctx); err != nil {
53+
logger.Error(err, "Failed to run pod reconciler example")
54+
os.Exit(1) //nolint:gocritic
55+
}
56+
}
57+
58+
func run(ctx context.Context) error {
59+
logger := log.FromContext(ctx)
60+
61+
// Create a scheme for controller-runtime
62+
scheme := runtime.NewScheme()
63+
if err := clientgoscheme.AddToScheme(scheme); err != nil {
64+
return err
65+
}
66+
67+
// Create a controller-runtime manager
68+
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
69+
Scheme: scheme,
70+
})
71+
if err != nil {
72+
return err
73+
}
74+
75+
// Setup KV Cache Index
76+
indexConfig := kvblock.DefaultIndexConfig()
77+
index, err := kvblock.NewIndex(ctx, indexConfig)
78+
if err != nil {
79+
logger.Error(err, "failed to create index")
80+
return err
81+
}
82+
83+
// Setup event pool
84+
poolConfig := kvevents.DefaultConfig()
85+
tokenProcessor := kvblock.NewChunkedTokenDatabase(kvblock.DefaultTokenProcessorConfig())
86+
pool := kvevents.NewPool(poolConfig, index, tokenProcessor)
87+
pool.Start(ctx)
88+
89+
// Create subscriber manager
90+
subscriberManager := kvevents.NewSubscriberManager(pool)
91+
92+
// Convert to internal reconciler config
93+
reconcilerConfig, err := controller.NewPodReconcilerConfig(
94+
poolConfig.PodDiscoveryConfig,
95+
poolConfig.TopicFilter,
96+
)
97+
if err != nil {
98+
logger.Error(err, "failed to create reconciler config")
99+
return err
100+
}
101+
102+
// Create and register the pod reconciler
103+
podReconciler := &controller.PodReconciler{
104+
Client: mgr.GetClient(),
105+
Scheme: mgr.GetScheme(),
106+
Config: reconcilerConfig,
107+
SubscriberManager: subscriberManager,
108+
}
109+
110+
if err := podReconciler.SetupWithManager(mgr); err != nil {
111+
logger.Error(err, "failed to setup pod reconciler")
112+
return err
113+
}
114+
115+
logger.Info("=== Pod Reconciler Example Started ===")
116+
logger.Info("Watching pods with label selector", "selector", poolConfig.PodDiscoveryConfig.PodLabelSelector)
117+
logger.Info("Topic filter", "filter", poolConfig.TopicFilter)
118+
119+
// Start the manager (this will start the reconciler)
120+
mgrCtx, mgrCancel := context.WithCancel(ctx)
121+
defer mgrCancel()
122+
123+
go func() {
124+
if err := mgr.Start(mgrCtx); err != nil {
125+
logger.Error(err, "failed to start manager")
126+
}
127+
}()
128+
129+
// Wait for shutdown
130+
<-ctx.Done()
131+
logger.Info("Shutting down pod reconciler example...")
132+
133+
// Shutdown subscriber manager
134+
subscriberManager.Shutdown(ctx)
135+
136+
// Shutdown pool
137+
pool.Shutdown(ctx)
138+
139+
logger.Info("Pod reconciler example shut down successfully")
140+
return nil
141+
}

go.sum

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@ github.com/alicebob/miniredis/v2 v2.35.0 h1:QwLphYqCEAo1eu1TqPRN2jgVMPBweeQcR21j
22
github.com/alicebob/miniredis/v2 v2.35.0/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM=
33
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
44
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
5+
github.com/blang/semver/v4 v4.0.0 h1:1PFHFE6yCCTv8C1TeyNNarDzntLi7wMI5i/pzqYIsAM=
6+
github.com/blang/semver/v4 v4.0.0/go.mod h1:IbckMUScFkM3pff0VJDNKRiT6TG/YpiHIM2yvyW5YoQ=
57
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
68
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
79
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
@@ -25,6 +27,12 @@ github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkp
2527
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
2628
github.com/emicklei/go-restful/v3 v3.11.0 h1:rAQeMHw1c7zTmncogyy8VvRZwtkmkZ4FxERmMY4rD+g=
2729
github.com/emicklei/go-restful/v3 v3.11.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc=
30+
github.com/evanphx/json-patch v0.5.2 h1:xVCHIVMUu1wtM/VkR9jVZ45N3FhZfYMMYGorLCR8P3k=
31+
github.com/evanphx/json-patch v0.5.2/go.mod h1:ZWS5hhDbVDyob71nXKNL0+PWn6ToqBHMikGIFbs31qQ=
32+
github.com/evanphx/json-patch/v5 v5.9.11 h1:/8HVnzMq13/3x9TPvjG08wUGqBTmZBsCWzjTM0wiaDU=
33+
github.com/evanphx/json-patch/v5 v5.9.11/go.mod h1:3j+LviiESTElxA4p3EMKAB9HXj3/XEtnUf6OZxqIQTM=
34+
github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA=
35+
github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM=
2836
github.com/fxamacker/cbor/v2 v2.7.0 h1:iM5WgngdRBanHcxugY4JySA0nk1wZorNOpTgCMedv5E=
2937
github.com/fxamacker/cbor/v2 v2.7.0/go.mod h1:pxXPTn3joSm21Gbwsv0w9OSA2y1HFR9qXEeXQVeNoDQ=
3038
github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
@@ -45,12 +53,16 @@ github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
4553
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
4654
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
4755
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
56+
github.com/google/btree v1.1.3 h1:CVpQJjYgC4VbzxeGVHfvZrv1ctoYCAI8vbl07Fcxlyg=
57+
github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4=
4858
github.com/google/gnostic-models v0.6.9 h1:MU/8wDLif2qCXZmzncUQ/BOfxWfthHi63KqpoNbWqVw=
4959
github.com/google/gnostic-models v0.6.9/go.mod h1:CiWsm0s6BSQd1hRn8/QmxqB6BesYcbSZxsz9b0KuDBw=
5060
github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
5161
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
5262
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
5363
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
64+
github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0=
65+
github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
5466
github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db h1:097atOisP2aRj7vFgYQBbFN4U4JNXUNYpxael3UzMyo=
5567
github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db/go.mod h1:vavhavw2zAxS5dIdcRluK6cSGGPlZynqzFM8NdvU144=
5668
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
@@ -174,6 +186,8 @@ golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8T
174186
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
175187
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
176188
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
189+
gomodules.xyz/jsonpatch/v2 v2.4.0 h1:Ci3iUJyx9UeRx7CeFN8ARgGbkESwJK+KB9lLcWxY/Zw=
190+
gomodules.xyz/jsonpatch/v2 v2.4.0/go.mod h1:AH3dM2RI6uoBZxn3LVrfvJ3E0/9dG4cSrbuBJT4moAY=
177191
google.golang.org/genproto/googleapis/rpc v0.0.0-20241209162323-e6fa225c2576 h1:8ZmaLZE4XWrtU3MyClkYqqtl6Oegr3235h7jxsDyqCY=
178192
google.golang.org/genproto/googleapis/rpc v0.0.0-20241209162323-e6fa225c2576/go.mod h1:5uTbfoYQed2U9p3KIj2/Zzm02PYhndfdmML0qC3q3FU=
179193
google.golang.org/grpc v1.68.1 h1:oI5oTa11+ng8r8XMMN7jAOmWfPZWbYpCFaMUTACxkM0=
@@ -192,6 +206,8 @@ gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
192206
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
193207
k8s.io/api v0.33.0 h1:yTgZVn1XEe6opVpP1FylmNrIFWuDqe2H0V8CT5gxfIU=
194208
k8s.io/api v0.33.0/go.mod h1:CTO61ECK/KU7haa3qq8sarQ0biLq2ju405IZAd9zsiM=
209+
k8s.io/apiextensions-apiserver v0.33.0 h1:d2qpYL7Mngbsc1taA4IjJPRJ9ilnsXIrndH+r9IimOs=
210+
k8s.io/apiextensions-apiserver v0.33.0/go.mod h1:VeJ8u9dEEN+tbETo+lFkwaaZPg6uFKLGj5vyNEwwSzc=
195211
k8s.io/apimachinery v0.33.0 h1:1a6kHrJxb2hs4t8EE5wuR/WxKDwGN1FKH3JvDtA0CIQ=
196212
k8s.io/apimachinery v0.33.0/go.mod h1:BHW0YOu7n22fFv/JkYOEfkUYNRN0fj0BlvMFWA7b+SM=
197213
k8s.io/client-go v0.33.0 h1:UASR0sAYVUzs2kYuKn/ZakZlcs2bEHaizrrHUZg0G98=

0 commit comments

Comments
 (0)