Skip to content

Commit c28c5ff

Browse files
kcp-ci-botkcp-ci-bot
authored
Merge pull request #3672 from gman0/virtual-resources-docs
Add docs for virtual resources and CachedResources
2 parents 7368f3a + 11b95f7 commit c28c5ff

File tree

6 files changed

+309
-8
lines changed

6 files changed

+309
-8
lines changed

‎config/examples/virtualresources/README.md‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# VirtualResources Example
22

3-
This example shows usage of VirtualResources, together with CachedResources.
4-
The goal of VirtualResources is to distribute static, read-only resources to multiple clusters
3+
This example shows usage of virtual resources, together with CachedResources.
4+
The goal of CachedResources is to distribute static, read-only resources to multiple clusters
55
in a scalable way.
66

77
## Setup
@@ -19,15 +19,16 @@ in a scalable way.
1919
kubectl ws create provider --enter
2020

2121
kubectl create -f config/examples/virtualresources/crd-instances.yaml
22-
# this this to work we always require apiresource schema to be present
22+
# for this to work we always require apiresource schema to be present
2323
kubectl create -f config/examples/virtualresources/apiresourceschema-instances.yaml
2424
kubectl create -f config/examples/virtualresources/instances.yaml
2525

26-
# create caching for the resources
26+
# create caching for the resources with a pre-made CachedResources identity
27+
kubectl create -f config/examples/virtualresources/cached-resource-identity.yaml
2728
kubectl create -f config/examples/virtualresources/cached-resource-instances.yaml
2829
```
2930

30-
3. Create a an APIResourceSchema for actual virtual machines to be distributed,
31+
3. Create an APIResourceSchema for actual virtual machines to be distributed,
3132
which will be using instance types.
3233

3334
```bash
@@ -52,4 +53,4 @@ in a scalable way.
5253

5354
```bash
5455
kubectl get instances.machines.svm.io
55-
```
56+
```

‎config/examples/virtualresources/apiexport.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,9 @@ spec:
1313
apiGroup: cache.kcp.io
1414
kind: CachedResourceEndpointSlice
1515
name: instances
16-
identityHash: 2857921554ab76ec50f25bf083b7aeb4f7808cd169fd2945b007429f426614ec
16+
identityHash: 676a1a997f7507fdfcfd1fcf35920efdff1297cd215cad4f486220c30a5c7090
1717
- name: virtualmachines
1818
group: machines.svm.io
1919
schema: today.virtualmachines.machines.svm.io
2020
storage:
21-
crd: {}
21+
crd: {}

‎config/examples/virtualresources/cached-resource-identity.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
apiVersion: v1
2+
kind: Secret
3+
metadata:
4+
name: instances-cr-identity
5+
namespace: default
6+
stringData:
7+
key: instances-cr-identity-123

‎config/examples/virtualresources/cached-resource-instances.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,7 @@ spec:
99
labelSelector:
1010
matchLabels:
1111
app.kubernetes.io/part-of: instances
12+
identity:
13+
secretRef:
14+
name: instances-cr-identity
15+
namespace: default

‎docs/content/concepts/apis/cached-resources.md‎

Lines changed: 233 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,233 @@
1+
# CachedResource API
2+
3+
!!! warning
4+
As of 0.29, this feature is of alpha-version quality. To use it, enable the `CachedAPIs` feature gate.
5+
6+
A CachedResource object replicates a user-defined resource from its workspace into kcp's [cache server](../sharding/cache-server.md). It may be used as an [APIExport virtual resource](./exporting-apis.md#virtual-resources) with the goal of distributing static, read-only resources to multiple clusters, in a scalable way.
7+
8+
**Example user story:**
9+
10+
- You, the **service provider**, run a cloud _Cloud Co._, including a compute service.
11+
- You offer a service of provisioning and running VM instances: users create an `Instance` object and voilà, they have a running VM!
12+
- But how do you design the API for configuring such a resource? How do you make your consumers know what configuration options are available, given the limited resources available in the cloud?
13+
- You've decided to offer different packages for CPUs, memory, storage, GPUs; these are represented as CRDs, e.g. `cpuflavors.cloud.example.com`, `memflavors.cloud.example.com`, with `cpu-small`, `cpu-medium`, `cpu-large`, `mem-medium`, `mem-large`, `mem-xlarge` respectively.
14+
- You've created a CachedResource for each flavor type.
15+
- You offer the service through an APIExport containing the main `instances.cloud.example.com` resource, as well as all flavors. These are exported as [virtual resources](./exporting-apis.md#virtual-resources).
16+
- **Consumers** binding to your APIExport can list and get the available flavors from within their workspace (e.g. with `kubectl get cpuflavors`), and refer to them in their `Instance` spec. They cannot create, delete or otherwise modify the flavor objects in any way.
17+
18+
See an example usage at [github.com/kcp-dev/kcp/tree/main/config/examples/virtualresources](https://github.com/kcp-dev/kcp/tree/main/config/examples/virtualresources).
19+
20+
## Resource replication
21+
22+
```yaml
23+
apiVersion: cache.kcp.io/v1alpha1
24+
kind: CachedResource
25+
metadata:
26+
name: cpuflavors-v1
27+
spec:
28+
group: cloud.example.com
29+
version: v1
30+
resource: cpuflavors
31+
```
32+
33+
The snippet above shows an example where all `cpuflavors.v1.cloud.example.com` objects in the workspace are replicated to the cache. There are some constraints on what resources may be replicated:
34+
35+
- There may be only one CachedResource for a particular group-version-resource triplet in the workspace.
36+
- The resource must be cluster scoped.
37+
- The resource must not be a [built-in API](./built-in.md) or kcp system API belonging to `apis.kcp.io` group.
38+
- The resource may be originating from a CRD or an APIBinding.
39+
40+
Once created, resource replication progress may be checked in CachedResource's status:
41+
42+
```yaml
43+
apiVersion: cache.kcp.io/v1alpha1
44+
kind: CachedResource
45+
metadata:
46+
name: cpuflavors-v1
47+
status:
48+
conditions:
49+
- lastTransitionTime: "2025-10-21T14:03:41Z"
50+
status: "True"
51+
type: IdentityValid
52+
- lastTransitionTime: "2025-10-21T14:03:42Z"
53+
status: "True"
54+
type: ReplicationStarted
55+
- lastTransitionTime: "2025-10-21T14:03:41Z"
56+
status: "True"
57+
type: ResourceValid
58+
identityHash: cd2eb0837...
59+
phase: Ready
60+
resourceCounts:
61+
cache: 8 # (1)
62+
local: 8 # (2)
63+
```
64+
65+
1. `cache` resource count refers to the count of objects currently in cache for this CachedResource.
66+
2. `local` resource count refers to the count of objects the CachedResource currently sees in its workspace.
67+
68+
The objects a CachedResource is watching are always replicated in the direction **from** CachedResource's workspace **into** cache. Note that this means the only way to modify the in-cache copies is to modify the original objects. In-cache objects can be then projected into a workspace as a read-only API. This is done by creating a respective APIExport with [CachedResource virtual resource](#exporting-cachedresources), and binding to it.
69+
70+
```mermaid
71+
flowchart TD
72+
cacheServer["Cache server"]
73+
74+
subgraph provider["API Provider Workspace"]
75+
cpuflavorsCRD["CPUFlavors CRD"]
76+
cpuflavorCRs["CPUFlavor objects..."]
77+
78+
cpuflavorsCachedResource["CPUFlavors CachedResource"]
79+
80+
cpuflavorCRs -."From".-> cpuflavorsCRD
81+
cpuflavorsCachedResource -.Watches.-> cpuflavorCRs
82+
end
83+
84+
cpuflavorsCachedResource -."Replicates CPUFlavor objects into".-> cacheServer
85+
```
86+
87+
You can optionally configure the following additional aspects of a CachedResource:
88+
89+
- its identity
90+
- its endpoint slice
91+
- resource selector
92+
93+
We'll talk about each of these next.
94+
95+
### CachedResource identity
96+
97+
Similar to the [APIExport identity](./exporting-apis.md#apiexport-identity) concept, there may be many CachedResources with the same group-version-resource triplet across kcp instances. To differentiate between them and identify owners, a CachedResource object uses a unique identity key stored in a secret.
98+
99+
The identity **key** is considered a private key and should not be shared.
100+
101+
**Hash** calculated from that key, found at `.status.identityHash`, is considered a public key. APIExport's virtual resource definition expects this identity hash to be supplied when exporting a CachedResource.
102+
103+
By default, creating a CachedResource object triggers creation of an identity secret with a randomly generated key in its `key` data item. You can provide your own key by referencing your secret in the object's spec:
104+
105+
```yaml
106+
apiVersion: v1
107+
kind: Secret
108+
metadata:
109+
name: my-cached-cpuflavors-identity
110+
namespace: default
111+
stringData:
112+
key: "<Your identity key>"
113+
---
114+
apiVersion: cache.kcp.io/v1alpha1
115+
kind: CachedResource
116+
metadata:
117+
name: cpuflavors-v1
118+
spec:
119+
identity:
120+
secretRef:
121+
name: my-cached-cpuflavors-identity
122+
namespace: default
123+
...
124+
```
125+
126+
### CachedResourceEndpointSlice
127+
128+
While CachedResources replicates their associated resources to the cache server, retrieval of these in-cache resources is done through the Replication [virtual workspace](../workspaces/virtual-workspaces.md). The Replication VW is an API server dedicated to CachedResources, and is able to access the resource's objects with read-only verbs get, list and watch.
129+
130+
The Replication VW endpoints are listed in CachedResourceEndpointSlice. This endpoint slice is compatible with APIExport's [virtual resources](./exporting-apis.md#virtual-resources) and can be used as storage source.
131+
132+
With that said, the Replication VW is consumer-aware, and needs a valid APIExport and APIBinding(s) to operate. Therefore it is much more convenient to access the in-cache resources through regular APIBindings in a workspace and/or the APIExport VW rather than accessing them through the Replication VW. Consider the Replication VW just as an implementation detail of the CachedResource API, and when consuming it, use APIExports.
133+
134+
```mermaid
135+
flowchart TD
136+
cacheServer["Cache server"]
137+
replicationVW["Replication VW"]
138+
139+
subgraph provider["API Provider Workspace"]
140+
cpuflavorsCachedResource["CPUFlavors CachedResource"]
141+
cpuflavorsCachedResourceEndpointSlice["CPUFlavors CachedResourceEndpointSlice"]
142+
143+
cpuflavorsCachedResourceEndpointSlice --> cpuflavorsCachedResource
144+
end
145+
146+
replicationVW -."Reads from".-> cacheServer
147+
cpuflavorsCachedResourceEndpointSlice -."Has an endpoint for".-> replicationVW
148+
```
149+
150+
For convenience, creating a CachedResource triggers creation of a CachedResourceEndpointSlice object under the same name. This behaviour can be disabled by adding `cachedresources.cache.kcp.io/skip-endpointslice` annotation to the CachedResource object. You can create your own like so:
151+
152+
```yaml
153+
apiVersion: cache.kcp.io/v1alpha1
154+
kind: CachedResourceEndpointSlice
155+
metadata:
156+
name: cpuflavors-v1
157+
spec:
158+
cachedResource:
159+
name: cpuflavors-v1 # (1)
160+
```
161+
162+
1. Name of the CachedResource this endpoint slice is referencing. Currently, both CachedResource and CachedResourceEndpointSlice must be co-located in the same workspace. See <https://github.com/kcp-dev/kcp/issues/3658>
163+
164+
### Selectors
165+
166+
CachedResource spec has an optional [`labelSelector`](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors) field which can be used to shape the set of objects it picks up.
167+
168+
```yaml
169+
apiVersion: cache.kcp.io/v1alpha1
170+
kind: CachedResource
171+
metadata:
172+
name: cpuflavors-v1
173+
spec:
174+
group: cloud.example.com
175+
version: v1
176+
resource: cpuflavors
177+
labelSelector:
178+
cloud.example.com/visibility: Public
179+
```
180+
181+
## Exporting CachedResources
182+
183+
You can project the replicated read-only objects of a CachedResource into a workspace using the standard APIExport-APIBinding relationship. Create an APIExport and define [virtual resource](./exporting-apis.md#virtual-resources) for the associated [CachedResourceEndpointSlice](#cachedresourceendpointslice). Consumers can then bind to it.
184+
185+
```yaml
186+
apiVersion: apis.kcp.io/v1alpha2
187+
kind: APIExport
188+
metadata:
189+
name: compute.cloud.example.com
190+
spec:
191+
resources:
192+
- group: cloud.example.com
193+
name: cpuflavors
194+
schema: v250801.cpuflavors.cloud.example.com # (1)
195+
storage:
196+
virtual:
197+
reference: # (2)
198+
apiGroup: cache.kcp.io
199+
kind: CachedResourceEndpointSlice
200+
name: cpuflavors-v1
201+
identityHash: cd2eb0837... # (3)
202+
```
203+
204+
1. Resource schema must match the schema used by the resource in the associated CachedResource.
205+
2. Reference to the CachedResourceEndpointSlice endpoint slice `cpuflavors-v1`.
206+
3. Identity hash of the `cpuflavors-v1` CachedResource object.
207+
208+
A `virtual` storage definition needs (1) a reference to an [endpoint slice](./exporting-apis.md#endpoint-slices) object, and (2) a virtual resource identity. In the case of CachedResources, the endpoint slice is provided by CachedResourceEndpointSlice. For convenience, a matching CachedResourceEndpointSlice object is automatically created when creating a CachedResource. The identity hash must match the one set in CachedResource's `.status.identityHash`.
209+
210+
```mermaid
211+
flowchart TD
212+
subgraph provider["API Provider Workspace"]
213+
export["CPUFlavors APIExport"]
214+
schema["CPUFlavors APIResourceSchema"]
215+
crd["CPUFlavors CRD"]
216+
cpuflavorsCachedResourceEndpointSlice["CPUFlavors CachedResourceEndpointSlice"]
217+
218+
export --> schema
219+
export --> cpuflavorsCachedResourceEndpointSlice
220+
schema -."Is equivalent to".-> crd
221+
end
222+
223+
subgraph consumer1["Consumer Workspace"]
224+
binding["CPUFlavors APIBinding"]
225+
boundCPUFlavorVirtualResources["CPUFlavor objects..."]
226+
227+
binding -."Projects CPUFlavors API from Replication VW".-> boundCPUFlavorVirtualResources
228+
end
229+
230+
export --> binding
231+
```
232+
233+
Note the APIResourceSchema referenced by the APIExport example above: the Replication VW follows that reference, and that schema is then used to serve the resource in consumers' workspaces. The provider must therefore ensure that it is kept in-sync with the schema of the original resource.

‎docs/content/concepts/apis/exporting-apis.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -534,3 +534,59 @@ widgets example.kcp.io/v1alpha1 false Widget
534534
```
535535

536536
Furthermore, you can use the `.status.boundResources` field to precisely identify which `APIResourceSchemas` have been imported.
537+
538+
## Resource storage
539+
540+
When defining a resource in an APIExport, its storage is set to `crd` by default. This means the export's consumers will be able to interact with that resource as-if it had regular CRD semantics: creating, getting, listing, updating, patching and deleting all happen local to the APIBinding's workspace. Consequently, objects of this resource "live" in this workspace, and are backed by their own distinct objects in the etcd store.
541+
542+
```yaml
543+
apiVersion: apis.kcp.io/v1alpha2
544+
kind: APIExport
545+
metadata:
546+
name: example.kcp.io
547+
spec:
548+
resources:
549+
- group: example.kcp.io
550+
name: widgets
551+
schema: v220801.widgets.example.kcp.io
552+
# (1)
553+
# storage:
554+
# crd: {}
555+
```
556+
557+
1. The `storage` block with `crd` declaration below is implicit. No need to define it manually.
558+
559+
560+
### Virtual resources
561+
562+
!!! warning
563+
As of 0.29, this feature is of alpha-version quality. To use it, enable the `CachedAPIs` feature gate.
564+
565+
An APIExport resource may be defined with `virtual` storage. This means the resource is provided by a [virtual workspace](../workspaces/virtual-workspaces.md), and is projected into APIBinding's workspace.
566+
567+
```yaml
568+
apiVersion: apis.kcp.io/v1alpha2
569+
kind: APIExport
570+
metadata:
571+
name: compute.cloud.example.com
572+
spec:
573+
resources:
574+
- group: cloud.example.com
575+
name: cpuflavors
576+
schema: v250801.cpuflavors.cloud.example.com
577+
storage:
578+
virtual:
579+
reference: # (1)
580+
apiGroup: cache.kcp.io
581+
kind: CachedResourceEndpointSlice
582+
name: cpuflavors-v1
583+
identityHash: cd2eb0837... # (2)
584+
585+
```
586+
587+
1. The `reference` block defines a reference to an [endpoint slice](#endpoint-slices) object.
588+
2. The `identityHash` refers to the identity hash owned by the virtual resource. This is different from the APIExport identity hash.
589+
590+
kcp currently supports one such virtual resource: see [CachedResource API](./cached-resources.md) for more information.
591+
592+
Virtual resources are generic, and as long as the source virtual workspace implements the endpoint slice machinery, it can be used in APIExport's `virtual` storage definition.

0 commit comments

Comments
 (0)