Skip to content

Commit bbe7bac

Browse files
AnnaMHuffAnnaMHuff
authored
Merge pull request #292632 from pjw711/pjw/break-out-overview
Align storage concepts with functional offerings
2 parents c8f7bb2 + 94b331a commit bbe7bac

File tree

3 files changed

+321
-286
lines changed

3 files changed

+321
-286
lines changed

‎articles/operator-nexus/TOC.yml

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,13 @@
1111
href: concepts-resource-types.md
1212
- name: Compute overview
1313
href: concepts-compute.md
14-
- name: Storage overview
15-
href: concepts-storage.md
14+
- name: Storage
15+
expanded: false
16+
items:
17+
- name: Storage overview
18+
href: concepts-storage.md
19+
- name: Storage for Nexus Kubernetes
20+
href: concepts-storage-kubernetes.md
1621
- name: Cluster deployment and upgrades
1722
expanded: false
1823
items:
@@ -304,7 +309,7 @@
304309
href: howto-check-runtime-version.md
305310
- name: Site decommission checklist
306311
href: howto-decommission-nexus-instance-checklist.md
307-
312+
308313
- name: Troubleshooting
309314
expanded: true
310315
items:

‎articles/operator-nexus/concepts-storage-kubernetes.md

Lines changed: 313 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,313 @@
1+
---
2+
title: Azure Operator Nexus storage for Kubernetes
3+
description: Get an overview of available storage classes for Kubernetes on Azure Operator Nexus.
4+
author: pjw711
5+
ms.author: peterwhiting
6+
ms.service: azure-operator-nexus
7+
ms.topic: conceptual
8+
ms.date: 01/06/2025
9+
ms.custom: template-concept
10+
---
11+
12+
# Azure Operator Nexus storage for Kubernetes
13+
14+
Each Azure Operator Nexus cluster uses a single storage appliance to provide persistent storage to Nexus Kubernetes cluster tenant workloads. The Azure Operator Nexus software Kubernetes stack offers two types of persistent storage. Operators select them through the Kubernetes StorageClass mechanism.
15+
16+
> [!IMPORTANT]
17+
> Azure Operator Nexus does not support ephemeral volumes. Nexus recommends that the persistent volume storage mechanisms described in this document are used for all Nexus Kubernetes cluster workload volumes as these provide the highest levels of performance and availability. All storage in Azure Operator Nexus is provided by the storage appliance. There is no support for storage provided by baremetal machine disks.
18+
19+
## Kubernetes storage classes
20+
21+
### StorageClass: nexus-volume
22+
23+
The default storage mechanism, *nexus-volume*, is the preferred choice for most users. It provides the highest levels of performance and availability. However, volumes can't be simultaneously shared across multiple worker nodes. Operators can access and manage these volumes by using the Azure API and portal, through the volume resource.
24+
25+
```yaml
26+
apiVersion: v1
27+
kind: PersistentVolumeClaim
28+
metadata:
29+
name: testPvc
30+
namespace: default
31+
spec:
32+
accessModes:
33+
- ReadWriteOnce
34+
resources:
35+
requests:
36+
storage: 107Mi
37+
storageClassName: nexus-volume
38+
volumeMode: Block
39+
volumeName: testVolume
40+
status:
41+
accessModes:
42+
- ReadWriteOnce
43+
capacity:
44+
storage: 107Mi
45+
phase: Bound
46+
```
47+
48+
### StorageClass: nexus-shared
49+
50+
In situations where a shared file system is required, the *nexus-shared* storage class is available. This storage class provides a highly available shared storage solution by enabling multiple pods in the same Nexus Kubernetes cluster to concurrently access and share the same volume. The *nexus-shared* storage class is backed by a highly available NFS storage service. This NFS storage service (storage pool currently limited to a maximum size of 1 TiB) is available per Cloud Service Network (CSN). The NFS storage service is deployed automatically on creation of a CSN resource. Any Nexus Kubernetes cluster attached to the CSN can provision persistent volumes from this shared storage pool. Nexus-shared supports both Read Write Once (RWO) and Read Write Many (RWX) access modes. What that means is that the workload applications can make use of either of these access modes to access the shared storage.
51+
52+
<!--- IMG ![Nexus Shared Volume](Docs/media/nexus-shared-volume.png) IMG --->
53+
:::image type="content" source="media/nexus-shared-volume.png" alt-text="Diagram depicting how nexus-shared provisions a volume for a workload in Nexus Kubernetes Cluster.":::
54+
55+
Figure: Nexus Shared Volume
56+
57+
Although the performance and availability of *nexus-shared* are sufficient for most applications, we recommend that workloads with heavy I/O requirements use the *nexus-volume* option for optimal performance.
58+
59+
#### Read Write Once (RWO)
60+
61+
In Read Write Once (RWO) mode, only one node or claimant can mount the nexus-shared volume at a time. ReadWriteOnce access mode still allows multiple pods to access the volume when the pods are running on the same node.
62+
63+
```yaml
64+
apiVersion: v1
65+
items:
66+
- apiVersion: v1
67+
kind: PersistentVolumeClaim
68+
metadata:
69+
name: test-pvc
70+
namespace: default
71+
spec:
72+
accessModes:
73+
- ReadWriteOnce
74+
resources:
75+
requests:
76+
storage: 5Gi
77+
storageClassName: nexus-shared
78+
volumeMode: Filesystem
79+
volumeName: TestVolume
80+
status:
81+
accessModes:
82+
- ReadWriteOnce
83+
capacity:
84+
storage: 5Gi
85+
phase: Bound
86+
```
87+
88+
#### Read Write Many (RWX)
89+
90+
In the Read Write Many (RWX) mode, multiple nodes or claimants can mount the nexus-shared volume at the same time.
91+
92+
```yaml
93+
apiVersion: v1
94+
items:
95+
- apiVersion: v1
96+
kind: PersistentVolumeClaim
97+
metadata:
98+
name: test-pvc
99+
namespace: default
100+
spec:
101+
accessModes:
102+
- ReadWriteMany
103+
resources:
104+
requests:
105+
storage: 5Gi
106+
storageClassName: nexus-shared
107+
volumeMode: Filesystem
108+
volumeName: TestVolume
109+
status:
110+
accessModes:
111+
- ReadWriteMany
112+
capacity:
113+
storage: 5Gi
114+
phase: Bound
115+
```
116+
117+
### Examples
118+
119+
#### Read Write Once (RWO) with nexus-volume storage class
120+
121+
This example manifest creates a StatefulSet with PersistentVolumeClaimTemplate using nexus-volume storage class in ReadWriteOnce mode.
122+
123+
```yaml
124+
apiVersion: apps/v1
125+
kind: StatefulSet
126+
metadata:
127+
name: test-sts-rwo
128+
labels:
129+
app: test-sts-rwo
130+
spec:
131+
serviceName: test-sts-rwo-svc
132+
replicas: 3
133+
selector:
134+
matchLabels:
135+
app: test-sts-rwo
136+
template:
137+
metadata:
138+
labels:
139+
app: test-sts-rwo
140+
spec:
141+
containers:
142+
- name: busybox
143+
command:
144+
- "/bin/sh"
145+
- "-c"
146+
- while true; do echo "$(date) -- $(hostname)" >> /mnt/hostname.txt; sleep 1; done
147+
image: busybox
148+
volumeMounts:
149+
- name: test-volume-rwo
150+
mountPath: /mnt/
151+
volumeClaimTemplates:
152+
- metadata:
153+
name: test-volume-rwo
154+
spec:
155+
accessModes: ["ReadWriteOnce"]
156+
resources:
157+
requests:
158+
storage: 10Gi
159+
storageClassName: nexus-volume
160+
```
161+
162+
Each pod of the StatefulSet has one PersistentVolumeClaim created.
163+
164+
```bash
165+
# kubectl get pvc
166+
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
167+
test-volume-rwo-test-sts-rwo-0 Bound pvc-e41fec47-cc43-4cd5-8547-5a4457cbdced 10Gi RWO nexus-volume 8m17s
168+
test-volume-rwo-test-sts-rwo-1 Bound pvc-1589dc79-59d2-4a1d-8043-b6a883b7881d 10Gi RWO nexus-volume 7m58s
169+
test-volume-rwo-test-sts-rwo-2 Bound pvc-82e3beac-fe67-4676-9c61-e982022d443f 10Gi RWO nexus-volume 12s
170+
```
171+
172+
```bash
173+
# kubectl get pods -o wide -w
174+
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
175+
test-sts-rwo-0 1/1 Running 0 8m31s 10.245.231.74 nexus-cluster-6a8c4018-agentpool2-md-vhhv6 <none> <none>
176+
test-sts-rwo-1 1/1 Running 0 8m12s 10.245.126.73 nexus-cluster-6a8c4018-agentpool1-md-27nw4 <none> <none>
177+
test-sts-rwo-2 1/1 Running 0 26s 10.245.183.9 nexus-cluster-6a8c4018-agentpool1-md-4jprt <none> <none>
178+
```
179+
180+
```bash
181+
# kubectl exec test-sts-rwo-0 -- cat /mnt/hostname.txt
182+
Thu Nov 9 21:57:25 UTC 2023 -- test-sts-rwo-0
183+
Thu Nov 9 21:57:26 UTC 2023 -- test-sts-rwo-0
184+
Thu Nov 9 21:57:27 UTC 2023 -- test-sts-rwo-0
185+
186+
# kubectl exec test-sts-rwo-1 -- cat /mnt/hostname.txt
187+
Thu Nov 9 21:57:19 UTC 2023 -- test-sts-rwo-1
188+
Thu Nov 9 21:57:20 UTC 2023 -- test-sts-rwo-1
189+
Thu Nov 9 21:57:21 UTC 2023 -- test-sts-rwo-1
190+
191+
# kubectl exec test-sts-rwo-s -- cat /mnt/hostname.txt
192+
Thu Nov 9 21:58:32 UTC 2023 -- test-sts-rwo-2
193+
Thu Nov 9 21:58:33 UTC 2023 -- test-sts-rwo-2
194+
Thu Nov 9 21:58:34 UTC 2023 -- test-sts-rwo-2
195+
```
196+
197+
#### Read Write Many (RWX) with nexus-shared storage class
198+
199+
The below manifest creates a Deployment with a PersistentVolumeClaim (PVC) using nexus-shared storage class in ReadWriteMany mode. The PVC created is shared by all the pods of the deployment and can be used to read and write by all of them simultaneously.
200+
201+
```yaml
202+
---
203+
apiVersion: v1
204+
kind: PersistentVolumeClaim
205+
metadata:
206+
name: test-volume-rwx
207+
spec:
208+
accessModes:
209+
- ReadWriteMany
210+
volumeMode: Filesystem
211+
resources:
212+
requests:
213+
storage: 3Gi
214+
storageClassName: nexus-shared
215+
---
216+
apiVersion: apps/v1
217+
kind: Deployment
218+
metadata:
219+
labels:
220+
app: test-deploy-rwx
221+
name: test-deploy-rwx
222+
spec:
223+
replicas: 3
224+
selector:
225+
matchLabels:
226+
app: test-deploy-rwx
227+
template:
228+
metadata:
229+
labels:
230+
app: test-deploy-rwx
231+
spec:
232+
affinity:
233+
podAntiAffinity:
234+
requiredDuringSchedulingIgnoredDuringExecution:
235+
- labelSelector:
236+
matchExpressions:
237+
- key: kubernetes.azure.com/agentpool
238+
operator: Exists
239+
values: []
240+
topologyKey: "kubernetes.io/hostname"
241+
containers:
242+
- name: busybox
243+
command:
244+
- "/bin/sh"
245+
- "-c"
246+
- while true; do echo "$(date) -- $(hostname)" >> /mnt/hostname.txt; sleep 1; done
247+
image: busybox
248+
volumeMounts:
249+
- name: test-volume-rwx
250+
mountPath: /mnt/
251+
volumes:
252+
- name: test-volume-rwx
253+
persistentVolumeClaim:
254+
claimName: test-volume-rwx
255+
...
256+
```
257+
258+
Once applied, there are three replicas of the deployment sharing the same PVC.
259+
260+
```bash
261+
# kubectl get pvc
262+
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
263+
test-volume-rwx Bound pvc-32f0717e-6b63-4d64-a458-5be4ffe21d37 3Gi RWX nexus-shared 6s
264+
```
265+
266+
```bash
267+
# kubectl get pods -o wide -w
268+
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
269+
test-deploy-rwx-fdb8f49c-86pv4 1/1 Running 0 18s 10.245.224.140 nexus-cluster-6a8c4018-agentpool1-md-s2dh7 <none> <none>
270+
test-deploy-rwx-fdb8f49c-9zsjf 1/1 Running 0 18s 10.245.126.74 nexus-cluster-6a8c4018-agentpool1-md-27nw4 <none> <none>
271+
test-deploy-rwx-fdb8f49c-wdgw7 1/1 Running 0 18s 10.245.231.75 nexus-cluster-6a8c4018-agentpool2-md-vhhv6 <none> <none>
272+
```
273+
274+
It can be observed from the below output that all pods are writing into the same PVC.
275+
276+
```bash
277+
# kubectl exec test-deploy-rwx-fdb8f49c-86pv4 -- cat /mnt/hostname.txt
278+
Thu Nov 9 21:51:41 UTC 2023 -- test-deploy-rwx-fdb8f49c-86pv4
279+
Thu Nov 9 21:51:41 UTC 2023 -- test-deploy-rwx-fdb8f49c-9zsjf
280+
Thu Nov 9 21:51:41 UTC 2023 -- test-deploy-rwx-fdb8f49c-wdgw7
281+
Thu Nov 9 21:51:42 UTC 2023 -- test-deploy-rwx-fdb8f49c-86pv4
282+
283+
# kubectl exec test-deploy-rwx-fdb8f49c-9zsjf -- cat /mnt/hostname.txt
284+
Thu Nov 9 21:51:41 UTC 2023 -- test-deploy-rwx-fdb8f49c-86pv4
285+
Thu Nov 9 21:51:41 UTC 2023 -- test-deploy-rwx-fdb8f49c-9zsjf
286+
Thu Nov 9 21:51:41 UTC 2023 -- test-deploy-rwx-fdb8f49c-wdgw7
287+
Thu Nov 9 21:51:42 UTC 2023 -- test-deploy-rwx-fdb8f49c-86pv4
288+
289+
# kubectl exec test-deploy-rwx-fdb8f49c-wdgw7 -- cat /mnt/hostname.txt
290+
Thu Nov 9 21:51:41 UTC 2023 -- test-deploy-rwx-fdb8f49c-86pv4
291+
Thu Nov 9 21:51:41 UTC 2023 -- test-deploy-rwx-fdb8f49c-9zsjf
292+
Thu Nov 9 21:51:41 UTC 2023 -- test-deploy-rwx-fdb8f49c-wdgw7
293+
Thu Nov 9 21:51:42 UTC 2023 -- test-deploy-rwx-fdb8f49c-86pv4
294+
```
295+
296+
## Volume size limits and capacity management
297+
298+
PVCs created using the nexus-volume and nexus-shared have minimum and maximum claim sizes.
299+
300+
| Storage Class | Minimum PVC Size | Maximum PVC Size |
301+
|---------------|------------------|------------------|
302+
| nexus-volume | 1 MiB | 12 TiB |
303+
| nexus-shared | None | 1 TiB |
304+
305+
> [!IMPORTANT]
306+
> Volumes that reach their consumption limit will cause out of disk space errors on the workloads that consume them. You must make sure that you provision suitable volume sizes for your workload requirements. You must monitor both the storage appliance and all NFS servers for their percentage storage consumption. You can do this using the metrics documented in the [list of available metrics](./list-of-metrics-collected.md).
307+
308+
- Both nexus-volume and nexus-shared PVCs have their requested storage capacity enforced as a consumption limit. A volume can't consume more storage than the associated PVC request.
309+
- All physical volumes are thin-provisioned. You must monitor the total storage consumption on your storage appliance and perform maintenance operations to free up storage space if necessary.
310+
- A nexus-volume PVC provisioning request fails if the requested size is less than the minimum or more than the maximum supported volume size.
311+
- Nexus-shared volumes are logically thin-provisioned on the backing NFS server. This NFS server has a fixed capacity of 1 TiB.
312+
- A nexus-shared PVC can be provisioned despite requesting more than 1 TiB of storage, however, only 1 TiB can be consumed.
313+
- It is possible to provision a set of PVCs where the sum of capacity requests is greater than 1 TiB. However, the consumption limit of 1 TiB applies; the set of associated PVs may not consume more than 1 TiB of storage.

0 commit comments

Comments
 (0)