|
| 1 | +--- |
| 2 | +type: "lab" |
| 3 | +description: "This lab explains how to add persistent storage to DigitalOcean Kubernetes (DOKS) clusters using PersistentVolumeClaims (PVCs) with kubectl. It covers creating a StatefulSet configuration to mount volumes, resizing volumes, setting permissions with initContainers, and troubleshooting common issues like PVC deletion stalls." |
| 4 | +title: "How to Add Volumes to Kubernetes Clusters" |
| 5 | +--- |
| 6 | +## Introduction |
| 7 | + |
| 8 | +DigitalOcean Kubernetes (DOKS) is a managed Kubernetes service offering a fully managed control plane, high availability, and autoscaling. DOKS integrates seamlessly with standard Kubernetes toolchains, DigitalOcean’s load balancers, volumes, CPU and GPU Droplets, API, and CLI. |
| 9 | + |
| 10 | +When you need to manage persistent data in a Kubernetes cluster, you can use DigitalOcean Volumes Block Storage by creating a PersistentVolumeClaim (PVC) as part of your deployment. This guide explains how to add volumes to your cluster using the Kubernetes command-line tool, `kubectl`. For more details on `kubectl`, see the [Overview of kubectl](https://kubernetes.io/docs/reference/kubectl/overview/). |
| 11 | + |
| 12 | +PVCs enable cluster workers to read and write persistent data, such as database records, user-generated website content, log files, and other data that should persist after a process completes. |
| 13 | + |
| 14 | +## Managing Persistent Volume Claims |
| 15 | + |
| 16 | +- **Deleting PVCs**: Deleting a deployment does not automatically delete associated PVCs. You must manually remove them using `kubectl delete pvc`. |
| 17 | +- **Volume Deletion Issues**: If a volume is deleted before the PVC API object is removed, the PVC may enter an inconsistent state, causing deletion attempts to stall or fail. Refer to the troubleshooting section for a fix. |
| 18 | +- **Existing PVC Conflicts**: If a PVC with the same name already exists, you’ll encounter an error like: |
| 19 | + |
| 20 | + ```bash |
| 21 | + Error from server (AlreadyExists): error when creating "pvc.yml": persistentvolumeclaims "csi-pvc" already exists |
| 22 | + ``` |
| 23 | + |
| 24 | +In this case, the existing volume will be mounted instead of creating a new one. |
| 25 | + |
| 26 | +- **Volume Creation**: Volumes created via the control panel or API cannot be used by Kubernetes clusters. You must create volumes within Kubernetes for PVCs to use them. |
| 27 | + |
| 28 | +## Create a Configuration File |
| 29 | + |
| 30 | +We recommend using pods that reference volumes owned by a `StatefulSet`. This section demonstrates how to create a `StatefulSet` to use a PVC as a volume for a pod. |
| 31 | + |
| 32 | +### Example StatefulSet Configuration |
| 33 | + |
| 34 | +```yaml |
| 35 | +apiVersion: apps/v1 |
| 36 | +kind: StatefulSet |
| 37 | +metadata: |
| 38 | + name: my-csi-app-set |
| 39 | +spec: |
| 40 | + selector: |
| 41 | + matchLabels: |
| 42 | + app: mypod |
| 43 | + serviceName: "my-frontend" |
| 44 | + replicas: 1 |
| 45 | + template: |
| 46 | + metadata: |
| 47 | + labels: |
| 48 | + app: mypod |
| 49 | + spec: |
| 50 | + containers: |
| 51 | + - name: my-frontend |
| 52 | + image: busybox |
| 53 | + args: |
| 54 | + - sleep |
| 55 | + - infinity |
| 56 | + volumeMounts: |
| 57 | + - mountPath: "/data" |
| 58 | + name: csi-pvc |
| 59 | + volumeClaimTemplates: |
| 60 | + - metadata: |
| 61 | + name: csi-pvc |
| 62 | + नेपाल: |
| 63 | + accessModes: |
| 64 | + - ReadWriteOnce |
| 65 | + resources: |
| 66 | + requests: |
| 67 | + storage: 5Gi |
| 68 | + storageClassName: do-block-storage |
| 69 | +``` |
| 70 | +
|
| 71 | +### Configuration Details |
| 72 | +
|
| 73 | +- **Pod Template**: Defines how the pod is created and specifies the container image. This example uses the Linux `BusyBox` image, mounts a volume named `csi-pvc`, and maps it to `/data` in the container’s filesystem. |
| 74 | +- **VolumeClaimTemplates**: Locates the volume by the name `csi-pvc`. If no volume exists with this name, one is created. If it exists, the existing volume is mounted. This example creates a 5 GB volume accessible to the cluster as `csi-pvc`. |
| 75 | + |
| 76 | +### Customizable Fields |
| 77 | + |
| 78 | +- **name**: Must be lowercase alphanumeric with dashes, unique within the cluster. |
| 79 | +- **accessModes**: Must be set to `ReadWriteOnce`. Other modes (`ReadOnlyMany`, `ReadWriteMany`) are not supported by DigitalOcean volumes. See [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes) for details. |
| 80 | +- **storage**: Specifies the volume size, ranging from 1 GB to 10,000 GB. |
| 81 | + |
| 82 | +### Applying the Configuration |
| 83 | + |
| 84 | +Use the following command to create the `StatefulSet` with the pod and mounted volume: |
| 85 | + |
| 86 | +```bash |
| 87 | +kubectl apply -f <config-file-name>.yml |
| 88 | +``` |
| 89 | + |
| 90 | +### Resizing Volumes |
| 91 | + |
| 92 | +If your DOKS version supports it, you can resize volumes by updating the PVC’s storage value: |
| 93 | + |
| 94 | +```bash |
| 95 | +kubectl edit pvc <your-pvc-name> |
| 96 | +``` |
| 97 | + |
| 98 | +Alternatively, use: |
| 99 | + |
| 100 | +```bash |
| 101 | +kubectl patch pvc <your-pvc-name> -p '{ "spec": { "resources": { "requests": { "storage": "<new-size>" }}}}' |
| 102 | +``` |
| 103 | + |
| 104 | +Resizing may take a few minutes or require restarting the application. Verify the new capacity in the volumes list or the cluster’s Kubernetes dashboard. |
| 105 | + |
| 106 | +> **Note**: Volumes can only be increased in size, not decreased. Billing begins when the volume is created and continues until the volume is explicitly deleted. Remove the PVC from the cluster before deleting the volume. |
| 107 | + |
| 108 | +## Show Volumes |
| 109 | + |
| 110 | +After applying the configuration, volumes appear in the **Resources** tab of your cluster in the DigitalOcean control panel. They are identified by the `name` parameter (e.g., `csi-pvc` in the example). |
| 111 | + |
| 112 | +In the DigitalOcean control panel, volume names start with `pvc-` followed by a unique identifier (e.g., `pvc-0213ed0abexample`). |
| 113 | + |
| 114 | +To list storage volumes associated with a cluster, use: |
| 115 | + |
| 116 | +```bash |
| 117 | +kubectl get pv |
| 118 | +``` |
| 119 | + |
| 120 | +Example output: |
| 121 | + |
| 122 | +``` |
| 123 | +NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE |
| 124 | +pvc-0213ed0abexample 5Gi RWO Delete Bound default/csi-pvc do-block-storage 11s |
| 125 | +``` |
| 126 | + |
| 127 | +## Setting Permissions on Volumes |
| 128 | + |
| 129 | +By default, the volume’s filesystem owner is `root:root`. If a pod runs as a non-root user and needs to create files or directories, it may fail due to insufficient permissions. DigitalOcean Kubernetes does not support `mountOptions` like `dir_mode=0777` or `file_mode=0777`. |
| 130 | + |
| 131 | +Instead, use an `initContainer` to change the permissions/ownership of the volume’s filesystem. |
| 132 | + |
| 133 | +### Example Pod Configuration with Permissions Fix |
| 134 | + |
| 135 | +```yaml |
| 136 | +apiVersion: v1 |
| 137 | +kind: Pod |
| 138 | +metadata: |
| 139 | + name: my-csi-app |
| 140 | +spec: |
| 141 | + containers: |
| 142 | + - name: my-db |
| 143 | + image: postgres:latest |
| 144 | + volumeMounts: |
| 145 | + - mountPath: "/var/lib/postgresql" |
| 146 | + name: my-do-volume |
| 147 | + initContainers: |
| 148 | + - name: pgsql-data-permission-fix |
| 149 | + image: busybox |
| 150 | + command: ["/bin/chmod", "-R", "777", "/data"] |
| 151 | + volumeMounts: |
| 152 | + - name: my-do-volume |
| 153 | + mountPath: /data |
| 154 | + volumes: |
| 155 | + - name: my-do-volume |
| 156 | + persistentVolumeClaim: |
| 157 | + claimName: csi-pvc |
| 158 | +``` |
| 159 | + |
| 160 | +This configuration: |
| 161 | + |
| 162 | +- Creates a pod (`my-csi-app`) using the `postgres:latest` image. |
| 163 | +- Names the `csi-pvc` volume as `my-do-volume` and mounts it at `/data`. |
| 164 | +- Uses an `initContainer` to temporarily mount the volume and set permissions to `777` for the `/data` path. The `initContainer` deletes itself after execution. |
| 165 | +- Optionally, use `securityContext` with `chown $userid` instead of `chmod 777` for specific user ownership: |
| 166 | + |
| 167 | +```yaml |
| 168 | +securityContext: |
| 169 | + runAsUser: 1000 |
| 170 | + fsGroup: 2000 |
| 171 | +``` |
| 172 | + |
| 173 | +### Check Volume Permissions |
| 174 | + |
| 175 | +Verify permissions by checking the pod logs: |
| 176 | + |
| 177 | +```bash |
| 178 | +kubectl logs my-csi-app |
| 179 | +``` |
| 180 | + |
| 181 | +Example output: |
| 182 | + |
| 183 | +``` |
| 184 | +The files belonging to this database system will be owned by user "postgres". |
| 185 | +This user must also own the server process. |
| 186 | +
|
| 187 | +The database cluster will be initialized with locale "en_US.utf8". |
| 188 | +The default database encoding has accordingly been set to "UTF8". |
| 189 | +The default text search configuration will be set to "english". |
| 190 | +
|
| 191 | +Data page checksums are disabled. |
| 192 | +
|
| 193 | +fixing permissions on existing directory /var/lib/postgresql/data ... ok |
| 194 | +creating subdirectories ... ok |
| 195 | +selecting default max_connections ... 100 |
| 196 | +selecting default shared_buffers ... 128MB |
| 197 | +selecting dynamic shared memory implementation ... posix |
| 198 | +creating configuration files ... ok |
| 199 | +running bootstrap script ... ok |
| 200 | +performing post-bootstrap initialization ... ok |
| 201 | +syncing data to disk ... ok |
| 202 | +``` |
| 203 | + |
| 204 | +## Troubleshooting |
| 205 | + |
| 206 | +> **Warning**: Cluster resources (worker nodes, load balancers, volumes) are listed in the DigitalOcean Control Panel outside the Kubernetes page. Modifying these resources directly in the control panel may render them unusable or trigger the reconciler to provision replacements. Manage resources exclusively with `kubectl` or the control panel’s Kubernetes page. |
| 207 | + |
| 208 | +If a volume is deleted before the PVC API object, the PVC deletion may hang. To resolve this: |
| 209 | + |
| 210 | +1. List volume attachments: |
| 211 | + |
| 212 | +```bash |
| 213 | +kubectl get volumeattachments |
| 214 | +``` |
| 215 | + |
| 216 | +Example output: |
| 217 | + |
| 218 | +```bash |
| 219 | +NAME CREATED AT |
| 220 | +$VOLUME_NAME 2019-03-08T21:58:24Z |
| 221 | +``` |
| 222 | + |
| 223 | +2. Gather information about the volume: |
| 224 | + |
| 225 | +```bash |
| 226 | +kubectl describe volumeattachments $VOLUME_NAME |
| 227 | +``` |
| 228 | + |
| 229 | +```bash |
| 230 | +kubectl describe volumeattachments $VOLUME_NAME |
| 231 | +``` |
| 232 | + |
| 233 | +3. Edit the volume attachment to remove the finalizer: |
| 234 | + |
| 235 | +```bash |
| 236 | +
|
| 237 | +```bash |
| 238 | +kubectl edit volumeattachment $VOLUME_NAME |
| 239 | +``` |
| 240 | + |
| 241 | +Remove the following from the `metadata` section: |
| 242 | + |
| 243 | +```yaml |
| 244 | +finalizers: |
| 245 | + - external-attacher/dobs-csi-digitalocean-com |
| 246 | +``` |
| 247 | +
|
| 248 | +4. Attempt to delete the PVC: |
| 249 | +
|
| 250 | +```bash |
| 251 | +kubectl delete pvc csi-pvc |
| 252 | +``` |
0 commit comments