Skip to content

Commit c5579c6

Browse files
elijah-rouelijah-rou
committed
feat: Add DaemonSet mount mode for shared StorageClass mount pods
Introduces a new DaemonSet deployment mode for JuiceFS mount pods when using StorageClass with mount sharing enabled. This feature provides better resource management by deploying mount pods as DaemonSets instead of individual shared pods, with configurable node affinity control through ConfigMaps. - **Three mount modes**: per-pvc (default), shared-pod, and daemonset - **ConfigMap-based configuration**: Control mount mode and node affinity via `juicefs-mount-config` ConfigMap - **Node affinity support**: Restrict DaemonSet deployment to specific nodes using standard Kubernetes node affinity - **Automatic fallback**: Falls back to shared-pod mode if DaemonSet cannot schedule on a node - **Seamless transition**: Works with existing StorageClasses without modification - `pkg/juicefs/mount/mount_selector.go`: Dynamic mount type selection based on configuration - `pkg/juicefs/mount/daemonset_mount.go`: DaemonSet mount implementation with scheduling error handling - `pkg/config/mount_config.go`: ConfigMap parsing for mount mode and node affinity - `pkg/config/mount_config_helper.go`: Helper functions for DaemonSet configuration - `pkg/juicefs/mount/builder/daemonset.go`: DaemonSet resource builder Mount modes are configured via ConfigMap in kube-system namespace: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: juicefs-mount-config namespace: kube-system data: default: | mode: shared-pod # Options: per-pvc, shared-pod, daemonset my-storageclass: | mode: daemonset nodeAffinity: # Required for daemonset mode requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: node.kubernetes.io/workload operator: In values: ["compute"] ``` - **pod_driver.go**: Skip DaemonSet pods in deletion/recreation handlers - **juicefs.go**: Add MountSelector for dynamic mount type selection and AuthFs serialization - **k8sclient**: Add DaemonSet CRUD operations - **RBAC**: Add DaemonSet permissions to CSI node service account - `docs/en/guide/daemonset-mount.md`: Comprehensive usage documentation - `docs/en/guide/mount-pod-configuration.md`: Mount pod configuration guide - `deploy/kubernetes/csi-daemonset-mount/`: Example DaemonSet configurations - `deploy/kubernetes/mount-config/`: Example ConfigMap configurations - Unit tests for mount selector logic - Unit tests for DaemonSet mount implementation - ConfigMap parsing and validation tests 1. **Better resource utilization**: One mount pod per node instead of per PVC 2. **Improved control**: Node affinity ensures mount pods only run where needed 3. **Simplified operations**: Easier to manage and monitor as DaemonSets 4. **Automatic lifecycle**: DaemonSet controller handles pod creation/deletion 5. **Backward compatible**: Works with existing StorageClasses and mount sharing 1. Enable `STORAGE_CLASS_SHARE_MOUNT` in CSI Controller and Node 2. Grant DaemonSet RBAC permissions (included in updated manifests) 3. Create ConfigMap with desired mount mode configuration 4. New PVCs will automatically use configured mount mode - Updated `deploy/k8s.yaml` and `deploy/k8s_before_v1_18.yaml` with DaemonSet RBAC - Added DaemonSet resource configurations - Updated Docker build process for consistency This feature enhances the JuiceFS CSI Driver's flexibility in managing mount pods, particularly beneficial for large-scale deployments where mount pod proliferation can become a resource management challenge.
1 parent 81d23bf commit c5579c6

28 files changed

+4507
-80
lines changed

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,4 @@ cov2.out
2020
yalc.lock
2121
.ropeproject
2222
dist/
23+
*.test

‎.markdownlint-cli2.jsonc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@
4747
},
4848
"link-fragments": false,
4949
"no-trailing-slash-in-links": true,
50+
"MD029": false,
5051
"enhanced-proper-names": {
5152
"code_blocks": false,
5253
"html_elements": false,

‎deploy/k8s.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -255,7 +255,19 @@ rules:
255255
resources:
256256
- nodes
257257
verbs:
258+
- get
259+
- list
260+
- apiGroups:
261+
- apps
262+
resources:
263+
- daemonsets
264+
verbs:
265+
- get
258266
- list
267+
- watch
268+
- create
269+
- update
270+
- patch
259271
---
260272
apiVersion: rbac.authorization.k8s.io/v1
261273
kind: ClusterRole

‎deploy/k8s_before_v1_18.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -255,7 +255,19 @@ rules:
255255
resources:
256256
- nodes
257257
verbs:
258+
- get
259+
- list
260+
- apiGroups:
261+
- apps
262+
resources:
263+
- daemonsets
264+
verbs:
265+
- get
258266
- list
267+
- watch
268+
- create
269+
- update
270+
- patch
259271
---
260272
apiVersion: rbac.authorization.k8s.io/v1
261273
kind: ClusterRole

‎deploy/kubernetes/base/resources.yaml‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -555,7 +555,27 @@ spec:
555555
dnsPolicy: ClusterFirstWithHostNet
556556
priorityClassName: system-node-critical
557557
tolerations:
558-
- key: CriticalAddonsOnly
558+
- operator: Exists
559+
- effect: NoExecute
560+
key: node.kubernetes.io/not-ready
561+
operator: Exists
562+
- effect: NoExecute
563+
key: node.kubernetes.io/unreachable
564+
operator: Exists
565+
- effect: NoSchedule
566+
key: node.kubernetes.io/disk-pressure
567+
operator: Exists
568+
- effect: NoSchedule
569+
key: node.kubernetes.io/memory-pressure
570+
operator: Exists
571+
- effect: NoSchedule
572+
key: node.kubernetes.io/pid-pressure
573+
operator: Exists
574+
- effect: NoSchedule
575+
key: node.kubernetes.io/unschedulable
576+
operator: Exists
577+
- effect: NoSchedule
578+
key: node.kubernetes.io/network-unavailable
559579
operator: Exists
560580
containers:
561581
- name: juicefs-plugin

‎deploy/kubernetes/csi-daemonset-mount/daemonset.yaml‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
kind: DaemonSet
2+
apiVersion: apps/v1
3+
metadata:
4+
name: juicefs-csi-node
5+
namespace: kube-system
6+
spec:
7+
selector:
8+
matchLabels:
9+
app: juicefs-csi-node
10+
template:
11+
metadata:
12+
labels:
13+
app: juicefs-csi-node
14+
spec:
15+
tolerations:
16+
- operator: Exists
17+
- effect: NoExecute
18+
key: node.kubernetes.io/not-ready
19+
operator: Exists
20+
- effect: NoExecute
21+
key: node.kubernetes.io/unreachable
22+
operator: Exists
23+
- effect: NoSchedule
24+
key: node.kubernetes.io/disk-pressure
25+
operator: Exists
26+
- effect: NoSchedule
27+
key: node.kubernetes.io/memory-pressure
28+
operator: Exists
29+
- effect: NoSchedule
30+
key: node.kubernetes.io/pid-pressure
31+
operator: Exists
32+
- effect: NoSchedule
33+
key: node.kubernetes.io/unschedulable
34+
operator: Exists
35+
- effect: NoSchedule
36+
key: node.kubernetes.io/network-unavailable
37+
operator: Exists
38+
containers:
39+
- name: juicefs-plugin
40+
image: juicedata/juicefs-csi-driver:latest
41+
args:
42+
- --endpoint=$(CSI_ENDPOINT)
43+
- --logtostderr
44+
- --nodeid=$(NODE_NAME)
45+
- --v=1
46+
- --enable-manager=true
47+
env:
48+
- name: CSI_ENDPOINT
49+
value: unix:/csi/csi.sock
50+
- name: NODE_NAME
51+
valueFrom:
52+
fieldRef:
53+
fieldPath: spec.nodeName
54+
- name: JUICEFS_MOUNT_NAMESPACE
55+
valueFrom:
56+
fieldRef:
57+
fieldPath: metadata.namespace
58+
- name: POD_NAME
59+
valueFrom:
60+
fieldRef:
61+
fieldPath: metadata.name
62+
- name: JUICEFS_MOUNT_PATH
63+
value: /var/lib/juicefs/volume
64+
- name: JUICEFS_CONFIG_PATH
65+
value: /var/lib/juicefs/config
66+
# Enable StorageClass mount sharing
67+
- name: STORAGE_CLASS_SHARE_MOUNT
68+
value: "true"
69+
# Enable DaemonSet deployment for shared mounts
70+
- name: STORAGE_CLASS_DAEMONSET
71+
value: "true"
72+
volumeMounts:
73+
- name: kubelet-dir
74+
mountPath: /var/lib/kubelet
75+
mountPropagation: Bidirectional
76+
- name: plugin-dir
77+
mountPath: /csi
78+
- name: device-dir
79+
mountPath: /dev
80+
mountPropagation: HostToContainer
81+
volumes:
82+
- name: kubelet-dir
83+
hostPath:
84+
path: /var/lib/kubelet
85+
type: Directory
86+
- name: plugin-dir
87+
hostPath:
88+
path: /var/lib/kubelet/plugins/csi.juicefs.com
89+
type: DirectoryOrCreate
90+
- name: device-dir
91+
hostPath:
92+
path: /dev
93+
type: Directory

‎deploy/kubernetes/csi-daemonset-mount/mount-config.yaml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
apiVersion: v1
2+
kind: ConfigMap
3+
metadata:
4+
name: juicefs-mount-config
5+
namespace: juicefs
6+
data:
7+
# Default configuration for all StorageClasses
8+
default: |
9+
nodeAffinity:
10+
requiredDuringSchedulingIgnoredDuringExecution:
11+
nodeSelectorTerms:
12+
- matchExpressions:
13+
- key: node-role.kubernetes.io/control-plane
14+
operator: DoesNotExist
15+
16+
# StorageClass "juicefs-sc" configuration
17+
juicefs-sc: |
18+
nodeAffinity:
19+
requiredDuringSchedulingIgnoredDuringExecution:
20+
nodeSelectorTerms:
21+
- matchExpressions:
22+
- key: juicefs/mount-node
23+
operator: In
24+
values:
25+
- "true"

‎deploy/kubernetes/csi-daemonset-mount/storageclass-example.yaml‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
apiVersion: storage.k8s.io/v1
2+
kind: StorageClass
3+
metadata:
4+
name: juicefs-sc-daemonset
5+
provisioner: csi.juicefs.com
6+
volumeBindingMode: Immediate
7+
parameters:
8+
csi.storage.k8s.io/provisioner-secret-name: juicefs-secret
9+
csi.storage.k8s.io/provisioner-secret-namespace: kube-system
10+
csi.storage.k8s.io/node-publish-secret-name: juicefs-secret
11+
csi.storage.k8s.io/node-publish-secret-namespace: kube-system
12+
13+
# JuiceFS specific parameters
14+
pathPattern: "${pvc.name}"
15+
16+
# Node affinity configuration for DaemonSet mount pods
17+
# This example will deploy mount pods only on nodes with specific labels
18+
nodeAffinity: |
19+
requiredDuringSchedulingIgnoredDuringExecution:
20+
nodeSelectorTerms:
21+
- matchExpressions:
22+
- key: juicefs/mount-node
23+
operator: In
24+
values:
25+
- "true"
26+
- key: node-role.kubernetes.io/worker
27+
operator: Exists
28+
preferredDuringSchedulingIgnoredDuringExecution:
29+
- weight: 100
30+
preference:
31+
matchExpressions:
32+
- key: juicefs/high-performance
33+
operator: In
34+
values:
35+
- "true"
36+
---
37+
apiVersion: v1
38+
kind: Secret
39+
metadata:
40+
name: juicefs-secret
41+
namespace: kube-system
42+
type: Opaque
43+
stringData:
44+
name: "test-volume"
45+
metaurl: "redis://redis-service:6379/1"
46+
storage: "s3"
47+
bucket: "https://mybucket.s3.us-west-2.amazonaws.com"
48+
access-key: "ACCESS_KEY"
49+
secret-key: "SECRET_KEY"

‎deploy/kubernetes/mount-config/configmap-examples.yaml‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
# Example 1: Default configuration with mixed modes
2+
apiVersion: v1
3+
kind: ConfigMap
4+
metadata:
5+
name: juicefs-mount-config
6+
namespace: kube-system
7+
data:
8+
# Default configuration for all StorageClasses
9+
default: |
10+
mode: shared-pod # Use shared mount pods by default
11+
12+
# High-performance StorageClass uses DaemonSet on specific nodes
13+
high-performance-sc: |
14+
mode: daemonset
15+
nodeAffinity:
16+
requiredDuringSchedulingIgnoredDuringExecution:
17+
nodeSelectorTerms:
18+
- matchExpressions:
19+
- key: node.kubernetes.io/instance-type
20+
operator: In
21+
values:
22+
- m5.xlarge
23+
- m5.2xlarge
24+
- m5.4xlarge
25+
26+
# Development StorageClass uses per-PVC pods for isolation
27+
development-sc: |
28+
mode: per-pvc
29+
30+
# Production StorageClass uses DaemonSet with node affinity
31+
production-sc: |
32+
mode: daemonset
33+
nodeAffinity:
34+
requiredDuringSchedulingIgnoredDuringExecution:
35+
nodeSelectorTerms:
36+
- matchExpressions:
37+
- key: juicefs/mount-node
38+
operator: In
39+
values:
40+
- "true"
41+
- key: node-role.kubernetes.io/control-plane
42+
operator: DoesNotExist
43+
preferredDuringSchedulingIgnoredDuringExecution:
44+
- weight: 100
45+
preference:
46+
matchExpressions:
47+
- key: juicefs/high-performance
48+
operator: In
49+
values:
50+
- "true"
51+
52+
---
53+
# Example 2: Simple configuration - all StorageClasses use shared pods
54+
apiVersion: v1
55+
kind: ConfigMap
56+
metadata:
57+
name: juicefs-mount-config
58+
namespace: kube-system
59+
data:
60+
default: |
61+
mode: shared-pod
62+
63+
---
64+
# Example 3: DaemonSet-only configuration with different node affinities
65+
apiVersion: v1
66+
kind: ConfigMap
67+
metadata:
68+
name: juicefs-mount-config
69+
namespace: kube-system
70+
data:
71+
# Default: DaemonSet on all worker nodes
72+
default: |
73+
mode: daemonset
74+
nodeAffinity:
75+
requiredDuringSchedulingIgnoredDuringExecution:
76+
nodeSelectorTerms:
77+
- matchExpressions:
78+
- key: node-role.kubernetes.io/worker
79+
operator: Exists
80+
81+
# GPU workloads: DaemonSet only on GPU nodes
82+
gpu-storage: |
83+
mode: daemonset
84+
nodeAffinity:
85+
requiredDuringSchedulingIgnoredDuringExecution:
86+
nodeSelectorTerms:
87+
- matchExpressions:
88+
- key: nvidia.com/gpu
89+
operator: Exists
90+
91+
# Database storage: DaemonSet on dedicated database nodes
92+
database-storage: |
93+
mode: daemonset
94+
nodeAffinity:
95+
requiredDuringSchedulingIgnoredDuringExecution:
96+
nodeSelectorTerms:
97+
- matchExpressions:
98+
- key: workload-type
99+
operator: In
100+
values:
101+
- database
102+
103+
---
104+
# Example 4: Migration scenario - gradually move from per-PVC to shared/DaemonSet
105+
apiVersion: v1
106+
kind: ConfigMap
107+
metadata:
108+
name: juicefs-mount-config
109+
namespace: kube-system
110+
data:
111+
# Default: Keep existing per-PVC behavior
112+
default: |
113+
mode: per-pvc
114+
115+
# New StorageClasses use shared pods
116+
juicefs-sc-v2: |
117+
mode: shared-pod
118+
119+
# High-traffic StorageClass uses DaemonSet
120+
juicefs-sc-high-traffic: |
121+
mode: daemonset
122+
nodeAffinity:
123+
preferredDuringSchedulingIgnoredDuringExecution:
124+
- weight: 50
125+
preference:
126+
matchExpressions:
127+
- key: node.kubernetes.io/instance-type
128+
operator: In
129+
values:
130+
- m5.large
131+
- m5.xlarge

‎depot.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
"id": "rbk8l125wg"
3+
}

0 commit comments

Comments
 (0)