Skip to content

Commit ae84eff

Browse files
memalhotmemalhot
committed
add rbac for system set up guidance
1 parent 4e49646 commit ae84eff

File tree

4 files changed

+193
-3
lines changed

4 files changed

+193
-3
lines changed

‎README.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
## Overview
44

55
`python-batchtools` is a CLI for students and researchers to
6-
submit **GPU batch jobs** through **Kueue-managed GPU queues** on an
6+
submit **GPU batch jobs** through Kueue-managed GPU queues on an
77
OpenShift cluster. It provides an inexpensive and accessible way to use
88
GPU hardware without reserving dedicated GPU nodes.
99

@@ -13,7 +13,7 @@ Users submit GPU jobs with a single command:
1313
batchtools br "./cuda_program"
1414
```
1515

16-
The CLI automatically:
16+
The CLI will automatically:
1717
- Creates the batch job<br>
1818
- Submits it to the appropriate Kueue-managed LocalQueue<br>
1919
- Tracks job status<br>
@@ -37,13 +37,14 @@ pip install -e .
3737

3838
## Prerequisites
3939

40-
1. A Kueue-enabled OpenShift cluster, with local-queues named: v100-localqueue, a100-localqueue, h100-localqueue, dummy-localqueue<br>
40+
1. A Kueue-enabled OpenShift cluster, with localqueues named: v100-localqueue, a100-localqueue, h100-localqueue, dummy-localqueue<br>
4141
2. An OpenShift account<br>
4242
3. The Python OpenShift client:
4343

4444
``` sh
4545
pip install openshift-client
4646
```
47+
4. RBAC permissions for the user to have access to jobs, kueue resources like localqueues and clusterqueues. See **rbac** folder for setup.
4748

4849
# Usage Examples
4950

‎rbac/clusterrole.yaml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
apiVersion: rbac.authorization.k8s.io/v1
2+
kind: ClusterRole
3+
metadata:
4+
name: kueue-clusterqueue-reader
5+
rules:
6+
- apiGroups: ["kueue.x-k8s.io"]
7+
resources: ["clusterqueues"]
8+
verbs: ["get", "list", "watch"]
9+
---
10+
apiVersion: rbac.authorization.k8s.io/v1
11+
kind: ClusterRole
12+
metadata:
13+
name: node-reader
14+
rules:
15+
- apiGroups: [""]
16+
resources: ["nodes"]
17+
verbs: ["get", "list"]
18+
---
19+
apiVersion: rbac.authorization.k8s.io/v1
20+
kind: ClusterRole
21+
metadata:
22+
name: pod-reader
23+
rules:
24+
- apiGroups: [""]
25+
resources: ["pods"]
26+
verbs: ["get", "list"]

‎rbac/jobs.yaml‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
apiVersion: template.openshift.io/v1
2+
kind: Template
3+
metadata:
4+
name: rbac-default-edit-and-jobs
5+
parameters:
6+
- name: NAMESPACE
7+
required: true
8+
- name: SERVICE_ACCOUNT_NB
9+
required: true
10+
objects:
11+
# OC AUTH WB (BINDINGS FOR WORKBOOKS)
12+
# ROLEBINDING FOR EXISTING EDIT ROLE
13+
- apiVersion: rbac.authorization.k8s.io/v1
14+
kind: RoleBinding
15+
metadata:
16+
name: ${SERVICE_ACCOUNT_NB}-edit
17+
namespace: ${NAMESPACE}
18+
roleRef:
19+
apiGroup: rbac.authorization.k8s.io
20+
kind: ClusterRole
21+
name: edit
22+
subjects:
23+
- kind: ServiceAccount
24+
name: ${SERVICE_ACCOUNT_NB}
25+
namespace: ${NAMESPACE}
26+
27+
# CREATE ROLE EDIT-JOBS
28+
- apiVersion: rbac.authorization.k8s.io/v1
29+
kind: Role
30+
metadata:
31+
name: edit-jobs
32+
namespace: ${NAMESPACE}
33+
rules:
34+
- apiGroups: ["batch"]
35+
resources: ["jobs"]
36+
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
37+
38+
# BIND EDIT-JOBS TO THE NOTEBOOK SA
39+
- apiVersion: rbac.authorization.k8s.io/v1
40+
kind: RoleBinding
41+
metadata:
42+
name: ${SERVICE_ACCOUNT_NB}-edit-jobs
43+
namespace: ${NAMESPACE}
44+
roleRef:
45+
apiGroup: rbac.authorization.k8s.io
46+
kind: Role
47+
name: edit-jobs
48+
subjects:
49+
- kind: ServiceAccount
50+
name: ${SERVICE_ACCOUNT_NB}
51+
namespace: ${NAMESPACE}
52+
53+
# CREATE ROLE FOR LOCAL QUEUE
54+
- apiVersion: rbac.authorization.k8s.io/v1
55+
kind: Role
56+
metadata:
57+
name: kueue-localqueue-reader
58+
namespace: ${NAMESPACE}
59+
rules:
60+
- apiGroups: ["kueue.x-k8s.io"]
61+
resources: ["localqueues"]
62+
verbs: ["get", "list", "watch"]
63+
64+
# BIND LOCAL QUEUE READER TO THE NOTEBOOK SA
65+
- apiVersion: rbac.authorization.k8s.io/v1
66+
kind: RoleBinding
67+
metadata:
68+
name: ${SERVICE_ACCOUNT_NB}-kueue-localqueue-reader
69+
namespace: ${NAMESPACE}
70+
roleRef:
71+
apiGroup: rbac.authorization.k8s.io
72+
kind: Role
73+
name: kueue-localqueue-reader
74+
subjects:
75+
- kind: ServiceAccount
76+
name: ${SERVICE_ACCOUNT_NB}
77+
namespace: ${NAMESPACE}
78+
79+
# OC AUTH EXEC (BINDINGS FOR DEFAULT SERVICE ACCOUNT)
80+
# BIND TO EXISTING EDIT ROLE
81+
- apiVersion: rbac.authorization.k8s.io/v1
82+
kind: RoleBinding
83+
metadata:
84+
name: default-edit
85+
namespace: ${NAMESPACE}
86+
roleRef:
87+
apiGroup: rbac.authorization.k8s.io
88+
kind: ClusterRole
89+
name: edit
90+
subjects:
91+
- kind: ServiceAccount
92+
name: default
93+
namespace: ${NAMESPACE}
94+
95+
# BIND TO INTERACTING WITH JOBS ROLE
96+
- apiVersion: rbac.authorization.k8s.io/v1
97+
kind: RoleBinding
98+
metadata:
99+
name: default-edit-jobs
100+
namespace: ${NAMESPACE}
101+
roleRef:
102+
apiGroup: rbac.authorization.k8s.io
103+
kind: Role
104+
name: edit-jobs
105+
subjects:
106+
- kind: ServiceAccount
107+
name: default
108+
namespace: ${NAMESPACE}
109+
110+
# PODS/EXEC ROLE
111+
- apiVersion: rbac.authorization.k8s.io/v1
112+
kind: Role
113+
metadata:
114+
name: default-edit-pods-exec
115+
namespace: ${NAMESPACE}
116+
rules:
117+
- apiGroups: [""]
118+
resources: ["pods/exec"]
119+
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
120+
121+
# BIND PODS/EXEC ROLE
122+
- apiVersion: rbac.authorization.k8s.io/v1
123+
kind: RoleBinding
124+
metadata:
125+
name: default-edit-pods-exec
126+
namespace: ${NAMESPACE}
127+
roleRef:
128+
apiGroup: rbac.authorization.k8s.io
129+
kind: Role
130+
name: default-edit-pods-exec
131+
subjects:
132+
- kind: ServiceAccount
133+
name: default
134+
namespace: ${NAMESPACE}

‎rbac/localqueue.yaml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
apiVersion: template.openshift.io/v1
2+
kind: Template
3+
metadata:
4+
name: localqueue
5+
parameters:
6+
- name: NAMESPACE
7+
required: true
8+
objects:
9+
- apiVersion: kueue.x-k8s.io/v1beta1
10+
kind: LocalQueue
11+
metadata:
12+
name: v100-localqueue
13+
namespace: ${NAMESPACE}
14+
spec:
15+
clusterQueue: v100-clusterqueue
16+
- apiVersion: kueue.x-k8s.io/v1beta1
17+
kind: LocalQueue
18+
metadata:
19+
name: a100-localqueue
20+
namespace: ${NAMESPACE}
21+
spec:
22+
clusterQueue: a100-clusterqueue
23+
- apiVersion: kueue.x-k8s.io/v1beta1
24+
kind: LocalQueue
25+
metadata:
26+
name: h100-localqueue
27+
namespace: ${NAMESPACE}
28+
spec:
29+
clusterQueue: h100-clusterqueue

0 commit comments

Comments
 (0)