Skip to content

Commit 28c702a

Browse files
authored
Merge pull request #44095 from sftim/20231126_improve_job_tutorials
Improve tutorials for Job
2 parents 08dc743 + 7e75688 commit 28c702a

File tree

4 files changed

+183
-125
lines changed

4 files changed

+183
-125
lines changed

content/en/docs/tasks/job/coarse-parallel-processing-work-queue.md

Lines changed: 91 additions & 85 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,21 @@
11
---
22
title: Coarse Parallel Processing Using a Work Queue
3-
min-kubernetes-server-version: v1.8
43
content_type: task
54
weight: 20
65
---
76

87

98
<!-- overview -->
109

11-
In this example, we will run a Kubernetes Job with multiple parallel
10+
In this example, you will run a Kubernetes Job with multiple parallel
1211
worker processes.
1312

1413
In this example, as each pod is created, it picks up one unit of work
1514
from a task queue, completes it, deletes it from the queue, and exits.
1615

1716
Here is an overview of the steps in this example:
1817

19-
1. **Start a message queue service.** In this example, we use RabbitMQ, but you could use another
18+
1. **Start a message queue service.** In this example, you use RabbitMQ, but you could use another
2019
one. In practice you would set up a message queue service once and reuse it for many jobs.
2120
1. **Create a queue, and fill it with messages.** Each message represents one task to be done. In
2221
this example, a message is an integer that we will do a lengthy computation on.
@@ -26,11 +25,16 @@ Here is an overview of the steps in this example:
2625
## {{% heading "prerequisites" %}}
2726

2827

29-
Be familiar with the basic,
28+
You should already be familiar with the basic,
3029
non-parallel, use of [Job](/docs/concepts/workloads/controllers/job/).
3130

3231
{{< include "task-tutorial-prereqs.md" >}}
3332

33+
You will need a container image registry where you can upload images to run in your cluster.
34+
35+
This task example also assumes that you have Docker installed locally.
36+
37+
3438
<!-- steps -->
3539

3640
## Starting a message queue service
@@ -43,21 +47,20 @@ cluster and reuse it for many jobs, as well as for long-running services.
4347
Start RabbitMQ as follows:
4448

4549
```shell
46-
kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/release-1.3/examples/celery-rabbitmq/rabbitmq-service.yaml
50+
# make a Service for the StatefulSet to use
51+
kubectl create -f https://kubernetes.io/examples/application/job/rabbitmq-service.yaml
4752
```
4853
```
4954
service "rabbitmq-service" created
5055
```
5156

5257
```shell
53-
kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/release-1.3/examples/celery-rabbitmq/rabbitmq-controller.yaml
58+
kubectl create -f https://kubernetes.io/examples/application/job/rabbitmq-statefulset.yaml
5459
```
5560
```
56-
replicationcontroller "rabbitmq-controller" created
61+
statefulset "rabbitmq" created
5762
```
5863

59-
We will only use the rabbitmq part from the [celery-rabbitmq example](https://github.com/kubernetes/kubernetes/tree/release-1.3/examples/celery-rabbitmq).
60-
6164
## Testing the message queue service
6265

6366
Now, we can experiment with accessing the message queue. We will
@@ -68,7 +71,7 @@ First create a temporary interactive Pod.
6871

6972
```shell
7073
# Create a temporary interactive container
71-
kubectl run -i --tty temp --image ubuntu:18.04
74+
kubectl run -i --tty temp --image ubuntu:22.04
7275
```
7376
```
7477
Waiting for pod default/temp-loe07 to be running, status is Pending, pod ready: false
@@ -77,76 +80,82 @@ Waiting for pod default/temp-loe07 to be running, status is Pending, pod ready:
7780

7881
Note that your pod name and command prompt will be different.
7982

80-
Next install the `amqp-tools` so we can work with message queues.
83+
Next install the `amqp-tools` so you can work with message queues.
84+
The next commands show what you need to run inside the interactive shell in that Pod:
8185

8286
```shell
83-
# Install some tools
84-
root@temp-loe07:/# apt-get update
85-
.... [ lots of output ] ....
86-
root@temp-loe07:/# apt-get install -y curl ca-certificates amqp-tools python dnsutils
87-
.... [ lots of output ] ....
87+
apt-get update && apt-get install -y curl ca-certificates amqp-tools python dnsutils
8888
```
8989

90-
Later, we will make a docker image that includes these packages.
90+
Later, you will make a container image that includes these packages.
9191

92-
Next, we will check that we can discover the rabbitmq service:
92+
Next, you will check that you can discover the Service for RabbitMQ:
9393

9494
```
95+
# Run these commands inside the Pod
9596
# Note the rabbitmq-service has a DNS name, provided by Kubernetes:
96-
97-
root@temp-loe07:/# nslookup rabbitmq-service
97+
nslookup rabbitmq-service
98+
```
99+
```
98100
Server: 10.0.0.10
99101
Address: 10.0.0.10#53
100102
101103
Name: rabbitmq-service.default.svc.cluster.local
102104
Address: 10.0.147.152
103-
104-
# Your address will vary.
105105
```
106+
(the IP addresses will vary)
106107

107-
If Kube-DNS is not set up correctly, the previous step may not work for you.
108-
You can also find the service IP in an env var:
108+
If the kube-dns addon is not set up correctly, the previous step may not work for you.
109+
You can also find the IP address for that Service in an environment variable:
109110

111+
```shell
112+
# run this check inside the Pod
113+
env | grep RABBITMQ_SERVICE | grep HOST
114+
```
110115
```
111-
# env | grep RABBIT | grep HOST
112116
RABBITMQ_SERVICE_SERVICE_HOST=10.0.147.152
113-
# Your address will vary.
114117
```
118+
(the IP address will vary)
115119

116-
Next we will verify we can create a queue, and publish and consume messages.
120+
Next you will verify that you can create a queue, and publish and consume messages.
117121

118122
```shell
123+
# Run these commands inside the Pod
119124
# In the next line, rabbitmq-service is the hostname where the rabbitmq-service
120125
# can be reached. 5672 is the standard port for rabbitmq.
121-
122-
root@temp-loe07:/# export BROKER_URL=amqp://guest:guest@rabbitmq-service:5672
126+
export BROKER_URL=amqp://guest:guest@rabbitmq-service:5672
123127
# If you could not resolve "rabbitmq-service" in the previous step,
124128
# then use this command instead:
125-
# root@temp-loe07:/# BROKER_URL=amqp://guest:guest@$RABBITMQ_SERVICE_SERVICE_HOST:5672
129+
BROKER_URL=amqp://guest:guest@$RABBITMQ_SERVICE_SERVICE_HOST:5672
126130

127131
# Now create a queue:
128132

129-
root@temp-loe07:/# /usr/bin/amqp-declare-queue --url=$BROKER_URL -q foo -d
133+
/usr/bin/amqp-declare-queue --url=$BROKER_URL -q foo -d
134+
```
135+
```
130136
foo
137+
```
131138

132-
# Publish one message to it:
133-
134-
root@temp-loe07:/# /usr/bin/amqp-publish --url=$BROKER_URL -r foo -p -b Hello
139+
Publish one message to the queue:
140+
```shell
141+
/usr/bin/amqp-publish --url=$BROKER_URL -r foo -p -b Hello
135142

136143
# And get it back.
137144

138-
root@temp-loe07:/# /usr/bin/amqp-consume --url=$BROKER_URL -q foo -c 1 cat && echo
145+
/usr/bin/amqp-consume --url=$BROKER_URL -q foo -c 1 cat && echo 1>&2
146+
```
147+
```
139148
Hello
140-
root@temp-loe07:/#
141149
```
142150

143-
In the last command, the `amqp-consume` tool takes one message (`-c 1`)
144-
from the queue, and passes that message to the standard input of an arbitrary command. In this case, the program `cat` prints out the characters read from standard input, and the echo adds a carriage
145-
return so the example is readable.
151+
In the last command, the `amqp-consume` tool took one message (`-c 1`)
152+
from the queue, and passes that message to the standard input of an arbitrary command.
153+
In this case, the program `cat` prints out the characters read from standard input, and
154+
the echo adds a carriage return so the example is readable.
146155

147-
## Filling the Queue with tasks
156+
## Fill the queue with tasks
148157

149-
Now let's fill the queue with some "tasks". In our example, our tasks are strings to be
158+
Now, fill the queue with some simulated tasks. In this example, the tasks are strings to be
150159
printed.
151160

152161
In a practice, the content of the messages might be:
@@ -157,33 +166,37 @@ In a practice, the content of the messages might be:
157166
- configuration parameters to a simulation
158167
- frame numbers of a scene to be rendered
159168

160-
In practice, if there is large data that is needed in a read-only mode by all pods
161-
of the Job, you will typically put that in a shared file system like NFS and mount
162-
that readonly on all the pods, or the program in the pod will natively read data from
163-
a cluster file system like HDFS.
169+
If there is large data that is needed in a read-only mode by all pods
170+
of the Job, you typically put that in a shared file system like NFS and mount
171+
that readonly on all the pods, or write the program in the pod so that it can natively read
172+
data from a cluster file system (for example: HDFS).
164173

165-
For our example, we will create the queue and fill it using the amqp command line tools.
166-
In practice, you might write a program to fill the queue using an amqp client library.
174+
For this example, you will create the queue and fill it using the AMQP command line tools.
175+
In practice, you might write a program to fill the queue using an AMQP client library.
167176

168177
```shell
178+
# Run this on your computer, not in the Pod
169179
/usr/bin/amqp-declare-queue --url=$BROKER_URL -q job1 -d
180+
```
181+
```
170182
job1
171183
```
184+
Add items to the queue:
172185
```shell
173186
for f in apple banana cherry date fig grape lemon melon
174187
do
175188
/usr/bin/amqp-publish --url=$BROKER_URL -r job1 -p -b $f
176189
done
177190
```
178191

179-
So, we filled the queue with 8 messages.
192+
You added 8 messages to the queue.
180193

181-
## Create an Image
194+
## Create a container image
182195

183-
Now we are ready to create an image that we will run as a job.
196+
Now you are ready to create an image that you will run as a Job.
184197

185-
We will use the `amqp-consume` utility to read the message
186-
from the queue and run our actual program. Here is a very simple
198+
The job will use the `amqp-consume` utility to read the message
199+
from the queue and run the actual work. Here is a very simple
187200
example program:
188201

189202
{{% code_sample language="python" file="application/job/rabbitmq/worker.py" %}}
@@ -194,9 +207,7 @@ Give the script execution permission:
194207
chmod +x worker.py
195208
```
196209

197-
Now, build an image. If you are working in the source
198-
tree, then change directory to `examples/job/work-queue-1`.
199-
Otherwise, make a temporary directory, change to it,
210+
Now, build an image. Make a temporary directory, change to it,
200211
download the [Dockerfile](/examples/application/job/rabbitmq/Dockerfile),
201212
and [worker.py](/examples/application/job/rabbitmq/worker.py). In either case,
202213
build the image with this command:
@@ -214,33 +225,27 @@ docker tag job-wq-1 <username>/job-wq-1
214225
docker push <username>/job-wq-1
215226
```
216227

217-
If you are using [Google Container
218-
Registry](https://cloud.google.com/tools/container-registry/), tag
219-
your app image with your project ID, and push to GCR. Replace
220-
`<project>` with your project ID.
221-
222-
```shell
223-
docker tag job-wq-1 gcr.io/<project>/job-wq-1
224-
gcloud docker -- push gcr.io/<project>/job-wq-1
225-
```
228+
If you are using an alternative container image registry, tag the
229+
image and push it there instead.
226230

227231
## Defining a Job
228232

229-
Here is a job definition. You'll need to make a copy of the Job and edit the
230-
image to match the name you used, and call it `./job.yaml`.
231-
233+
Here is a manifest for a Job. You'll need to make a copy of the Job manifest
234+
(call it `./job.yaml`),
235+
and edit the name of the container image to match the name you used.
232236

233237
{{% code_sample file="application/job/rabbitmq/job.yaml" %}}
234238

235239
In this example, each pod works on one item from the queue and then exits.
236240
So, the completion count of the Job corresponds to the number of work items
237-
done. So we set, `.spec.completions: 8` for the example, since we put 8 items in the queue.
241+
done. That is why the example manifest has `.spec.completions` set to `8`.
238242

239243
## Running the Job
240244

241-
So, now run the Job:
245+
Now, run the Job:
242246

243247
```shell
248+
# this assumes you downloaded and then edited the manifest already
244249
kubectl apply -f ./job.yaml
245250
```
246251

@@ -264,14 +269,14 @@ Labels: controller-uid=41d75705-92df-11e7-b85e-fa163ee3c11f
264269
Annotations: <none>
265270
Parallelism: 2
266271
Completions: 8
267-
Start Time: Wed, 06 Sep 2017 16:42:02 +0800
272+
Start Time: Wed, 06 Sep 2022 16:42:02 +0000
268273
Pods Statuses: 0 Running / 8 Succeeded / 0 Failed
269274
Pod Template:
270275
Labels: controller-uid=41d75705-92df-11e7-b85e-fa163ee3c11f
271276
job-name=job-wq-1
272277
Containers:
273278
c:
274-
Image: gcr.io/causal-jigsaw-637/job-wq-1
279+
Image: container-registry.example/causal-jigsaw-637/job-wq-1
275280
Port:
276281
Environment:
277282
BROKER_URL: amqp://guest:guest@rabbitmq-service:5672
@@ -293,30 +298,31 @@ Events:
293298

294299

295300

296-
All the pods for that Job succeeded. Yay.
297-
301+
All the pods for that Job succeeded! You're done.
298302

299303

300304
<!-- discussion -->
301305

302306
## Alternatives
303307

304-
This approach has the advantage that you
305-
do not need to modify your "worker" program to be aware that there is a work queue.
308+
This approach has the advantage that you do not need to modify your "worker" program to be
309+
aware that there is a work queue. You can include the worker program unmodified in your container
310+
image.
306311

307-
It does require that you run a message queue service.
312+
Using this approach does require that you run a message queue service.
308313
If running a queue service is inconvenient, you may
309314
want to consider one of the other [job patterns](/docs/concepts/workloads/controllers/job/#job-patterns).
310315

311316
This approach creates a pod for every work item. If your work items only take a few seconds,
312317
though, creating a Pod for every work item may add a lot of overhead. Consider another
313-
[example](/docs/tasks/job/fine-parallel-processing-work-queue/), that executes multiple work items per Pod.
318+
design, such as in the [fine parallel work queue example](/docs/tasks/job/fine-parallel-processing-work-queue/),
319+
that executes multiple work items per Pod.
314320

315-
In this example, we use the `amqp-consume` utility to read the message
316-
from the queue and run our actual program. This has the advantage that you
321+
In this example, you used the `amqp-consume` utility to read the message
322+
from the queue and run the actual program. This has the advantage that you
317323
do not need to modify your program to be aware of the queue.
318-
A [different example](/docs/tasks/job/fine-parallel-processing-work-queue/), shows how to
319-
communicate with the work queue using a client library.
324+
The [fine parallel work queue example](/docs/tasks/job/fine-parallel-processing-work-queue/)
325+
shows how to communicate with the work queue using a client library.
320326

321327
## Caveats
322328

@@ -327,11 +333,11 @@ If the number of completions is set to more than the number of items in the queu
327333
then the Job will not appear to be completed, even though all items in the queue
328334
have been processed. It will start additional pods which will block waiting
329335
for a message.
336+
You would need to make your own mechanism to spot when there is work
337+
to do and measure the size of the queue, setting the number of completions to match.
330338

331339
There is an unlikely race with this pattern. If the container is killed in between the time
332-
that the message is acknowledged by the amqp-consume command and the time that the container
340+
that the message is acknowledged by the `amqp-consume` command and the time that the container
333341
exits with success, or if the node crashes before the kubelet is able to post the success of the pod
334-
back to the api-server, then the Job will not appear to be complete, even though all items
342+
back to the API server, then the Job will not appear to be complete, even though all items
335343
in the queue have been processed.
336-
337-

0 commit comments

Comments
 (0)