1
1
---
2
2
title : 使用工作队列进行精细的并行处理
3
3
content_type : task
4
- min-kubernetes-server-version : v1.8
5
4
weight : 30
6
5
---
7
6
<!--
8
7
title: Fine Parallel Processing Using a Work Queue
9
8
content_type: task
10
9
weight: 30
11
- min-kubernetes-server-version: v1.8
12
10
-->
13
11
14
12
<!-- overview -->
15
13
16
14
<!--
17
- In this example, we will run a Kubernetes Job with multiple parallel
18
- worker processes in a given pod .
15
+ In this example, you will run a Kubernetes Job that runs multiple parallel
16
+ tasks as worker processes, each running as a separate Pod .
19
17
-->
20
- 在这个例子中,我们会运行一个 Kubernetes Job,其中的 Pod 会运行多个并行工作进程。
18
+ 在此示例中,你将运行一个 Kubernetes Job,该 Job 将多个并行任务作为工作进程运行,
19
+ 每个任务在单独的 Pod 中运行。
21
20
22
21
<!--
23
22
In this example, as each pod is created, it picks up one unit of work
@@ -30,17 +29,19 @@ Here is an overview of the steps in this example:
30
29
下面是这个示例的步骤概述:
31
30
32
31
<!--
33
- 1. **Start a storage service to hold the work queue.** In this example, we use Redis to store
34
- our work items. In the previous example, we used RabbitMQ. In this example, we use Redis and
35
- a custom work-queue client library because AMQP does not provide a good way for clients to
32
+ 1. **Start a storage service to hold the work queue.** In this example, you will use Redis to store
33
+ work items. In the [previous example](/docs/tasks/job/coarse-parallel-processing-work-queue),
34
+ you used RabbitMQ. In this example, you will use Redis and a custom work-queue client library;
35
+ this is because AMQP does not provide a good way for clients to
36
36
detect when a finite-length work queue is empty. In practice you would set up a store such
37
37
as Redis once and reuse it for the work queues of many jobs, and other things.
38
38
-->
39
39
40
- 1 . ** 启动存储服务用于保存工作队列。** 在这个例子中,我们使用 Redis 来存储工作项。
41
- 在上一个例子中,我们使用了 RabbitMQ。
40
+ 1 . ** 启动存储服务用于保存工作队列。** 在这个例子中,你将使用 Redis 来存储工作项。
41
+ 在[ 上一个例子中] ( /zh-cn/docs/tasks/job/coarse-parallel-processing-work-queue ) ,
42
+ 你使用了 RabbitMQ。
42
43
在这个例子中,由于 AMQP 不能为客户端提供一个良好的方法来检测一个有限长度的工作队列是否为空,
43
- 我们使用了 Redis 和一个自定义的工作队列客户端库。
44
+ 你将使用 Redis 和一个自定义的工作队列客户端库。
44
45
在实践中,你可能会设置一个类似于 Redis 的存储库,并将其同时用于多项任务或其他事务的工作队列。
45
46
46
47
<!--
@@ -61,6 +62,20 @@ Here is an overview of the steps in this example:
61
62
62
63
{{< include "task-tutorial-prereqs.md" >}}
63
64
65
+ <!--
66
+ You will need a container image registry where you can upload images to run in your cluster.
67
+ The example uses [Docker Hub](https://hub.docker.com/), but you could adapt it to a different
68
+ container image registry.
69
+
70
+ This task example also assumes that you have Docker installed locally. You use Docker to
71
+ build container images.
72
+ -->
73
+ 你将需要一个容器镜像仓库,可以向其中上传镜像以在集群中运行。
74
+ 此示例使用的是 [ Docker Hub] ( https://hub.docker.com/ ) ,
75
+ 当然你可以将其调整为别的容器镜像仓库。
76
+
77
+ 此任务示例还假设你已在本地安装了 Docker,并使用 Docker 来构建容器镜像。
78
+
64
79
<!-- steps -->
65
80
66
81
<!--
@@ -74,13 +89,13 @@ non-parallel, use of [Job](/docs/concepts/workloads/controllers/job/).
74
89
<!--
75
90
## Starting Redis
76
91
77
- For this example, for simplicity, we will start a single instance of Redis.
92
+ For this example, for simplicity, you will start a single instance of Redis.
78
93
See the [Redis Example](https://github.com/kubernetes/examples/tree/master/guestbook) for an example
79
94
of deploying Redis scalably and redundantly.
80
95
-->
81
96
## 启动 Redis
82
97
83
- 对于这个例子,为了简单起见,我们将启动一个单实例的 Redis。
98
+ 对于这个例子,为了简单起见,你将启动一个单实例的 Redis。
84
99
了解如何部署一个可伸缩、高可用的 Redis 例子,请查看
85
100
[ Redis 示例] ( https://github.com/kubernetes/examples/tree/master/guestbook )
86
101
@@ -97,9 +112,9 @@ You could also download the following files directly:
97
112
- [ ` worker.py ` ] ( /examples/application/job/redis/worker.py )
98
113
99
114
<!--
100
- ## Filling the Queue with tasks
115
+ ## Filling the queue with tasks
101
116
102
- Now let's fill the queue with some "tasks". In our example, our tasks are strings to be
117
+ Now let's fill the queue with some "tasks". In this example, our tasks are strings to be
103
118
printed.
104
119
105
120
Start a temporary interactive pod for running the Redis CLI.
@@ -121,12 +136,14 @@ Hit enter for command prompt
121
136
```
122
137
123
138
<!--
124
- Now hit enter, start the redis CLI, and create a list with some work items in it.
139
+ Now hit enter, start the Redis CLI, and create a list with some work items in it.
125
140
-->
126
141
现在按回车键,启动 Redis 命令行界面,然后创建一个存在若干个工作项的列表。
127
142
128
143
``` shell
129
- # redis-cli -h redis
144
+ redis-cli -h redis
145
+ ```
146
+ ``` console
130
147
redis:6379 > rpush job2 " apple"
131
148
(integer) 1
132
149
redis:6379 > rpush job2 " banana"
@@ -158,9 +175,9 @@ redis:6379> lrange job2 0 -1
158
175
```
159
176
160
177
<!--
161
- So, the list with key `job2` will be our work queue.
178
+ So, the list with key `job2` will be the work queue.
162
179
-->
163
- 因此,这个键为 ` job2 ` 的列表就是我们的工作队列 。
180
+ 因此,这个键为 ` job2 ` 的列表就是工作队列 。
164
181
165
182
<!--
166
183
Note: if you do not have Kube DNS setup correctly, you may need to change
@@ -170,23 +187,24 @@ the first step of the above block to `redis-cli -h $REDIS_SERVICE_HOST`.
170
187
` redis-cli -h $REDIS_SERVICE_HOST ` 。
171
188
172
189
<!--
173
- ## Create an Image
190
+ ## Create a container image {#create-an-image}
174
191
175
- Now we are ready to create an image that we will run .
192
+ Now you are ready to create an image that will process the work in that queue .
176
193
177
- We will use a python worker program with a redis client to read
194
+ You're going to use a Python worker program with a Redis client to read
178
195
the messages from the message queue.
179
196
180
197
A simple Redis work queue client library is provided,
181
- called rediswq.py ([Download](/examples/application/job/redis/rediswq.py)).
198
+ called ` rediswq.py` ([Download](/examples/application/job/redis/rediswq.py)).
182
199
-->
183
- ## 创建镜像
200
+ ## 创建容器镜像 {#create-an-image}
184
201
185
- 现在我们已经准备好创建一个我们要运行的镜像 。
202
+ 现在你已准备好创建一个镜像来处理该队列中的工作 。
186
203
187
- 我们会使用一个带有 Redis 客户端的 Python 工作程序从消息队列中读出消息。
204
+ 你将使用一个带有 Redis 客户端的 Python 工作程序从消息队列中读出消息。
188
205
189
- 这里提供了一个简单的 Redis 工作队列客户端库,名为 rediswq.py ([ 下载] ( /examples/application/job/redis/rediswq.py ) )。
206
+ 这里提供了一个简单的 Redis 工作队列客户端库,名为 ` rediswq.py `
207
+ ([ 下载] ( /zh-cn/examples/application/job/redis/rediswq.py ) )。
190
208
191
209
<!--
192
210
The "worker" program in each Pod of the Job uses the work queue
@@ -200,11 +218,12 @@ Job 中每个 Pod 内的“工作程序” 使用工作队列客户端库获取
200
218
You could also download [`worker.py`](/examples/application/job/redis/worker.py),
201
219
[`rediswq.py`](/examples/application/job/redis/rediswq.py), and
202
220
[`Dockerfile`](/examples/application/job/redis/Dockerfile) files, then build
203
- the image:
221
+ the container image. Here's an example using Docker to do the image build :
204
222
-->
205
223
你也可以下载 [ ` worker.py ` ] ( /examples/application/job/redis/worker.py ) 、
206
224
[ ` rediswq.py ` ] ( /examples/application/job/redis/rediswq.py ) 和
207
- [ ` Dockerfile ` ] ( /examples/application/job/redis/Dockerfile ) 文件。然后构建镜像:
225
+ [ ` Dockerfile ` ] ( /examples/application/job/redis/Dockerfile ) 文件。然后构建容器镜像。
226
+ 以下是使用 Docker 进行镜像构建的示例:
208
227
209
228
``` shell
210
229
docker build -t job-wq-2 .
@@ -234,52 +253,40 @@ your private repository](/docs/concepts/containers/images/).
234
253
你需要将镜像 push 到一个公共仓库或者
235
254
[ 配置集群访问你的私有仓库] ( /zh-cn/docs/concepts/containers/images/ ) 。
236
255
237
- <!--
238
- If you are using [Google Container
239
- Registry](https://cloud.google.com/tools/container-registry/), tag
240
- your app image with your project ID, and push to GCR. Replace
241
- `<project>` with your project ID.
242
- -->
243
- 如果你使用的是 [ Google Container Registry] ( https://cloud.google.com/tools/container-registry/ ) ,
244
- 请先用你的 project ID 给你的镜像打上标签,然后 push 到 GCR。请将 ` <project> ` 替换为你自己的 project ID。
245
-
246
- ``` shell
247
- docker tag job-wq-2 gcr.io/< project> /job-wq-2
248
- gcloud docker -- push gcr.io/< project> /job-wq-2
249
- ```
250
-
251
256
<!--
252
257
## Defining a Job
253
258
254
- Here is the job definition :
259
+ Here is a manifest for the Job you will create :
255
260
-->
256
261
## 定义一个 Job
257
262
258
- 这是 Job 定义 :
263
+ 以下是你将创建的 Job 的清单 :
259
264
260
265
{{% code_sample file="application/job/redis/job.yaml" %}}
261
266
267
+ {{< note >}}
262
268
<!--
263
- Be sure to edit the job template to
269
+ Be sure to edit the manifest to
264
270
change `gcr.io/myproject` to your own path.
265
271
-->
266
- 请确保将 Job 模板中的 ` gcr.io/myproject ` 更改为你自己的路径。
272
+ 请确保将 Job 清单中的 ` gcr.io/myproject ` 更改为你自己的路径。
273
+ {{< /note >}}
267
274
268
275
<!--
269
276
In this example, each pod works on several items from the queue and then exits when there are no more items.
270
277
Since the workers themselves detect when the workqueue is empty, and the Job controller does not
271
278
know about the workqueue, it relies on the workers to signal when they are done working.
272
- The workers signal that the queue is empty by exiting with success. So, as soon as any worker
273
- exits with success, the controller knows the work is done, and the Pods will exit soon.
274
- So, we set the completion count of the Job to 1. The job controller will wait for the other pods to complete
275
- too.
279
+ The workers signal that the queue is empty by exiting with success. So, as soon as ** any** worker
280
+ exits with success, the controller knows the work is done, and that the Pods will exit soon.
281
+ So, we need to set the completion count of the Job to 1. The job controller will wait for
282
+ the other pods to complete too.
276
283
-->
277
284
在这个例子中,每个 Pod 处理了队列中的多个项目,直到队列中没有项目时便退出。
278
285
因为是由工作程序自行检测工作队列是否为空,并且 Job 控制器不知道工作队列的存在,
279
286
这依赖于工作程序在完成工作时发出信号。
280
287
工作程序以成功退出的形式发出信号表示工作队列已经为空。
281
- 所以,只要有任意一个工作程序成功退出 ,控制器就知道工作已经完成了,所有的 Pod 将很快会退出。
282
- 因此,我们将 Job 的完成计数(Completion Count)设置为 1。
288
+ 所以,只要有 ** 任意 ** 一个工作程序成功退出 ,控制器就知道工作已经完成了,所有的 Pod 将很快会退出。
289
+ 因此,我们需要将 Job 的完成计数(Completion Count)设置为 1。
283
290
尽管如此,Job 控制器还是会等待其它 Pod 完成。
284
291
285
292
<!--
@@ -291,14 +298,18 @@ So, now run the Job:
291
298
292
299
现在运行这个 Job:
293
300
301
+ <!--
302
+ # this assumes you downloaded and then edited the manifest already
303
+ -->
294
304
``` shell
305
+ # 这假设你已经下载并编辑了清单
295
306
kubectl apply -f ./job.yaml
296
307
```
297
308
298
309
<!--
299
- Now wait a bit, then check on the job.
310
+ Now wait a bit, then check on the Job:
300
311
-->
301
- 稍等片刻,然后检查这个 Job。
312
+ 稍等片刻,然后检查这个 Job:
302
313
303
314
``` shell
304
315
kubectl describe jobs/job-wq-2
@@ -313,14 +324,14 @@ Labels: controller-uid=b1c7e4e3-92e1-11e7-b85e-fa163ee3c11f
313
324
Annotations: <none>
314
325
Parallelism: 2
315
326
Completions: <unset>
316
- Start Time: Mon, 11 Jan 2016 17:07:59 -0800
327
+ Start Time: Mon, 11 Jan 2022 17:07:59 +0000
317
328
Pods Statuses: 1 Running / 0 Succeeded / 0 Failed
318
329
Pod Template:
319
330
Labels: controller-uid=b1c7e4e3-92e1-11e7-b85e-fa163ee3c11f
320
331
job-name=job-wq-2
321
332
Containers:
322
333
c:
323
- Image: gcr.io /exampleproject/job-wq-2
334
+ Image: container-registry.example /exampleproject/job-wq-2
324
335
Port:
325
336
Environment: <none>
326
337
Mounts: <none>
@@ -353,9 +364,9 @@ Working on lemon
353
364
```
354
365
355
366
<!--
356
- As you can see, one of our pods worked on several work units.
367
+ As you can see, one of the pods for this Job worked on several work units.
357
368
-->
358
- 你可以看到,其中的一个 Pod 处理了若干个工作单元。
369
+ 你可以看到,此 Job 中的一个 Pod 处理了若干个工作单元。
359
370
360
371
<!-- discussion -->
361
372
@@ -374,10 +385,10 @@ want to consider one of the other
374
385
375
386
<!--
376
387
If you have a continuous stream of background processing work to run, then
377
- consider running your background workers with a ` ReplicaSet` instead,
388
+ consider running your background workers with a ReplicaSet instead,
378
389
and consider running a background processing library such as
379
390
[https://github.com/resque/resque](https://github.com/resque/resque).
380
391
-->
381
- 如果你有持续的后台处理业务,那么可以考虑使用 ` ReplicaSet ` 来运行你的后台业务,
392
+ 如果你有持续的后台处理业务,那么可以考虑使用 ReplicaSet 来运行你的后台业务,
382
393
和运行一个类似 [ https://github.com/resque/resque ] ( https://github.com/resque/resque )
383
394
的后台处理库。
0 commit comments