Skip to content

Commit 0ab2e59

Browse files
committed
Fix docker compose environment, replace slave by worker in documentation
1 parent 32e72e0 commit 0ab2e59

File tree

2 files changed

+19
-12
lines changed

2 files changed

+19
-12
lines changed

infra/docker/docker-compose.yml

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
version: '2'
1+
version: '3'
22
services:
33
master:
44
image: luisbelloch/spark
@@ -13,10 +13,8 @@ services:
1313
worker:
1414
image: luisbelloch/spark
1515
command: /opt/spark/sbin/start-slave.sh spark://master:7077
16+
environment:
1617
SPARK_PUBLIC_DNS: localhost
1718
SPARK_NO_DAEMONIZE: 1
18-
# Yeah, port festival, classic :-)
19-
# ports:
20-
# - 8081:8081
21-
links:
22-
- master
19+
ports:
20+
- 8081:8081

infra/docker/docker.md

Lines changed: 15 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ $ docker run -p 8080:8080 -p 7077:7077 -d luisbelloch/spark start-master.sh
5959

6060
Note that workers connect to master node through 7077 exposed to actual physical machine. Remember to configure port forwarding if you run docker inside a virtual machine.
6161

62-
After it starts, go to [localhost:8080](http://localhost:8080) and get the master URL. In our case is `spark://11168790f9c1:7077`. You will also need the container alias, `nervous_noyce`, to enable a link between master and slave containers. List containers with `docker ps` to retrieve it.
62+
After it starts, go to [localhost:8080](http://localhost:8080) and get the master URL. In our case is `spark://11168790f9c1:7077`. You will also need the container alias, `nervous_noyce`, to enable a link between master and worker containers. List containers with `docker ps` to retrieve it.
6363

6464
```
6565
$ docker ps
@@ -68,7 +68,7 @@ CONTAINER ID IMAGE NAMES
6868
6969
$ docker run -p 8081:8081 \
7070
--link nervous_noyce \
71-
-d luisbelloch/spark start-slave.sh spark://11168790f9c1:7077
71+
-d luisbelloch/spark start-worker.sh spark://11168790f9c1:7077
7272
```
7373

7474
The worker node should be displayed in the master UI.
@@ -84,17 +84,26 @@ $ docker run -p 8081:8081 \
8484

8585
## Using Docker Compose
8686

87+
To bring up a mini-cluster with a master node and one worker:
88+
8789
```
88-
$ docker-compose up
90+
$ docker compose up
8991
```
9092

91-
Running `docker ps` will show containers and their ports mapped. Slaves can connect to master using internal DNS resolution, we've exposed the master node as `master`. Note that exposing worker nodes port is not straight-forward and we've leaved commented port mapping definition - we'll discuss that in class.
93+
The master UI should be available at [localhost:8080](http://localhost:8080).
94+
95+
Then you can also connect to it via `pyspark`:
96+
97+
```
98+
$ docker compose run -p 4040:4040 master pyspark --master spark://master:7077
99+
```
100+
101+
Running `docker ps` will show containers and their ports mapped. Workers can connect to master using internal DNS resolution, we've exposed the master node as `master`. Note that exposing worker nodes port is not straight-forward, we'll discuss that in class.
92102

93103
To scale up/down the cluster:
94104

95105
```
96-
$ docker-compose scale worker=3
106+
$ docker compose scale worker=3
97107
```
98108

99109
Beware desired state persist between runs.
100-

0 commit comments

Comments
 (0)