You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Note that workers connect to master node through 7077 exposed to actual physical machine. Remember to configure port forwarding if you run docker inside a virtual machine.
61
61
62
-
After it starts, go to [localhost:8080](http://localhost:8080) and get the master URL. In our case is `spark://11168790f9c1:7077`. You will also need the container alias, `nervous_noyce`, to enable a link between master and slave containers. List containers with `docker ps` to retrieve it.
62
+
After it starts, go to [localhost:8080](http://localhost:8080) and get the master URL. In our case is `spark://11168790f9c1:7077`. You will also need the container alias, `nervous_noyce`, to enable a link between master and worker containers. List containers with `docker ps` to retrieve it.
The worker node should be displayed in the master UI.
@@ -84,17 +84,26 @@ $ docker run -p 8081:8081 \
84
84
85
85
## Using Docker Compose
86
86
87
+
To bring up a mini-cluster with a master node and one worker:
88
+
87
89
```
88
-
$ docker-compose up
90
+
$ dockercompose up
89
91
```
90
92
91
-
Running `docker ps` will show containers and their ports mapped. Slaves can connect to master using internal DNS resolution, we've exposed the master node as `master`. Note that exposing worker nodes port is not straight-forward and we've leaved commented port mapping definition - we'll discuss that in class.
93
+
The master UI should be available at [localhost:8080](http://localhost:8080).
94
+
95
+
Then you can also connect to it via `pyspark`:
96
+
97
+
```
98
+
$ docker compose run -p 4040:4040 master pyspark --master spark://master:7077
99
+
```
100
+
101
+
Running `docker ps` will show containers and their ports mapped. Workers can connect to master using internal DNS resolution, we've exposed the master node as `master`. Note that exposing worker nodes port is not straight-forward, we'll discuss that in class.
0 commit comments