Docker Cookbook

324 | Chapter 10: Application Use Cases

---

Figure 10-6. The Spark master UI
This is it. The ease of deployment comes from the fact that the Spark workers try to
reach the master node with the hostname 
spark-master
. Because the Docker Net‐
work manages name resolution, each container automatically knows the IP of the
master and can reach it.
Discussion
If you check the network services that have been published, you see your four con‐
tainers on the multihost network (i.e., 
spark-master
, 
worker-1
, 
worker-2
,
worker-3
). But since you also published the ports for the UI, each container was also
attached to the bridge network. In the following example, you see only the worker
nodes on the bridge because this lists the services on the node that is not running the
master. If you check the Docker host that is running the master, you will see that the
spark-master
is also on the bridge network:
$ docker service ls
SERVICE ID NAME NETWORK CONTAINER
92e90b6556b5 worker-1 bridge ba80b36e5abc
1831b9378d37 worker-2 bridge c1c8bec01a2a
bc64584793df worker-3 bridge f7be3797affb
2bbe00afc559 worker-1 multihost ba80b36e5abc
7be77369a0ac worker-2 multihost c1c8bec01a2a
10.7 DATA: Creating a Spark Cluster | 325

---

3a576b7233b6 worker-3 multihost f7be3797affb
e3c75728c402 spark-master multihost fa44cce982df
Since you exposed the Spark worker’s web interface port, you can access the UI.
Figure 10-7
 shows a snapshot of a task that has already completed on this worker.
Figure 10-7. The Spark worker UI
The task shown in the dashboard is the result of running the 
Spark shell
, which is a
quick way to start learning Spark and running tasks on your containerized Spark
cluster. You can run the Spark shell via another interactive container as shown here:
$ docker run -it gcr.io/google_containers/spark-base
root@ac912dd21619:/# . ./setup_client.sh spark-master 7077
root@ac912dd21619:/# pyspark
Python 2.7.9 (default, Mar 1 2015, 12:57:24)
...
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 1.4.0
/_/
...
>>>