Wiki - YARN - Run tasks in docker containers

[davidonlaptop]

Hadoop 2.6+ allows YARN to run the tasks inside Docker containers.. Explain how does it work if YARN itself is ran inside a docker container.

Obviously, this issue depends on #7 .

[momozaza]

Do you want an overview explanation about Docker Container Executor (DCE) or an example of a yarn launched docker containers?

[davidonlaptop]

Hadoop 2.6 offers a new feature that allows to launch a MapReduce application in a Docker container: https://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html

It is said in the documentation, that the "The DockerContainerExecutor requires Docker daemon to be running on the NodeManagers" which would be easy to do if we had a bare-metal installation of Hadoop. However, we'll be running the NodeManager itself in a Docker container. Therefore there will be no Docker daemon inside the NodeManager; the Docker daemon will be running in the host of the NodeManager's container. So our setup is different than the Hadoop doc because we're running YARN itself in Docker containers.

Like mentioned in the description, this issue depends on issue #7 which has been solved by @codingtony . Therefore, if you were to work on this issue, you should probably start your work based on his branch to add the DCE on top of it. @codingtony, could you advise @momozaza on best procedure to do that?

My idea on how to solve this would be:

1. Add docker client inside the Hadoop's Dockerfile. But do not start the docker daemon.
2. When launching the YARN ResourceManager, pass an environment variable to the Docker run command that tells where the docker host is located (will be different for every team and every users of this feature).

This is an advanced feature, especially since you have not learned MapReduce/YARN in class. @mikefaille may be able to help on the Docker / networking side and @codingtony may help on YARN concepts. Even if the feature is not completed, it may still provide some help for the future.