Switch to docker for doc build

Author: mengxr

dev/build-docs-in-docker.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+set -euxo pipefail
+
+dev/run.py env-setup -d -c > ./python/.setup.sh && source ./python/.setup.sh
+
+
+cd ./docs
+SKIP_SCALADOC=1 PRODUCTION=1 jekyll build

dev/build-docs.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+# build the base image
+docker build -t databricks/sparkdl --build-arg PYTHON_VERSION=2.7 .
+
+# build the docs image
+docker build -t databricks/sparkdl-docs docs/
+
+# create the assembly jar on host because we have ivy/maven cache
+build/sbt assembly
+
+# build the API docs
+docker run --rm -v "$(pwd):/mnt/sparkdl" databricks/sparkdl-docs dev/build-docs-in-docker.sh

docs/Dockerfile

@@ -0,0 +1,17 @@
+# An docker image `databricks/sparkdl` need to be generated by run
+#     $docker build -t databricks/sparkdl --build-arg PYTHON_VERSION=2.7 .
+# on base directory. Currently we need to build it with Python2.
+
+FROM databricks/sparkdl
+
+RUN apt-get update && \
+    apt-get install -y ruby ruby-dev && \
+    apt-get clean
+
+RUN gem install jekyll bundler
+RUN gem install jekyll-redirect-from pygments.rb
+
+WORKDIR /mnt/sparkdl
+
+# redefine the entrypoint to empty so that we can run the shell command with `docker run`.
+ENTRYPOINT []

docs/README.md

@@ -7,8 +7,6 @@ Read on to learn more about viewing documentation in plain text (i.e., markdown)
 documentation yourself. Why build it yourself? So that you have the docs that correspond to
 whichever version of Deep Learning Pipelines you currently have checked out of revision control.
 
-## Generating the Documentation HTML
-
 We include the Deep Learning Pipelines documentation as part of the source (as opposed to using a hosted wiki, such as
 the github wiki, as the definitive documentation) to enable the documentation to evolve along with
 the source code and be captured by revision control (currently git). This way the code automatically
@@ -18,23 +16,14 @@ you have checked out or downloaded.
 In this directory you will find textfiles formatted using Markdown, with an ".md" suffix. You can
 read those text files directly if you want. Start with index.md.
 
-The markdown code can be compiled to HTML using the [Jekyll tool](http://jekyllrb.com).
-`Jekyll` and a few dependencies must be installed for this to work. We recommend
-installing via the Ruby Gem dependency manager. Since the exact HTML output
-varies between versions of Jekyll and its dependencies, we list specific versions here
-in some cases (`Jekyll 3.4.3`):
-
-    $ sudo gem install jekyll bundler
-    $ sudo gem install jekyll-redirect-from pygments.rb
-
+## Generating the Documentation HTML
+To generate the documentation HTML, you can run the script from the base directory:
 
-Then run the prepare script to setup prerequisites and generate a wrapper "jekyll" script
-	$ ./prepare -s <path_to_spark_home> -t <path_to_tensorframes_home>
+    $ dev/build-docs.sh
 
-Execute `./jekyll build` from the `docs/` directory to compile the site. Compiling the site with Jekyll will create a directory
-called `_site` containing index.html as well as the rest of the compiled files.
+It compiles the site with Jekyll will create a directory called `_site` containing index.html as well as the rest of the compiled files.
 
-You can modify the default Jekyll build as follows:
+You can modify the default Jekyll build in the docs docker container as follows:
 
     # Skip generating API docs (which takes a while)
     $ SKIP_API=1 ./jekyll build
@@ -43,13 +32,9 @@ You can modify the default Jekyll build as follows:
     # Build the site with extra features used on the live page
     $ PRODUCTION=1 ./jekyll build
 
-Note that `SPARK_HOME` must be set to your local Spark installation in order to generate the docs.
-
 ## Pygments
 
-We also use pygments (http://pygments.org) for syntax highlighting in documentation markdown pages,
-so you will also need to install that (it requires Python) by running `sudo pip install Pygments`.
-
+We also use pygments (http://pygments.org) for syntax highlighting in documentation markdown pages.
 To mark a block of code in your markdown to be syntax highlighted by jekyll during the compile
 phase, use the following sytax:
 
@@ -60,8 +45,7 @@ phase, use the following sytax:
 
 ## Sphinx
 
-We use Sphinx to generate Python API docs, so you will need to install it by running
-`sudo pip install sphinx`.
+We use Sphinx to generate Python API docs.
 
 ## API Docs (Scaladoc, Sphinx)
 

docs/_config.yml

@@ -13,10 +13,9 @@ include:
   - _modules
 
 # These allow the documentation to be updated with newer releases
-# of Spark, Scala, and Mesos.
-SPARKDL_VERSION: 0.2.0
-#SCALA_BINARY_VERSION: "2.10"
-#SCALA_VERSION: "2.10.4"
-#MESOS_VERSION: 0.21.0
+# of Spark and Scala.
+SPARKDL_VERSION: 1.2.0
+#SCALA_BINARY_VERSION: "2.11"
+#SCALA_VERSION: "2.11.8"
 #SPARK_ISSUE_TRACKER_URL: https://issues.apache.org/jira/browse/SPARK
 #SPARK_GITHUB_URL: https://github.com/apache/spark

docs/prepare

@@ -334,3 +334,6 @@
 
 # If false, no index is generated.
 #epub_use_index = True
+
+# Merge __init__ docstring into class doc.
+autoclass_content = 'both'

python/docs/conf.py

@@ -15,19 +15,22 @@ Contents:
 .. toctree::
    :maxdepth: 2
 
-   sparkdl
+    sparkdl.graph
+    sparkdl.image
+    sparkdl.transformers
+    sparkdl.udf
+    sparkdl.utils
 
-Core classes:
--------------
+Module contents
+---------------
 
-    :class:`sparkdl.OurCoolClass`
-
-    Description of OurCoolClass
+.. automodule:: sparkdl
+    :members:
+    :undoc-members:
+    :show-inheritance:
 
 
 Indices and tables
 ====================================================================================
 
-* :ref:`genindex`
-* :ref:`modindex`
 * :ref:`search`

python/docs/index.rst

@@ -0,0 +1,18 @@
+sparkdl.graph module
+====================
+
+sparkdl.graph.input module
+--------------------------
+
+.. automodule:: sparkdl.graph.input
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+sparkdl.graph.utils module
+--------------------------
+
+.. automodule:: sparkdl.graph.utils
+    :members:
+    :undoc-members:
+    :show-inheritance:

python/docs/sparkdl.graph.rst

@@ -0,0 +1,10 @@
+sparkdl.image module
+====================
+
+sparkdl.image.imageIO module
+----------------------------
+
+.. automodule:: sparkdl.image.imageIO
+    :members:
+    :undoc-members:
+    :show-inheritance: