Improve local Dockerfile (and remove its HTTP server)

This changes the Docker-mediated build (i.e. build.sh --docker) to behave the same as the non-Docker-mediated version. In particular, it:

* Operates on local input/cache/output directories, instead of copying the input into the container every build, and managing cache and output entirely within the container
* No longer creates and exposes a HTTP server, since the output is written directly to the disk (outside the container)
* Respects all command-line options, including --no-update

Other improvements:

* Changes the strategy for clearing directories, since deleting the directory was causing some problems in my local testing
* Slims down the size of the Docker container significantly, by using debian:stable-slim, copying less of html-build in, and no longer copying the source directory in repeatedly
* Does not bother checking for local Wattsi/highlighter if we're going the Docker route

Author: domenic

.dockerignore

@@ -1,5 +1,6 @@
-.git/
-.cache/
-.temp/
-output/
-html/.git/
+*
+!entities/out
+!quotes/out
+!*.pl
+!build.sh
+!lint.sh

Dockerfile

@@ -1,35 +1,13 @@
-FROM debian:stable
-
-## dependency installation: nginx and other build tools
+FROM debian:stable-slim
 RUN apt-get update && \
-    apt-get install -y ca-certificates curl git unzip nginx python3 python3-pip && \
-    rm -rf /etc/nginx/sites-enabled/* && \
+    apt-get install -y ca-certificates curl git unzip python3 python3-pip && \
     rm -rf /var/lib/apt/lists/*
 
 COPY --from=whatwg/wattsi:latest /whatwg/wattsi/bin/wattsi /bin/wattsi
 
-ADD . /whatwg/build
-
 RUN pip3 install bs-highlighter
 
-ARG html_source_dir
-ADD $html_source_dir /whatwg/html
-ENV HTML_SOURCE /whatwg/html
-
-WORKDIR /whatwg/build
-
-## build and copy assets to final nginx dir
-
-ARG verbose_or_quiet_flag
-ARG no_update_flag
-ARG sha_override
-
-# no_update_flag doesn't really work; .cache directory is re-created empty each time
-RUN SKIP_BUILD_UPDATE_CHECK=true SHA_OVERRIDE=$sha_override \
-    ./build.sh $verbose_or_quiet_flag $no_update_flag && \
-    rm -rf /var/www/html && \
-    mv output /var/www/html && \
-    chmod -R o+rX /var/www/html && \
-    cp site.conf /etc/nginx/sites-enabled/
+COPY . /whatwg/html-build/
 
-CMD ["nginx", "-g", "daemon off;"]
+ENV SKIP_BUILD_UPDATE_CHECK true
+ENTRYPOINT ["bash", "/whatwg/html-build/build.sh"]

README.md

@@ -34,28 +34,9 @@ Run the `build.sh` script from inside your `html-build` working directory, like
 
 The first time this runs, it will ask for your input on where to clone the HTML source from, or where on your system to find it if you've already done that. If you're working to submit a pull request to [whatwg/html](https://github.com/whatwg/html), be sure to give it the URL of your fork.
 
-### Output
-
-After you complete the build steps above, the build will run and generate the single-page version of the spec, the multipage version, and more. If all goes well, you should very soon have all the following in your `output/` directory:
-
-- `404.html`
-- `demos/*`
-- `dev/*`
-- `entities.json`
-- `fonts/*`
-- `html-dfn.js`
-- `images/*`
-- `index.html`
-- `link-fixup.js`
-- `multipage/*`
-- `robots.txt`
-- `xrefs.json`
-
-Now you're ready to edit the `html/source` file—and after you make your changes, you can run the `build.sh` script again to see the new output.
-
 ## Building using a Docker container
 
-The Dockerized version of the build allows you to run the build entirely inside a "container" (lightweight virtual machine). This includes tricky dependencies like a local copy of Wattsi, as well an HTTP server setup similar to that of https://html.spec.whatwg.org.
+The Dockerized version of the build allows you to run the build entirely inside a "container" (lightweight virtual machine). This includes tricky dependencies like a local copy of Wattsi and Python.
 
 To perform a Dockerized build, use the `--docker` flag:
 
@@ -65,9 +46,13 @@ To perform a Dockerized build, use the `--docker` flag:
 
 The first time you do this, Docker will download a bunch of stuff to set up the container properly, but subsequent runs will simply build the standard and be very fast.
 
-After building the standard, this will launch a HTTP server that allows you to view the result at `http://localhost:8080`. (OS X and Windows users will need to use the IP address of their docker-machine VM instead of `localhost`. You can get this with the `docker-machine env` command.)
+If you get permissions errors on Windows, you need to first [configure](https://docs.docker.com/docker-for-windows/#file-sharing) your `html-build/` and `html/` directories to be shareable with Docker.
 
-Note that due to the way Docker works, the HTML source repository must be contained in a subdirectory of the `html-build` working directory. This will happen automatically if you let `build.sh` clone for you, but if you have a preexisting clone you'll need to move it.
+## Output
+
+After you complete the build steps above, the build will run and generate the single-page version of the spec, the multipage version, and more. If all goes well, you should very soon have an `output/` directory containing important files like `index.html`, `multipage/`, and `dev/`.
+
+Now you're ready to edit the `html/source` file—and after you make your changes, you can run the `build.sh` script again to see the new output.
 
 ## A note on Git history
 

build.sh

@@ -36,14 +36,14 @@ export HTML_TEMP
 # Used specifically when the Dockerfile calls this script
 SKIP_BUILD_UPDATE_CHECK=${SKIP_BUILD_UPDATE_CHECK:-false}
 SHA_OVERRIDE=${SHA_OVERRIDE:-}
-HIGHLIGHT_SERVER_URL="http://127.0.0.1:8080" # this needs to be coordinated with the bs-highlighter package
+BUILD_SHA_OVERRIDE=${BUILD_SHA_OVERRIDE:-}
+
+# This needs to be coordinated with the bs-highlighter package
+HIGHLIGHT_SERVER_URL="http://127.0.0.1:8080"
 
 function main {
   processCommandLineArgs "$@"
 
-  checkWattsi
-  ensureHighlighterInstalled
-
   # $SKIP_BUILD_UPDATE_CHECK is set inside the Dockerfile so that we don't check for updates both inside and outside
   # the Docker container.
   if [[ $DO_UPDATE == "true" && $SKIP_BUILD_UPDATE_CHECK != "true" ]]; then
@@ -52,14 +52,22 @@ function main {
 
   findHTMLSource
 
-  HTML_GIT_DIR="$HTML_SOURCE/.git/"
-  HTML_SHA=${SHA_OVERRIDE:-$(git --git-dir="$HTML_GIT_DIR" rev-parse HEAD)}
+  clearDir "$HTML_OUTPUT"
+  # Set these up so rsync will not complain about either being missing
+  mkdir -p "$HTML_OUTPUT/commit-snapshots"
+  mkdir -p "$HTML_OUTPUT/review-drafts"
 
   if [[ $USE_DOCKER == "true" ]]; then
     doDockerBuild
     exit 0
   fi
 
+  checkWattsi
+  ensureHighlighterInstalled
+
+  HTML_GIT_DIR="$HTML_SOURCE/.git/"
+  HTML_SHA=${SHA_OVERRIDE:-$(git --git-dir="$HTML_GIT_DIR" rev-parse HEAD)}
+
   $QUIET || echo "Linting the source file..."
   ./lint.sh "$HTML_SOURCE/source" || {
     echo
@@ -71,11 +79,6 @@ function main {
 
   updateRemoteDataFiles
 
-  rm -rf "$HTML_OUTPUT" && mkdir -p "$HTML_OUTPUT"
-  # Set these up so rsync will not complain about either being missing
-  mkdir -p "$HTML_OUTPUT/commit-snapshots"
-  mkdir -p "$HTML_OUTPUT/review-drafts"
-
   startHighlightServer
 
   processSource "source" "default"
@@ -110,7 +113,7 @@ function processCommandLineArgs {
   do
     case $arg in
       clean)
-        rm -rf "$HTML_CACHE"
+        clearDir "$HTML_CACHE"
         exit 0
         ;;
       help)
@@ -120,7 +123,7 @@ function processCommandLineArgs {
         echo "  $0 help   Show this usage statement."
         echo
         echo "Build options:"
-        echo "  -d|--docker     Use Docker to build in and serve from a container."
+        echo "  -d|--docker     Use Docker to build in a container."
         echo "  -n|--no-update  Don't update before building; just build."
         echo "  -q|--quiet      Don't emit any messages except errors/warnings."
         echo "  -v|--verbose    Show verbose output from every build step."
@@ -182,9 +185,8 @@ function checkHTMLBuildIsUpToDate {
 # - Output:
 #   - Either bs-highlighter-server will be in the $PATH, or a warning will be echoed
 function ensureHighlighterInstalled {
-  # If we're using Docker then this will be installed inside the container.
   # If we're not using local Wattsi then we won't use the local highlighter.
-  if [[ $USE_DOCKER != "true" && $LOCAL_WATTSI == "true" ]]; then
+  if [[ $LOCAL_WATTSI == "true" ]]; then
     if hash pip3 2>/dev/null; then
       if ! hash bs-highlighter-server 2>/dev/null; then
         pip3 install bs-highlighter
@@ -386,33 +388,23 @@ function relativePath {
 # Arguments: none
 # Output: A web server with the build output will be running inside the Docker container
 function doDockerBuild {
-  if [[ $HTML_SOURCE != $(pwd)/* ]]; then
-    echo "When using Docker, the HTML source must be checked out in a subdirectory of the html-build repo. Cannot continue."
-    exit 1
-  fi
-
-  # $SOURCE_RELATIVE helps on Windows with Git Bash, where /c/... is a symlink, which Docker doesn't like.
-  SOURCE_RELATIVE=$(relativePath "$(pwd)" "$HTML_SOURCE")
-
-  VERBOSE_OR_QUIET_FLAG=""
-  $QUIET && VERBOSE_OR_QUIET_FLAG+="--quiet"
-  $VERBOSE && VERBOSE_OR_QUIET_FLAG+="--verbose"
-
-  NO_UPDATE_FLAG="--no-update"
-  $DO_UPDATE && NO_UPDATE_FLAG=""
-
-  DOCKER_ARGS=( --tag whatwg-html \
-                --build-arg "html_source_dir=$SOURCE_RELATIVE" \
-                --build-arg "verbose_or_quiet_flag=$VERBOSE_OR_QUIET_FLAG" \
-                --build-arg "no_update_flag=$NO_UPDATE_FLAG" \
-                --build-arg "sha_override=$HTML_SHA" )
-  if $QUIET; then
-    DOCKER_ARGS+=( --quiet )
-  fi
-
-  docker build "${DOCKER_ARGS[@]}" .
-  echo "Running server on http://localhost:8080"
-  docker run --rm -it -p 8080:80 whatwg-html
+  DOCKER_BUILD_ARGS=( --tag whatwg-html )
+  $QUIET && DOCKER_BUILD_ARGS+=( --quiet )
+
+  docker build "${DOCKER_BUILD_ARGS[@]}" .
+
+  DOCKER_RUN_ARGS=( whatwg-html )
+  $QUIET && DOCKER_RUN_ARGS+=( --quiet )
+  $VERBOSE && DOCKER_RUN_ARGS+=( --verbose )
+  $DO_UPDATE || DOCKER_RUN_ARGS+=( --no-update )
+
+  # Pass in the html-build SHA (since there's no .git directory inside the container)
+  docker run --rm --interactive --tty \
+             --env "BUILD_SHA_OVERRIDE=$(git rev-parse HEAD)" \
+             --mount "type=bind,source=$HTML_SOURCE,destination=/whatwg/html-build/html,readonly=1" \
+             --mount "type=bind,source=$HTML_CACHE,destination=/whatwg/html-build/.cache" \
+             --mount "type=bind,source=$HTML_OUTPUT,destination=/whatwg/html-build/output" \
+             "${DOCKER_RUN_ARGS[@]}"
 }
 
 # Clears the $HTML_CACHE directory if the build tools have been updated since last run.
@@ -422,13 +414,12 @@ function doDockerBuild {
 function clearCacheIfNecessary {
   if [[ -d "$HTML_CACHE" ]]; then
     PREV_BUILD_SHA=$( cat "$HTML_CACHE/last-build-sha.txt" 2>/dev/null || echo )
-    CURRENT_BUILD_SHA=$( git rev-parse HEAD )
+    CURRENT_BUILD_SHA=${BUILD_SHA_OVERRIDE:-$(git rev-parse HEAD)}
 
     if [[ $PREV_BUILD_SHA != "$CURRENT_BUILD_SHA" ]]; then
       $QUIET || echo "Build tools have been updated since last run; clearing the cache..."
       DO_UPDATE=true
-      rm -rf "$HTML_CACHE"
-      mkdir -p "$HTML_CACHE"
+      clearDir "$HTML_CACHE"
       echo "$CURRENT_BUILD_SHA" > "$HTML_CACHE/last-build-sha.txt"
     fi
   else
@@ -477,7 +468,7 @@ function updateRemoteDataFiles {
 # - Output:
 #   - $HTML_OUTPUT will contain the built files
 function processSource {
-  rm -rf "$HTML_TEMP" && mkdir -p "$HTML_TEMP"
+  clearDir "$HTML_TEMP"
 
   $QUIET || echo "Pre-processing the source..."
   SOURCE_LOCATION="$1"
@@ -529,10 +520,9 @@ function processSource {
     cp -p "$HTML_TEMP/wattsi-output/xrefs.json" "$HTML_OUTPUT"
 
     # Multipage HTML and Dev Edition
-    rm -rf "$HTML_OUTPUT/multipage"
     mv "$HTML_TEMP/wattsi-output/multipage-html" "$HTML_OUTPUT/multipage"
     mv "$HTML_TEMP/wattsi-output/multipage-dev" "$HTML_OUTPUT/dev"
-    rm -rf "$HTML_TEMP"
+    clearDir "$HTML_TEMP"
 
     echo "User-agent: *
 Disallow: /commit-snapshots/
@@ -583,8 +573,7 @@ function checkWattsi {
 #   - $HTML_TEMP/wattsi-output directory will contain the output from Wattsi on success
 #   - $HTML_TEMP/wattsi-output.txt will contain the output from Wattsi, on both success and failure
 function runWattsi {
-  rm -rf "$2"
-  mkdir "$2"
+  clearDir "$2"
 
   WATTSI_ARGS=()
   if $QUIET; then
@@ -680,4 +669,17 @@ function stopHighlightServer {
   fi
 }
 
+# Ensures the given directory exists, but is empty
+# Arguments:
+# - $1: the directory to clear
+# Output: the directory will be empty (but guaranteed to exist)
+function clearDir {
+  # We use this implementation strategy, instead of `rm -rf`ing the directory, because deleting the
+  # directory itself can run into permissions issues, e.g. if the directory is open in another
+  # program, or in the Docker case where we have permission to write to the directory but not delete
+  # it.
+  mkdir -p "$1"
+  find "$1" -mindepth 1 -delete
+}
+
 main "$@"