ci(gcb): misc cleanups (#6281)

These are a couple small cleanups and documentation improvements that I
had lying around.

Author: devjgm

ci/cloudbuild/README.md

@@ -2,18 +2,69 @@
 
 This directory contains the files needed for our GCB presubmit ("PR builds")
 and postsubmit ("CI builds") builds. The `cloudbuild.yaml` file is the main
-config file for cloud build. Cloud builds may be managed from the command line
-with the `gcloud builds` command. To make this process easier, the `build.sh`
-script can be used to submit builds to GCB, run builds locally in a docker
-container, or even run builds directly in your local environment. See `build.sh
---help` for more details.
+config file for Google Cloud Build (GCB). Cloud builds may be managed from the
+command line with the `gcloud builds` command. To make this process easier, the
+`build.sh` script can be used to submit builds to GCB, run builds locally in a
+docker container, or even run builds directly in your local environment. See
+`build.sh --help` for more details.
+
+Build scripts live in the `builds/` directory. We want to keep these scripts as
+simple as possible. Ideally they will be just a simple listing of the commands
+that a human would type with minimal logic. Many of these build scripts should
+be only a few lines. A little duplication is even OK if the resulting build
+script is simpler. Sometimes there are non-trivial things we need to do
+repeatedly (e.g., run the quickstart builds), and these live in `builds/lib/`.
+
+Each build runs in an environment defined by a Dockerfile in the `dockerfiles/`
+directory. Some Dockerfiles are used by multiple builds, some only one. Build
+scripts and Dockerfiles are associated in a trigger file.
 
 GCB triggers can be configured in the http://console.cloud.google.com/ UI, but
-we prefer to configure them with SCM controlled YAML files that live in the
+we prefer to configure them with version controlled YAML files that live in the
 `triggers/` directory. Managing triggers can be done with the `gcloud` command
 also, but we have the local `trigger.sh` script to make this process a bit
 easier. See `trigger.sh --help` for more details.
 
+Adding a new build is pretty simple, and can be done in a single PR. For
+example, see https://github.com/googleapis/google-cloud-cpp/pull/6252, which
+adds the `cxx17-pr` and `cxx17-ci` builds. The steps to add a new build are:
+
+1. (Optional) If your build needs to run in a unique environment, you should
+   start by creating a new Dockerfile in `dockerfiles/`. If you don't need a
+   specific environment, we typically use the `dockerfiles/fedora.Dockerfile`
+   image, which is simply called the "fedora" distro. Here's an [example
+   PR](https://github.com/googleapis/google-cloud-cpp/pull/6259) that adds a
+   build with its own Dockerfile.
+2. Create the new build script in `builds/` that runs the commands you want.
+   * You can test this script right away with `build.sh` by explicitly
+     specifying the `--distro` you want your build to run in.  For example:
+   ```
+   $ build.sh --distro fedora my-new-build # or ...
+   $ build.sh --distro fedora my-new-build --docker
+   ```
+3. Create your trigger file(s) in the `triggers/` directory. If you want both
+   PR (presubmit) and CI (postsubmit) builds you can generate the trigger files
+   with the command `trigger.sh --generate my-new-build`, which will write the
+   new files in the `triggers/` directory. You may need to tweak the files at
+   this point, for example to change the distro (fedora is the default).
+4. At this point, you're pretty much done. You can now test your build using
+   the trigger name as shown here:
+   ```
+   $ build.sh -t my-new-build-pr # or ...
+   $ build.sh -t my-new-build-pr --docker # or ...
+   $ build.sh -t my-new-build-pr --project cloud-cpp-testing-resources
+   ```
+5. Send a PR! Google Cloud Build will not know about your new trigger files yet
+   so they will not be run for your PR. This is working as intended. Get the PR
+   reviewed and merge it.
+6. FINAL STEP: Now that the code for your new build is checked in, tell GCB
+   about the triggers so the can be run automatically for all future PRs and
+   merges.
+   ```
+   $ trigger.sh --import triggers/my-new-build-pr.yaml
+   $ trigger.sh --import triggers/my-new-build-ci.yaml
+   ```
+
 ## Testing principles
 
 We want our code to work for our customers. We don't know their exact
@@ -45,9 +96,3 @@ while achieving high likelihood of the code working for our customers.
 * Sanitizer builds need to run integration tests
 * We want to test our user-facing instructions (i.e., how to install and use)
   as much as possible (this is difficult on macOS and Windows without docker)
-
-In addition to the guiding principles above, we'd like to keep the build
-scripts in `builds/` as simple as possible; ideally just a simple listing of
-the commands that a human would type with minimal logic. Many of these build
-scripts should be only a few lines. A little duplication is even OK if the
-resulting build script is simpler.

ci/cloudbuild/build.sh

@@ -220,7 +220,7 @@ if [[ "${DOCKER_FLAG}" = "true" ]]; then
   io::log_h2 "Building docker image: ${image}"
   docker build -t "${image}" "--build-arg=NCPU=$(nproc)" \
     -f "ci/cloudbuild/dockerfiles/${DISTRO_FLAG}.Dockerfile" ci
-  io::log_h2 "Starting container for ${image} running ${BUILD_NAME}"
+  io::log_h2 "Starting docker container: ${image}"
   run_flags=(
     "--interactive"
     "--tty"
@@ -242,8 +242,9 @@ if [[ "${DOCKER_FLAG}" = "true" ]]; then
   )
   cmd=(ci/cloudbuild/build.sh --local "${BUILD_NAME}")
   if [[ "${SHELL_FLAG}" = "true" ]]; then
-    io::log "Starting shell, to manually run the requested build use:"
-    echo "==> ${cmd[*]}"
+    printf "To run the build manually:\n  "
+    printf " %q" "${cmd[@]}"
+    printf "\n\n"
     cmd=("bash")
   fi
   docker run "${run_flags[@]}" "${image}" "${cmd[@]}"
@@ -264,16 +265,17 @@ fi
 # Uses Google Cloud build to run the specified build.
 io::log_h1 "Starting cloud build: ${BUILD_NAME}"
 account="$(gcloud config list account --format "value(core.account)")"
-subs="_DISTRO=${DISTRO_FLAG}"
-subs+=",_BUILD_NAME=${BUILD_NAME}"
-subs+=",_CACHE_TYPE=manual-${account}"
-subs+=",_PR_NUMBER=" # Must be empty or a number, and this is not a PR
-subs+=",BRANCH_NAME=${BRANCH_NAME}"
-subs+=",COMMIT_SHA=${COMMIT_SHA}"
-io::log "Substitutions ${subs}"
+subs=("_DISTRO=${DISTRO_FLAG}")
+subs+=("_BUILD_NAME=${BUILD_NAME}")
+subs+=("_CACHE_TYPE=manual-${account}")
+subs+=("_PR_NUMBER=") # Must be empty or a number, and this is not a PR
+subs+=("BRANCH_NAME=${BRANCH_NAME}")
+subs+=("COMMIT_SHA=${COMMIT_SHA}")
+printf "Substitutions:\n"
+printf "  %s\n" "${subs[@]}"
 args=(
   "--config=ci/cloudbuild/cloudbuild.yaml"
-  "--substitutions=${subs}"
+  "--substitutions=$(printf "%s," "${subs[@]}")"
 )
 if [[ -n "${PROJECT_FLAG}" ]]; then
   args+=("--project=${PROJECT_FLAG}")