Add docs detailing our ASF Jenkins usage (#2767)

Hopefully these docs will answer a lot of questions I had to dig up
answers to on my own following a recent outage of our ASF Jenkins
agents.

Author: gerlowskija

dev-docs/asf-jenkins.adoc

@@ -0,0 +1,72 @@
+= ASF Jenkins Setup
+:toc: left
+
+The Solr project uses a Jenkins instance provided by the Apache Software Foundation ("ASF") for running tests, validation, etc.
+
+This file aims to document our [ASF Jenkins](https://ci-builds.apache.org/job/Solr/) usage and administration, to prevent it from becoming "tribal knowledge" understood by just a few.
+
+== Jobs
+
+We run a number of jobs on Jenkins, each validating an overlapping set of concerns:
+
+* `Solr-Artifacts-*` - daily jobs that run `./gradlew assemble` to ensure that build artifacts (except docker images) can be created successfully
+* `Solr-check-*` - "hourly" jobs that run all project tests and static analysis (i.e. `test`, `integrationTest`, and `check`)
+* `Solr-Docker-Nightly-*` - daily jobs that `./gradlew testDocker dockerPush` to validate docker image packaging.  Snapshot images are pushed to hub.docker.com
+* `Solr-reference-guide-*` - hourly jobs that build the Solr reference guide via `./gradlew checkSite` and push the resulting artifact to the staging/preview site `nightlies.apache.org`
+* `Solr-Smoketest-*` - daily jobs that produce a snapshot release (via the `assembleRelease` task) and run the release smoketester
+
+Most jobs that validate particular build artifacts are run "daily", which is sufficient to prevent any large breaks from creeping into the build.
+
+On the other hand, jobs that run tests are triggered "hourly" in order to squeeze as many test runs as possible out of our Jenkins hardware.
+This is a necessary consequence of Solr's heavy use of randomization in its test-suite.
+"Hourly" scheduling ensures that a test run is either currently running or in the build queue at all times, and enables us to get the maximum data points from our hardware.
+
+== Jenkins Agents
+
+All Solr jobs run on Jenkins agents marked with the 'solr' label.
+Currently, this maps to two Jenkins agents:
+
+* `lucene-solr-1` - available at lucene1-us-west.apache.org
+* `lucene-solr-2` - available (confusingly) at lucene-us-west.apache.org
+
+These agents are "project-specific" VMs shared by the Lucene and Solr projects.
+That is: they are VMs requested by a project for their exclusive use.
+(INFRA policy appears to be that each Apache project may request 1 dedicated VM; it's unclear how Solr ended up with 2.)
+
+Maintenance of these agent VMs falls into a bit of a gray area.
+INFRA will still intervene when asked: to reboot nodes, to deploy OS upgrades, etc.
+But some burden also falls on Lucene and Solr as project teams to monitor the the VMs and keep them healthy.
+
+=== Accessing Jenkins Agents
+
+With a few steps, Solr committers can access our project's Jenkins agent VMs via SSH to troubleshoot and resolve issues.
+
+1. Ensure your account on id.apache.org has an SSH key associated with it.
+2. Ask INFRA to give your Apache ID SSH access to these boxes. (See [this JIRA ticket](https://issues.apache.org/jira/browse/INFRA-3682) for an example.)
+3. SSH into the desired box with: `ssh <apache-id>@$HOSTNAME` (where `$HOSTNAME` is either `lucene1-us-west.apache.org` or `lucene-us-west.apache.org`)
+
+Often, SSH access on the boxes is not sufficient, and administrators require "root" access to diagnose and solve problems.
+Sudo/su priveleges can be accessed via a one-time pad ("OTP") challenge, managed by the "Orthrus PAM" module.
+Users in need of root access can perform the following steps:
+
+1. Open the ASF's [OTP Generator Tool](https://selfserve.apache.org/otp-calculator.html) in your browser of choice
+2. Run `ortpasswd` on the machine.  This will print out a OTP "challenge" (e.g. `otp-md5 497 lu6126`) and provide a password prompt.  This password prompt should be given a OTP password, generated in steps 3-5 below.
+3. Copy the "challenge" from the previous step into the relevant field on the "OTP Generator Tool" form.
+4. Choose a password to use for OTP Challenges (or recall one you've used in the past), and type this into the relevant field on the "OTP Generator Tool" form.
+5. Click "Compute", and copy the first line from the "Response" box into your SSH session's password prompt.  You're now established in the "Orthrus PAM" system.
+6. Run a command requesting `su` escalation (e.g. `sudo su -`).  This should print another "challenge" and password prompt.  Repeat steps 3-5.
+
+If this fails at any point, open a ticket with INFRA.
+You may need to be added to the 'sudoers' file for the VM(s) in question.
+
+=== Known Jenkins Issues
+
+One recurring problem with the Jenkins agents is that they periodically run out of disk-space.
+Usually this happens when enough "workspaces" are orphaned or left behind, consuming all of the agent's disk space.
+
+Solr Jenkins jobs are currently configured to clean up the previous workspace at the *start* of the subsequent run.
+This avoids orphans in the common case but leaves workspaces behind any time a job is renamed or deleted (as happens during the Solr release process).
+
+Luckily, this has an easy fix: SSH into the agent VM and delete any workspaces no longer needed in `/home/jenkins/jenkins-slave/workspace/Solr`.
+Any workspace that doesn't correspond to a [currently existing job](https://ci-builds.apache.org/job/Solr/) can be safely deleted.
+(It may also be worth comparing the Lucene workspaces in `/home/jenkins/jenkins-slave/workspace/Lucene` to [that project's list of jobs](https://ci-builds.apache.org/job/Lucene/).)