[300] Update section on running the server

Author: korydraughn

docs/getting_started/running.md

@@ -1,63 +1,116 @@
 #
 
-The iRODS Server can be run in one of three modes:
+## Starting the Server
 
-- [Syslog Mode](#syslog-mode)
-- [Standard Output Mode](#standard-output-mode)
-- [Test Mode](#test-mode)
+The iRODS server can be started in several ways, but the most common form is as a daemon process. This is accomplished by running the following as the service account user.
 
-## Syslog Mode
-This mode is the default when operating the iRODS Server.
+```bash
+irodsServer -d
+```
+
+The server will perform various checks during startup to make sure it is safe to accept client requests. The most important check is validation of `server_config.json`. Failing validation will result in the server terminating immediately. All validation errors are reported via stderr.
+
+Logs are written to syslog. See [The Server Log](../../system_overview/troubleshooting/#the-server-log) for information regarding syslog.
+
+### Server PID File
+
+By default, the server writes its process ID (PID) to `/var/run/irods/irods-server.pid`.
+
+Use the `--pid-file` option to specify a different location. This is useful when running multiple iRODS servers on the same host.
+
+## Stopping the Server
 
-This mode instructs the server and all of its children to write log messages to **syslog**.
+iRODS supports two modes of shutdown.
 
-syslog can be configured to write messages locally, remotely, or both.
+### Fast Shutdown
+
+A fast shutdown is triggered by sending a `SIGTERM` or `SIGINT` to the main server process. For example:
 
 ```bash
-$ ./irodsctl start
+kill -TERM $(cat /var/run/irods/irods-server.pid)
 ```
 
-Messages can be rotated on a regular basis, most commonly through the **logrotate** service.
+This signal instructs agents to complete their current client request and terminate immediately (and cleanly), ending communication with the client.
+
+### Graceful Shutdown
 
-## Standard Output Mode
-This mode instructs the server and all of its children to write log messages to **stdout** instead of **syslog**. This mode can be enabled in two ways.
+A graceful shutdown is triggered by sending a `SIGQUIT` to the main server process. For example:
 
-### Option 1 (recommended) - Use the control script in the iRODS service account
 ```bash
-$ ./irodsctl start --stdout
+kill -QUIT $(cat /var/run/irods/irods-server.pid)
 ```
 
-### Option 2 - Launch the server binary directly
+This signal instructs agents to wait a limited amount of time before entering the [fast shutdown](#fast-shutdown) phase. During a graceful shutdown, agents will continue servicing client requests until the timeout. The timeout can be modified via the `graceful_shutdown_timeout_in_seconds` configuration property in `server_config.json`. See the section about [configuration](../../system_overview/configuration/#etcirodsserver_configjson) to learn more.
+
+## Reloading Server Configuration
+
+Unlike iRODS 4, iRODS 5 does not automatically load changes to `server_config.json`. Reloading the server's configuration requires sending a `SIGHUP` to the server. For example:
+
 ```bash
-$ /usr/sbin/irodsServer -u
+kill -HUP $(cat /var/run/irods/irods-server.pid)
 ```
 
-Regardless of the method, it is important to remember that once the server has been launched in this mode, the terminal will not return control until the server is shutdown or killed.
+On receiving `SIGHUP`, the server:
+
+1. Validates `server_config.json` against its schema.
+2. If validation **fails**, the reload is **rejected** and the previous configuration is used.
+3. If validation **succeeds**, the server:
+
+    - Updates its in-memory configuration
+    - Sends `SIGTERM` to the delay server (if running)
+    - Sends `SIGQUIT` to the agent factory
+    - Waits for the agent factory to close its listening socket
+    - Launches a new agent factory
+
+## Managing iRODS via the Service Manager
+
+iRODS 5 has full compatibility with the service manager.
+
+Given that systemd is the most widely used service manager, iRODS provides a service file template which should prove helpful to users looking to manage their iRODS server via systemd.
+
+The service file template is located at `/var/lib/irods/packaging/irods.service.template`.
+
+!!! Note
+    As a best practice, prefer modifying a copy of the template files. Doing that will help with avoiding issues with upgrades. Template files should always be treated as readonly.
+
+Configuring systemd to use the service file is out of scope for this documentation and is left as an exercise for the reader. Below are a few links which you may find helpful.
+
+- <https://systemd.io>
+- <https://www.freedesktop.org/software/systemd/man/latest/systemd.html>
 
 ## Test Mode
+
+!!! Note
+    This section is primarily for developers working on the server, plugins, and/or clients.
+
 This mode instructs the server to write all log messages to an additional log file. Messages will still be written to stdout or syslog depending on how the server was launched. This secondary log file is located at the following location:
+
 ```bash
 /var/lib/irods/log/test_mode_output.log
 ```
 
-This mode was introduced to guarantee that log messages appear in the log file immediately. This mode is only meant to be used for testing. The log file produced by this mode is never rotated.
+This mode was introduced to guarantee that log messages appear in the log file immediately. This mode is only meant to be used for testing. **The log file produced by this mode is never rotated.**
 
 To enable this mode, you must first define the following environment variable in the iRODS service account:
+
 ```bash
 export IRODS_ENABLE_TEST_MODE=1
 ```
-This environment variable instructs the IrodsController python class to start the server in test mode.
+
+This environment variable makes various components (e.g. `IrodsController`) aware the server is to be operated in test mode. This is very important for tests which restart the server.
 
 **If you are using the run_tests.py script to run tests, then you can ignore the rest of this section because run_tests.py restarts the server in test mode automatically.**
 
-Next, you'll want to start (or restart) the server in test mode. This can be done using either of the following methods.
+Next, you'll want to start (or restart) the server in test mode.
 
-### Option 1 (recommended) - Use the control script in the iRODS service account
 ```bash
-$ ./irodsctl start --test
+irodsServer -t
 ```
 
-### Option 2 - Launch the server binary directly
+## Additional Information
+
+For more options and information, see the server's built-in help:
+
 ```bash
-$ /usr/sbin/irodsServer -t
+irodsServer --help
 ```