Update documentation for Raspberry Pi provisioning

- Enhanced README with links to configuration reference documentation and added troubleshooting sections for device provisioning.
- Introduced detailed guidance for connecting Raspberry Pi 4, 5, and Zero 2 W devices, including GPIO configuration and troubleshooting tips.
- Added a new document for mass provisioning scaling guidance, outlining connection requirements, provisioning strategies, and performance optimization.
- Updated API documentation to clarify manufacturing database access and error response formats.

Author: tdewey-rpi

README.adoc

@@ -69,13 +69,16 @@ $ xdg-open http://localhost:3142
 ----
 
 At the minimum, you must consult the 'Options' and 'Images' pages, having uploaded your OS image and set the options as guided by the inline
-help text (which is generated from the reference documentation in docs/)
+help text (which is generated from the <<docs/config_vars.adoc,configuration reference documentation>>)
 
 Once you have followed all those steps, `rpi-sb-provisioner` should be correctly configured and ready to run.
 
+[#using-rpi-sb-provisioner]
 == Using rpi-sb-provisioner
 `rpi-sb-provisioner` is composed of three `systemd` services that are triggered by the connection of a device in RPIBOOT mode to a USB port. With `rpi-sb-provisioner` configured to your requirements, all that is therefore required is to connect your target Raspberry Pi device in RPIBOOT mode.
 
+NOTE: For non-compute module connection instructions, see: <<docs/device-guidance/pi4.adoc,Raspberry Pi 4>>, <<docs/device-guidance/pi5.adoc,Raspberry Pi 5>>, or <<docs/device-guidance/zero2.adoc,Zero 2 W>> guidance documents.
+
 For any Raspberry Pi Compute Module on the matching Raspberry Pi Compute Module IO Board, you can do this by using the single Jumper Wire to connect the `disable eMMC Boot` pins on the 12-pin header at the top of the board.
 
 [pdfwidth=90%]
@@ -114,13 +117,22 @@ After these steps have been completed, your device should display both the `acti
 
 No further intervention is required in the success case.
 
-WARNING: `rpi-sb-provisioner` will not, by default, block JTAG access. If you wish to make use of this facility, you _must_ use the `RPI_DEVICE_LOCK_JTAG` configuration option.
+WARNING: `rpi-sb-provisioner` will not, by default, block JTAG access. If you wish to make use of this facility, you _must_ use the <<docs/config_vars.adoc#rpi_device_lock_jtag,RPI_DEVICE_LOCK_JTAG>> configuration option.
 
-== Tips and Tricks
+[#troubleshooting]
+== Troubleshooting
 
 === Observing active provisioning operations
 
-As `rpi-sb-provisioner` is implemented using `systemd` services, you can use the typical `systemctl` commands to observe the services as they provision your device.
+The easiest way to observe active provisioning operations is through the **Web UI**:
+
+1. **Open your browser** and navigate to `http://localhost:3142`
+2. **Click the 'Services' tab** to see all provisioning services and their current status
+3. **View real-time updates** of device provisioning progress, including serial numbers and current phases
+
+==== Command Line Alternative
+
+As `rpi-sb-provisioner` is implemented using `systemd` services, you can also use the typical `systemctl` commands to observe the services as they provision your device.
 
 To see active provisioning operations, and the serial numbers of the devices involved, type into a Terminal window:
 
@@ -130,7 +142,16 @@ $ systemctl list-units rpi-sb-provisioner*
 
 === Observing logs
 
-Logs are stored on a per-device, per-phase basis, where logs for a given device are stored at `/var/log/rpi-sb-provisioner/<serial>/<phase>.log`.
+The most convenient way to observe device provisioning logs is through the **Web UI**:
+
+1. **Open your browser** and navigate to `http://localhost:3142`
+2. **Click the 'Services' tab** to see all provisioning services
+3. **Click on any service** to view its detailed logs with real-time updates
+4. **Monitor progress** as logs are automatically refreshed during provisioning
+
+==== Command Line Alternatives
+
+Logs are also stored on a per-device, per-phase basis, where logs for a given device are stored at `/var/log/rpi-sb-provisioner/<serial>/<phase>.log`.
 
 For example, to observe the progress of an individual device through a phase, you could use `tail`:
 
@@ -149,9 +170,16 @@ Where the `-f` flag will follow the logs as they are written, letting you observ
 
 === Processing the manufacturing database
 
-If you have enabled the manufacturing database (using RPI_SB_PROVISIONER_MANUFACTURING_DB), you can create a comma-separated value (CSV) file from the manufacturing database for use with other software.
+If you have enabled the manufacturing database (using <<docs/config_vars.adoc#rpi_sb_provisioner_manufacturing_db,RPI_SB_PROVISIONER_MANUFACTURING_DB>>), you can view and export the manufacturing data through the **Web UI**:
+
+1. **Open your browser** and navigate to `http://localhost:3142`
+2. **Navigate to the Manufacturing Database section** to view all provisioned devices
+3. **Review device information** including serial numbers, board types, MAC addresses, and provisioning status
+4. **Export as CSV** using the built-in export function to create a comma-separated values file for use with other software
+
+==== Command Line Alternative
 
-To do so, you must use the *sqlite3* program:
+You can also create a CSV file directly using the *sqlite3* program:
 
 ----
 $ sqlite3 ${RPI_SB_PROVISIONER_MANUFACTURING_DB} -cmd ".headers on" -cmd ".mode csv" -cmd ".output mfg_db.csv" "SELECT * FROM rpi_sb_provisioner;"

docs/api_endpoints.adoc

@@ -5,6 +5,7 @@
 
 This document describes the API endpoints available in the provisioner-service. These endpoints can be used for integration with other systems, building custom dashboards, or automating operations.
 
+[#manufacturing-database-api]
 == Manufacturing Database API
 
 The Manufacturing Database API provides access to device provisioning data collected during the provisioning process. This API allows for integration with other systems and custom UIs.
@@ -70,7 +71,7 @@ The endpoint returns a JSON array where each element represents a provisioned de
 |processor|Processor type (e.g., "BCM2712")
 |memory|Device memory size (e.g., "2GB")
 |manufacturer|Device manufacturer
-|secure|Whether secure boot is enabled ("yes" or "no")
+|secure|Whether a device has been provisioned with a device unique identity
 |provision_ts|Timestamp of when the device was provisioned
 |===
 
@@ -118,7 +119,7 @@ On error, the endpoint returns a JSON object with error details:
 - This endpoint can be used for building custom dashboards or integrating with other monitoring systems.
 - The data is ordered by provision timestamp in descending order (newest first).
 - For large datasets, it is recommended to use pagination to improve performance.
-- The database path is configured using the `RPI_SB_PROVISIONER_MANUFACTURING_DB` setting as described in the configuration documentation.
+- The database path is configured using the <<config_vars.adoc#rpi_sb_provisioner_manufacturing_db,RPI_SB_PROVISIONER_MANUFACTURING_DB>> setting as described in the configuration documentation.
 
 == Devices API
 
@@ -513,6 +514,7 @@ If accessing an unauthorized service:
 - Access is restricted to services with approved prefixes for security
 - This endpoint is optimized for polling and provides lighter responses than the HTML view
 
+[#error-response-format]
 == Error Response Format
 
 All API endpoints follow a standard error response format:

docs/config_vars.adoc

@@ -10,7 +10,7 @@ This document describes the configuration variables used in /etc/rpi-sb-provisio
 === PROVISIONING_STYLE
 *Mandatory, with a default*
 
-Select the provisioning style you wish to use. Supported values are `secure-boot`, `fde-only` and `naked`.
+Select the provisioning style you wish to use. Supported values are `secure-boot`, `fde-only` and `naked`. For details on what each provisioning style does, see the <<../README.adoc#using-rpi-sb-provisioner,main documentation>>.
 
 If `PROVISIONING_STYLE` is not specified, it defaults to `secure-boot`.
 
@@ -89,6 +89,7 @@ This setting is typically configured through the firmware selection web interfac
 
 WARNING: Ensure the specified firmware file exists and is compatible with your target device family before provisioning.
 
+[#rpi_device_lock_jtag]
 === RPI_DEVICE_LOCK_JTAG
 *Optional*
 
@@ -105,12 +106,13 @@ Raspberry Pi devices that use an EEPROM as part of their boot flow can configure
 
 Set to any value to enable EEPROM write protection.
 
+[#rpi_sb_provisioner_manufacturing_db]
 === RPI_SB_PROVISIONER_MANUFACTURING_DB
 *Optional*
 
 Store manufacturing data in a sqlite3 database. This will include the board serial, board revision, the boot ROM version, the MAC address of the ethernet port, any set hash of the customer signing key, the JTAG lock state, the board attributes and the advanced boot flags.
 
-You must not specify the path of a database stored on a network drive or similar storage, as this mechanism is only safe to use on a single provisioning system. For merging the output with multiple provisioning systems, consider "Processing the manufacturing database" later in this document.
+You must not specify the path of a database stored on a network drive or similar storage, as this mechanism is only safe to use on a single provisioning system. For merging the output with multiple provisioning systems, consider <<../README.adoc#_processing_the_manufacturing_database,Processing the manufacturing database>> in the main documentation.
 
 Set to the path of a file to contain a SQLite database stored on local storage. The WebUI will create this file if it does not exist.
 

docs/device-guidance/pi4.adoc

@@ -0,0 +1,100 @@
+= Raspberry Pi 4 Device Guidance
+
+== Connecting Raspberry Pi 4 to rpi-sb-provisioner
+
+=== Prerequisites - GPIO Configuration
+
+Unlike Raspberry Pi 5, the Raspberry Pi 4 does not have a built-in power button to force RPIBOOT mode. Before attempting to use rpi-sb-provisioner with a Raspberry Pi 4 device, you must first configure the nRPIBOOT GPIO pin.
+
+**Recommended GPIO**: GPIO 8
+
+=== Step 1 - Configure nRPIBOOT GPIO using recovery.bin
+
+Before provisioning, you must assign the nRPIBOOT function to a specific GPIO pin using recovery.bin:
+
+1. **Download and prepare recovery.bin** following the instructions at https://github.com/raspberrypi/usbboot/tree/master/secure-boot-recovery#step-2---select-the-nrpiboot-gpio[GitHub usbboot secure-boot-recovery documentation]
+2. **Configure GPIO 8** as the nRPIBOOT pin using the recovery.bin process
+3. **Verify the configuration** has been applied to the device EEPROM (see <<verifying-gpio-8-configuration,Verifying GPIO 8 Configuration>> below)
+
+This configuration step only needs to be performed **once per device** and will persist in the device EEPROM.
+
+=== Step 2 - Provisioning Connection
+
+To provision a Raspberry Pi 4 device after GPIO configuration:
+
+1. **Prepare a jumper wire or connection** to short GPIO 8 to ground
+2. **Connect GPIO 8 to ground** (GND pin) on the Raspberry Pi 4
+3. **While maintaining the GPIO 8 to ground connection**, plug the USB cable into the device to connect it to your provisioning machine
+4. **Keep the GPIO connection in place throughout the entire provisioning process**
+
+Unlike Raspberry Pi 5, **no re-connection procedure is required** - the device will proceed through all provisioning phases automatically while the GPIO remains connected to ground.
+
+=== Important Notes
+
+* **One-time setup**: The GPIO configuration using recovery.bin only needs to be done once per device
+* **GPIO 8 recommended**: We recommend using GPIO 8 for consistency, as referenced in the https://github.com/raspberrypi/usbboot/tree/master/secure-boot-recovery#step-2---select-the-nrpiboot-gpio[usbboot documentation]
+* **Maintain connection**: The GPIO 8 to ground connection must be maintained throughout the entire provisioning process
+* **No re-plug required**: Unlike Raspberry Pi 5, no disconnection and re-connection is needed
+* **Cable quality**: Ensure you are using a high-quality USB cable as specified in the main documentation
+* **Complete process**: Do not release the GPIO connection until provisioning is fully complete
+
+[#verifying-gpio-8-configuration]
+== Verifying GPIO 8 Configuration
+
+To verify that GPIO 8 has been successfully assigned as the nRPIBOOT pin:
+
+1. **Boot the device normally** (without shorting any GPIO pins)
+2. **Run the following command** to read the current EEPROM configuration:
++
+----
+$ sudo rpi-eeprom-config
+----
+
+3. **Look for the GPIO configuration** in the output. You should see a line similar to:
++
+----
+WAKE_ON_GPIO=1
+GPIO_EXPANDER_CFG=0x00000008
+----
++
+Or other GPIO-related configuration entries that indicate GPIO 8 has been configured for nRPIBOOT functionality.
+
+4. **Alternative verification**: You can also extract the full EEPROM image and inspect it:
++
+----
+$ sudo rpi-eeprom-config --out /tmp/current-eeprom.bin
+$ rpi-eeprom-config /tmp/current-eeprom.bin
+----
+
+If the GPIO configuration is not present or incorrect, you will need to repeat the recovery.bin process to properly configure GPIO 8.
+
+== Troubleshooting
+
+=== GPIO Configuration Issues
+
+If you encounter issues with the initial GPIO configuration:
+
+* **Follow official documentation**: Refer to the complete instructions at https://github.com/raspberrypi/usbboot/tree/master/secure-boot-recovery#step-2---select-the-nrpiboot-gpio[GitHub usbboot documentation]
+* **Verify EEPROM update**: Use the verification commands above (`sudo rpi-eeprom-config`) to confirm the GPIO configuration was successfully written to EEPROM
+* **Check for configuration entries**: Look for `WAKE_ON_GPIO=1` and `GPIO_EXPANDER_CFG` entries in the EEPROM output
+* **Repeat recovery.bin process**: If verification shows incorrect or missing GPIO configuration, repeat the recovery.bin configuration process
+* **Try alternative GPIO**: Consider using a different GPIO pin if GPIO 8 is not accessible on your setup
+
+=== Device not entering RPIBOOT mode
+
+If your Raspberry Pi 4 device does not enter RPIBOOT mode:
+
+* **Verify GPIO configuration**: Use the verification steps above to confirm GPIO 8 was successfully configured as nRPIBOOT
+* **Check physical connection**: Verify GPIO 8 is properly connected to a ground (GND) pin
+* **Try different GPIO**: If GPIO 8 doesn't work, try configuring and using a different GPIO pin
+* **Cable and port**: Try a shorter, higher quality USB cable and different USB port on your host machine
+* **Power supply**: Ensure the provisioning machine has sufficient power supply capability
+
+=== Device not proceeding through provisioning
+
+If the device does not proceed through the provisioning phases:
+
+* **Maintain GPIO connection**: Ensure GPIO 8 to ground connection is maintained throughout the entire process
+* **Check provisioning logs**: Review logs for any error messages
+* **Verify device recognition**: Confirm the device was initially recognized by the provisioning system
+* **Restart procedure**: Disconnect the device, ensure GPIO connection is secure, and try again 

docs/device-guidance/pi5.adoc

@@ -0,0 +1,48 @@
+= Raspberry Pi 5 Device Guidance
+
+== Connecting Raspberry Pi 5 to rpi-sb-provisioner
+
+=== Initial Connection - Entering RPIBOOT Mode
+
+To provision a Raspberry Pi 5 device, you must force it into RPIBOOT mode:
+
+1. **Hold the power button** on the Raspberry Pi 5 device
+2. **While continuing to hold the power button**, plug the USB-C cable into the device to connect it to your provisioning machine
+3. Keep holding the power button until the device is recognized by the provisioning system
+
+=== Re-connection Procedure
+
+After the initial connection, you will need to perform a re-connection procedure:
+
+1. **Monitor the device state** until you see the status "bootstrap-fastboot-initialisation-started"
+2. **Unplug the device** from the USB-C cable
+3. **Hold the power button** on the Raspberry Pi 5 device
+4. **While continuing to hold the power button**, re-plug the USB-C cable to reconnect the device
+5. Keep holding the power button until the device proceeds to the next phase
+
+=== Important Notes
+
+* Ensure you are using a high-quality USB-A to USB-C cable as specified in the main documentation
+* The power button must be held during both the initial connection and the re-connection procedure
+* Do not release the power button until you confirm the device has been recognized by the provisioning system
+* If the device fails to enter RPIBOOT mode, try using a different USB port on your host machine
+
+== Troubleshooting
+
+=== Device not entering RPIBOOT mode
+
+If your Raspberry Pi 5 device does not enter RPIBOOT mode:
+
+* Ensure you are holding the power button **before** connecting the USB cable
+* Try a shorter, higher quality USB-A to USB-C cable
+* Try a different USB port on your host machine
+* Ensure the provisioning machine has sufficient power supply capability
+
+=== Device not responding after re-connection
+
+If the device does not proceed after the re-connection step:
+
+* Verify that you observed the "bootstrap-fastboot-initialisation-started" state before disconnecting
+* Ensure you held the power button during the re-connection
+* Check the provisioning logs for any error messages
+* Try the complete procedure again from the beginning 

docs/zero2_troubleshooting.adoc renamed to docs/device-guidance/zero2.adoc

@@ -16,4 +16,4 @@ If your Zero 2 W device does not enter RPIBOOT mode, try the following:
 
 == Using secure-boot provisioning style on Zero 2 W devices
 
-=== Secure boot is not available on Zero 2 W devices, and forcing the provisioning style will produce undefined results.
+=== Secure boot is not available on Zero 2 W devices. Provisioning with secure boot will produce undefined results - but absolutely nothing that will work as expected.