Merge branch 'main' of github.com:Pioreactor/docs.pioreactor

* 'main' of github.com:Pioreactor/docs.pioreactor:
  note
  more pds
  new build
  updates
  update some docs
  more about custom models
  note on v1.5
  Fix broken link
  getting started page for v1.5
  new modal on startup
  new modal on startup

Author: Kelly Tran

developer-guide/09-Storage and the filesystem/01-data-stores/01-data-stores-intro.md

@@ -18,13 +18,13 @@ The Pioreactor software will automatically backup the SQLite database via a sche
 
 ## Local key-value datastore
 
-SQLite3 is also used by the library *diskcache*. This is essentially a fast key-value store on the Raspberry Pi. For Pioreactor, we use it to store "machine-specific" data, like calibration curves, locks on GPIOs, state of LEDs, jobs running, etc. Instead of one large file containing all these keys, we have split them into multiple locations based on category and level of persistence. The persistent databases are stored in `/home/pioreactor/.pioreactor/storage` and the temporary databases are in `/tmp`. You can access them from Python using `pioreactor.utils.local_persistent_storage` and `pioreactor.utils.local_intermittent_storage`, respectively.
+SQLite3 is also used for simpler database (caches) on each Pioreactor (worker and leader). The persistent databases are stored in `/home/pioreactor/.pioreactor/storage` and the temporary databases are in `/run/pioreactor/cache`. You can access them from Python using `pioreactor.utils.local_persistent_storage` and `pioreactor.utils.local_intermittent_storage`, respectively.
 
 :::info
 What are temporary and persistent? Something like GPIO locks or LED state are physically reset between cycles of the Raspberry Pi. So when the Pi power-cycles, the state is wiped, and by have the database in `/tmp`, the databases are wiped as well.
 :::
 
-You can use `pio view-cache <name>` to view the contents of `<name>`, and `pio clear-cache <name> <key>` to clear contents.
+You can use `pio cache view <name>` to view the contents of `<name>`, and `pio cache clear <name> <key>` to clear contents.
 
 
 ## MQTT
@@ -33,9 +33,9 @@ The inter- and intra-Pioreactor communications are handled by MQTT, a pub/sub se
 
 A principle we have stood by is to not let MQTT turn into our database. That is,
 
-1. we shouldn't store important information in MQTT _only_ (also use `mqtt_to_db_streaming` job to store important data in the SQLite3 database, or handle it locally using `local_intermitant_storage` or `local_persistance_storage`),
-2. we shouldn't store information in MQTT that is "machine-specific", like calibrations (better to use  `local_intermitant_storage` or `local_persistance_storage`)
-3. we shouldn't use MQTT as a source of truth (trust the database, `local_intermitant_storage` or `local_persistance_storage` more).
+1. we shouldn't store important information in MQTT _only_ (also use `mqtt_to_db_streaming` job to store important data in the SQLite3 database, or handle it locally using `local_intermittent_storage` or `local_persistent_storage`),
+2. we shouldn't store information in MQTT that is "machine-specific", like calibrations (better to use  `local_intermittent_storage` or `local_persistent_storage`)
+3. we shouldn't use MQTT as a source of truth (trust the database, `local_intermittent_storage` or `local_persistent_storage` more).
 
 
 #### Serialization of MQTT messages

developer-guide/09-Storage and the filesystem/01-data-stores/02-sqlite.md

@@ -18,7 +18,7 @@ You can either use `.tables` in the sqlite shell, or visit all the tables [schem
 
 ### How can I add new SQL tables?
 
-Using the sqlite shell (`pio db`), you can execute any SQL statements, including creating new tables. Plugins can also automatically create new tables when installed, [see instructions here](developer-guide/plugin-as-python-package#5-optional-adding-tables-to-the-sql-store).
+Using the sqlite shell (`pio db`), you can execute any SQL statements, including creating new tables. Plugins can also automatically create new tables when installed, [see instructions here](plugin-as-python-package#5-optional-adding-tables-to-the-sql-store).
 
 ### How do I populate my own SQL tables?
 

developer-guide/09-Storage and the filesystem/02-filesystem.md

@@ -4,49 +4,29 @@ slug: /filesystem-locations
 hide_table_of_contents: true
 ---
 
-
-Below is a list of important locations and files on the filesystem for the Pioreactor:
-
-
-### UI
-
- - `/var/www/pioreactorui/` is the source of the web-app.
- - `/var/www/pioreactorui/.env` holds some configuration for the UI.
- - `/var/www/pioreactorui/contrib/` holds yaml files that display automation data, job data, and chart data.
- - Note: `/home/pioreactor/.pioreactor/plugins/ui/contrib/` also holds yaml files, like the above.
-
-
-### Logs
-
- - `/var/log/pioreactor.log` is the log file for the Pioreactor app
- - `/var/log/pioreactorui.log` is the (deprecated) log file for the Pioreactor UI. It's now the same as above.
-
-
-### Storage
-
- - `/home/pioreactor/.pioreactor/storage/` holds the main database, backup of the database, and persistent caches.
- - `/tmp/pioreactor_cache/` holds temporary caches. Files in here are not kept between reboots.
-
-
-### Bash scripts
- - `/usr/local/bin/` store bash scripts that are used when working with the filesystem: installing plugins, updating code, etc.
-
-
-### Config
-
- - `/home/pioreactor/.pioreactor/` holds all configuration files for the Pioreactor UI and app, the main one being `config.ini`
-
-
-### Plugins
-
- - `/home/pioreactor/.pioreactor/plugins/` is where Python files can be added.
- - `/home/pioreactor/.pioreactor/plugins/ui/contrib/` also holds UI yaml files for custom Python code.
-
-### Experiment profiles
-
- - `/home/pioreactor/.pioreactor/experiment_profiles/` is where experiment profiles (yaml) are stored.
-
-### Calibrations
-
- - `/home/pioreactor/.pioreactor/storage/calibrations/` is where calibrations are stored.
-
+# Important Raspberry Pi Locations for Pioreactor Images
+
+Reference list of on-device paths that matter once a custom Pioreactor Raspberry Pi OS image is flashed and running.
+
+| Path | What lives here | Notes |
+| --- | --- | --- |
+| `/etc/pioreactor.env` | Shared environment file exported to services | Provides canonical `DOT_PIOREACTOR`, `RUN_PIOREACTOR`, `LG_WD`, and virtualenv paths so systemd units pick up the same locations. |
+| `/home/pioreactor/.pioreactor/` | Pioreactor home (`$DOT_PIOREACTOR`) | Contains configs, plugins, storage, hardware definitions, logs, and other per-device state. Ensure ownership stays `pioreactor:pioreactor`. |
+| `/home/pioreactor/.pioreactor/config.ini` | Cluster-wide configuration | Holds network, MQTT, UI, logging, and automation settings replicated to workers. |
+| `/home/pioreactor/.pioreactor/unit_config.ini` | Unit overrides | Optional per-device overrides synced from the leader when a worker is added. |
+| `/home/pioreactor/.pioreactor/plugins/` | Plugin root | Mirrors the `.pioreactor` layout for plugin-provided files; typically subdivided into `python/`, `ui/`, `exportable_datasets/`, etc. |
+| `/home/pioreactor/.pioreactor/plugins/ui/` | UI plugin assets | Static contrib YAML, templates, and other UI resources merged into the Pioreactor UI. |
+| `/home/pioreactor/.pioreactor/plugins/exportable_datasets/` | Exportable dataset plugins | Each plugin ships SQL queries or serializers that surface new datasets to the UI. |
+| `/home/pioreactor/.pioreactor/storage/` | Persistent databases | Holds SQLite data such as `pioreactor.sqlite` (experiments) and `local_persistent_pioreactor_metadata.sqlite`. |
+| `/home/pioreactor/.pioreactor/storage/pioreactor.sqlite` | Primary experiment database | Used by the leader, replicated to workers for durability. Back up before major upgrades. |
+| `/home/pioreactor/.pioreactor/storage/local_persistent_pioreactor_metadata.sqlite` | Persistent cache | Stores long-lived metadata that survives reboots. |
+| `/home/pioreactor/.pioreactor/hardware/` | Hardware definition packs | YAML that describes hats, models, sensors, and calibration values consumed by the Pioreactor hardware subsystem. |
+| `/home/pioreactor/.pioreactor/experiment_profiles/` | User experiment profiles | Drop `.py` / `.json` profile definitions here so they show up in the UI for scheduling. |
+| `/home/pioreactor/.ssh/` | Device SSH keys | Created during image build so leaders can add workers securely. |
+| `/run/pioreactor/` | Runtime scratch space (`$RUN_PIOREACTOR`) | tmpfs directory for caches, sockets, Huey queues, and general runtime coordination. Cleared on reboot. |
+| `/run/pioreactor/cache/` | Volatile caches | Includes `local_intermittent_pioreactor_metadata.sqlite` and UI/Huey cache files that can be safely discarded. |
+| `/run/pioreactor/exports/` | Web export staging | Lighttpd serves `/exports/` from here so downloads never touch persistent storage. |
+| `/var/log/pioreactor.log` | System log for Pioreactor services | Configured via `config.ini` and read by both CLI and UI. Rotate with journald/logrotate as needed. |
+| `/opt/pioreactor/venv/` | System Python virtual environment | Hosts the Pioreactor Python installation (`pio`, `pios`, services). Activate manually for debugging. |
+| `/home/pioreactor/.local/bin/` | User-level executables | Supplementary scripts installed via `pip --user`. |
+| `/tmp/` | Temporary workspace | Referenced by `TMPDIR` in `pioreactor.env` for short-lived files outside tmpfs. |

developer-guide/10-Hardware/05-i2c-addresses.md

@@ -0,0 +1,83 @@
+---
+title: Customize the hardware interface
+slug: /custom-hardware
+description: Extend or override Pioreactor hardware definitions by layering YAML files that hardware.py reads.
+hide_table_of_contents: true
+---
+
+Pioreactor's hardware layer is intentionally data-driven. Everything in [`core/pioreactor/hardware.py`](https://github.com/pioreactor/pioreactor/blob/main/core/pioreactor/hardware.py) loads user-editable YAML files from `~/.pioreactor/hardware/` (or the folder pointed to by the `DOT_PIOREACTOR` env var). By editing these files you can rewire pins, add new peripherals, or describe an entirely new bioreactor model without touching the Python code. Pair these configs with [custom bioreactor model definitions](/developer-guide/custom-bioreactor-models) so the UI, safety limits, and wiring stay in sync.
+
+## How the loader works
+
+`hardware.py` deep-merges two directories for each **mod** (subsystem):
+
+1. `~/.pioreactor/hardware/hats/<hat_version>/<mod>.yaml` – physical wiring that ships with a specific HAT revision (`hardware_version_info`).
+2. `~/.pioreactor/hardware/models/<model>/<version>/<mod>.yaml` – intent for a particular bioreactor model returned by `get_pioreactor_model()`.
+
+Later layers override earlier ones. Missing files are allowed; missing keys throw `HardwareError`s when helpers such as `get_pwm_to_pin_map()` run.
+
+Common mods defined inside `hardware.py` are:
+
+- `pwm.yaml` (`get_pwm_controller`, `get_heater_pwm_channel`, `get_pwm_to_pin_map`)
+- `gpio.yaml` (`get_pcb_led_pin`, `get_hall_sensor_pin`, `get_sda_pin`, `get_scl_pin`)
+- `temp.yaml` (`get_temp_address`, used by `is_heating_pcb_present`)
+- `adc.yaml` (`get_adc_curriers`, which subsequently powers PD channel helpers)
+- `dac.yaml` (`get_dac_address`)
+
+You can add additional mods with any key/value structure—`get_layered_mod_config("my_mod")` will still combine the YAML files.
+
+## Step-by-step: add a custom hardware profile
+
+1. **Create directories** for your HAT and model version.
+   ```bash
+   mkdir -p ~/.pioreactor/hardware/hats/<hat_version>
+   mkdir -p ~/.pioreactor/hardware/models/<model_name>/<model_version>
+   ```
+   Use `pioreactor.version.hardware_version_info` (for hats) and your `pioreactor.models` entry for models.
+
+2. **Describe the wiring** (hat layer). Example `~/.pioreactor/hardware/hats/0.2/pwm.yaml`:
+   ```yaml
+   controller: hat_mcu
+   heater_pwm_channel: "5"
+   pwm_to_pin:
+     "1": 17
+     "2": 13
+     "5": 18
+   ```
+   These keys map directly to `_load_pwm_cfg()` in `hardware.py`.
+
+3. **Describe the capabilities** (model layer). Suppose you added a new auxiliary photodiode ADC:
+   ```yaml title="~/.pioreactor/hardware/models/my_model/v1.0/adc.yaml"
+   pd1:
+     driver: ADS1115
+     address: 0x48
+     channel: 0
+   aux:
+     driver: pico
+     address: 0x40
+     channel: 0
+   salty:
+     driver: ADS1115
+     address: 0x49
+     channel: 2
+   ```
+   `get_adc_curriers()` will automatically expose `.keys()` for every entry (`pd1`, `aux`, `salty`). Downstream helpers, like `get_available_pd_channels()`, treat `pd*` names as photodiode channels.
+
+4. **Reference new mods in code**. Anywhere in your plugin/automation you can import:
+   ```python
+   from pioreactor.hardware import get_layered_mod_config
+
+   stir_cfg = get_layered_mod_config("stirrer")  # merges hats/<hat>/stirrer.yaml and models/<model>/stirrer.yaml
+   if stir_cfg.get("controller") == "pico":
+       ...
+   ```
+   This keeps custom hardware logic outside Pioreactor core.
+
+## Tips for customization
+
+- Use integers for all addresses and pins; `hardware.py` casts everything via `int()` and will raise if parsing fails.
+- Keep YAML minimal—only override keys that differ from the hat defaults to take advantage of deep-merge layering.
+- The helper predicates (`is_ADC_present`, `is_heating_pcb_present`, etc.) are safe ways to gate optional hardware in your jobs.
+
+
+Once you describe the hardware this way, the rest of the stack—jobs, automations, plugins—just asks `hardware.py` for the capabilities it needs, so new hardware becomes a configuration exercise instead of a fork of the core codebase.

developer-guide/10-Hardware/10-custom-hardware.md

@@ -0,0 +1,75 @@
+---
+title: Custom bioreactor models
+slug: /custom-bioreactor-models
+hide_table_of_contents: true
+---
+
+:::tip
+This feature is actively under development - let us know what else you would like to see!
+:::
+
+If you are building a novel bioreactor on top of our software, you can use the `~/.pioreactor/models` dir to add your own bioreactors. This metadata flows into the Inventory UI, safety interlocks, analytics, and can be consumed from Python via `pioreactor.whoami.get_pioreactor_model()`.
+
+## Directory layout
+
+1. Create the models folder if it does not exist:
+   ```bash
+   mkdir -p ~/.pioreactor/models
+   ```
+2. Add one YAML file per model+version, for example `~/.pioreactor/models/custom_100ml__v1_0.yaml`.
+3. Restart the Pioreactor UI to pick up the new definition.
+4. Assign pioreactors to this new model on the Inventory UI page or via `pio workers update-model ...` tool.
+
+## Required fields
+
+Each YAML is validated against `pioreactor.structs.Model`, so include at least:
+
+- `model_name`: slug-style identifier (`custom_100ml`).
+- `model_version`: semantic-ish version (`"1.0"`).
+- `display_name`: friendly label shown in the Inventory.
+- `reactor_capacity_ml`: physical vessel size.
+- `reactor_max_fill_volume_ml`: operational maximum before spilling.
+- `reactor_diameter_mm`: used by stirring + growth-rate heuristics.
+- `max_temp_to_reduce_heating`, `max_temp_to_disable_heating`, `max_temp_to_shutdown`: temperature safety thresholds in °C.
+
+## Example definition
+
+```yaml title="~/.pioreactor/models/custom_100ml.yaml"
+model_name: custom_100ml
+model_version: "1.0"
+display_name: "Custom 100 mL, v1.0"
+reactor_capacity_ml: 100.0
+reactor_max_fill_volume_ml: 95.0
+reactor_diameter_mm: 50.0
+max_temp_to_reduce_heating: 80.0
+max_temp_to_disable_heating: 85.0
+max_temp_to_shutdown: 90.0
+```
+
+After saving the file, the Inventory page will list “Custom 100 mL, v1.0” in each unit’s model dropdown. The same data becomes available to jobs and plugins:
+
+```python
+from pioreactor.whoami import get_pioreactor_model
+
+model = get_pioreactor_model()
+print(model.display_name)
+print(model.reactor_capacity_ml)
+```
+
+## Coordinating with hardware profiles
+
+Model definitions describe intent (volumes, limits), while the hardware loader handles wiring, pin assignments, and sensor addresses. When you introduce a new model, you usually need a matching directory under `~/.pioreactor/hardware/models/<model_name>/<model_version>/` so [`hardware.py` can layer the right YAMLs](/developer-guide/custom-hardware). Keeping the names and versions identical lets software map UI selections → hardware capabilities.
+
+If you are basing your custom model off an existing Pioreactor model, can reuse the hardware for our Pioreactor models:
+
+```
+mkdir -p ~/.pioreactor/hardware/models/<model_name>/<model_version>
+cp ~/.pioreactor/hardware/models/pioreactor_40ml/1.5/* ~/.pioreactor/hardware/models/<model_name>/<model_version>
+
+```
+
+## Validation and troubleshooting
+
+- Run `python -m pioreactor.models` to ensure your YAML parses; validation errors are also logged via the `models` logger.
+- If a model is missing from the dropdown, confirm the filename ends with `.yaml`/`.yml`, the process has read permissions, and the Pioreactor services have restarted.
+- Treat this directory like code: commit the YAMLs to version control alongside any CAD notes or lab SOPs so teammates understand the assumptions behind each model.

developer-guide/10-Hardware/11-custom_models.md

@@ -26,6 +26,7 @@
   border-radius: 5px; /* Rounded corners */
   font-size: 0.9em;
   font-weight: bold;
+  z-index: 1; /* Keep the badge above the zoomed image */
 }
 
 .cardImage {
@@ -47,4 +48,4 @@
 .cardHeader {
   padding: 10px;
   text-align: center;
-}
+}