diff --git a/docs/settings-system-overview.md b/docs/settings-system-overview.md new file mode 100644 index 000000000..296415690 --- /dev/null +++ b/docs/settings-system-overview.md @@ -0,0 +1,328 @@ +# Bottlerocket Settings System Overview + +This document describes how Bottlerocket's settings system works, including storage, organization, and the flow from settings to configuration files. + +**Keywords:** settings, storage, datastore, filesystem, persistent, /var/lib/bottlerocket, key-value, live, pending, transactions, /etc, tmpfs, configuration, templates, defaults, immutable + +## Overview + +Bottlerocket uses a filesystem-based key/value data store as the central storage location for all settings [1]. +This architecture separates persistent settings from generated configuration files, enabling reliable updates and helping to prevent configuration drift. + +**Key principles:** + +- Settings are stored persistently in `/var/lib/bottlerocket/datastore/` [1][2] +- Configuration files in `/etc` are non-persistent and regenerated on every boot [3] +- Templates stored on the immutable root filesystem define how settings become configuration files [4] +- All settings changes go through the API and are tracked in transactions [1] + +## Data Store Location and Structure + +### Filesystem Layout + +``` +/var/lib/bottlerocket/datastore/ +├── current -> v1/ # Symlink to active data store version +└── v1/ # Versioned data store + ├── live/ # Committed, active settings + │ └── settings/ + │ ├── motd # Setting value (JSON) + │ ├── motd.affected-services # Metadata + │ └── kubernetes/ + │ └── cluster-name + └── pending/ # Uncommitted transactions + └── / + └── settings/ +``` + +The data store is located at `/var/lib/bottlerocket/datastore/` [1][2]. +The `current` symlink points to the active version (e.g., `v1`), allowing the data store format to evolve over time [5]. + +### Live vs. Pending + +**live/** - Contains committed settings that are active and available to the system [1][6]. +Services and configuration file rendering use settings from this directory. + +**pending/** - Contains uncommitted settings organized by transaction name [1][6]. +Settings remain here until explicitly committed, allowing atomic updates of multiple related settings. + +The most common transaction is `bottlerocket-launch`, which coordinates settings from multiple boot-time services [6]. + +## Key-to-Path Mapping + +The data store maps dotted key names to filesystem paths [2][7]: + +| Key | Filesystem Path | +| ---------------------------------------- | ---------------------------------------- | +| `settings.motd` | `settings/motd` | +| `settings.kubernetes.cluster-name` | `settings/kubernetes/cluster-name` | +| `settings.host-containers.admin.enabled` | `settings/host-containers/admin/enabled` | + +**Implementation details:** + +- Each key segment (separated by `.`) becomes a directory or file component [7] +- Segments are percent-encoded for filesystem safety [7] +- Only alphanumeric characters, underscores, and hyphens are allowed unencoded [7] +- Values are stored as JSON for human readability [2] + +**Example:** + +```bash +# Setting the key +apiclient set settings.motd="Welcome to Bottlerocket!" + +# Results in file +/var/lib/bottlerocket/datastore/current/live/settings/motd +# Containing +"Welcome to Bottlerocket!" +``` + +## Metadata Storage + +Metadata about settings is stored alongside the data using a dot-prefix pattern [7]: + +``` +settings/motd # The setting value +settings/motd.affected-services # Metadata: which services to restart +settings/motd.template-path # Metadata: template location +``` + +Metadata files use the pattern `.` [7]. +This allows the system to track additional information about each setting, such as: + +- Which services are affected by changes (`affected-services`) +- Which configuration files use this setting (`configuration-files`) +- How to generate the setting value (`setting-generator`) + +## Settings Lifecycle + +### 1. Default Values + +Default settings are defined in variant-specific `defaults.d` directories [1][8]. +These TOML files specify initial values and configuration file mappings. + +**Example from defaults:** + +```toml +[settings] +motd = "Welcome to Bottlerocket!" + +[configuration-files.motd] +path = "/etc/motd" +template-path = "/usr/share/templates/motd" +``` + +### 2. Data Store Population + +On first boot, `storewolf` creates the data store and populates it with default values [1][8]: + +1. Creates `/var/lib/bottlerocket/datastore/` directory structure +1. Writes default settings to the `pending/bottlerocket-launch/` transaction +1. Settings remain uncommitted until `settings-committer` runs + +### 3. Runtime Settings + +Additional settings are added during boot by: + +- **early-boot-config** - Applies user data from cloud providers (first boot only) [1] +- **sundog** - Generates settings that can only be determined at runtime (e.g., primary IP address) [1] +- **pluto** - Generates Kubernetes-specific settings (e.g., cluster DNS) [1] + +All of these write to the `bottlerocket-launch` transaction in pending state. + +### 4. Commit + +`settings-committer` moves settings from `pending/bottlerocket-launch/` to `live/` [6]. +This makes them available to the rest of the system. +The commit is atomic - all settings in the transaction become live together. + +### 5. Configuration File Generation + +`thar-be-settings` renders configuration files from templates using committed settings [1][9]: + +1. Reads settings from `live/` in the data store (via Bottlerocket API) +1. Loads templates from `/usr/share/templates/` (immutable root filesystem) +1. Renders templates using the `schnauzer` library [9] +1. Writes configuration files to `/etc` (tmpfs) +1. Restarts affected services as needed + +### 6. Service Restart + +Services are restarted based on metadata in the data store [1]. +The `affected-services` metadata for each setting determines which services need to be restarted when that setting changes. + +## Persistence Model + +Bottlerocket has three storage layers with different persistence characteristics: + +### Persistent: Data Store + +**Location:** `/var/lib/bottlerocket/datastore/` + +**Characteristics:** + +- Survives reboots and OS updates [1] +- Stored on the data partition (not the OS partition) +- Migrated automatically during OS upgrades [5] +- Only modified through the API + +**What's stored:** + +- All user-configured settings +- Generated settings (IP addresses, cluster information) +- Setting metadata + +### Non-Persistent: Configuration Files + +**Location:** `/etc` (tmpfs - memory-backed filesystem) [3] + +**Characteristics:** + +- Regenerated on every boot from templates [1][3] +- Changes do not survive reboot +- Cannot be directly modified by users [3] +- Provides protection against persistent configuration tampering + +**What's stored:** + +- Service configuration files (e.g., `/etc/containerd/config.toml`) +- Network configuration (e.g., `/etc/resolv.conf`) +- System configuration files + +### Immutable: Templates and Defaults + +**Location:** `/usr/share/templates/`, `/etc/storewolf/defaults.toml` (root filesystem) [1][8] + +**Characteristics:** + +- Read-only, dm-verity protected [3] +- Part of the OS image +- Updated only through OS upgrades +- Cannot be modified at runtime + +**What's stored:** + +- Configuration file templates +- Default setting values +- Template rendering logic + +## Configuration Flow + +The complete flow from settings to running services: + +``` +┌─────────────────┐ +│ Variant │ +│ defaults.d/ │ Default values defined +└────────┬────────┘ + │ + ▼ +┌─────────────────┐ +│ storewolf │ Populate data store +│ /var/lib/ │ (pending state) +│ bottlerocket/ │ +│ datastore/ │ +└────────┬────────┘ + │ + ▼ +┌─────────────────┐ +│ settings- │ Commit transaction +│ committer │ (pending → live) +└────────┬────────┘ + │ + ▼ +┌─────────────────┐ +│ thar-be- │ Render templates +│ settings │ using live settings +└────────┬────────┘ + │ + ▼ +┌─────────────────┐ +│ /etc/ │ Configuration files +│ (tmpfs) │ written to memory +└────────┬────────┘ + │ + ▼ +┌─────────────────┐ +│ Services │ Services (re)started +│ (systemd) │ with new configuration +└─────────────────┘ +``` + +## User Interaction + +```bash +# Change a setting +apiclient set settings.motd="Custom message" + +# Read a single setting +apiclient get settings.motd + +# Read all settings +apiclient get settings +``` + +## Related Documentation + +- [API System Overview](../sources/api/README.md) - How API components work together +- [Boot Process](boot-process.md) - When settings are populated and applied during boot +- [apiserver](../sources/api/apiserver/README.md) - API server implementation +- [datastore](../sources/api/datastore/README.md) - Data store library implementation +- [storewolf](../sources/api/storewolf/README.md) - Data store initialization +- [thar-be-settings](../sources/api/thar-be-settings/README.md) - Configuration file rendering +- [schnauzer](../sources/api/schnauzer/README.md) - Template rendering library + +## Sources + +[1] [`sources/api/README.md`](../sources/api/README.md) + +- API system overview and component workflow +- Data store description and boot process +- Settings lifecycle from defaults to services + +[2] [`sources/api/apiserver/README.md`](../sources/api/apiserver/README.md) + +- Data store key/value structure +- Default location: `/var/lib/bottlerocket/datastore/current` +- JSON serialization of values + +[3] [`SECURITY_FEATURES.md`](https://github.com/bottlerocket-os/bottlerocket/blob/develop/SECURITY_FEATURES.md) (bottlerocket repo) + +- Stateless tmpfs for `/etc` +- Immutable rootfs backed by dm-verity +- Configuration regeneration on boot +- Security implications of the storage model + +[4] [`sources/api/schnauzer/README.md`](../sources/api/schnauzer/README.md) + +- Template rendering with handlebars +- Settings extensions and helpers +- Template frontmatter format + +[5] [`sources/api/migration/migrator/README.md`](../sources/api/migration/migrator/README.md) + +- Data store versioning and migration +- Version detection from symlink naming + +[6] [`sources/api/settings-committer/README.md`](../sources/api/settings-committer/README.md) + +- Transaction commit process +- `bottlerocket-launch` transaction +- Pending to live transition + +[7] [`sources/api/datastore/src/filesystem.rs`](../sources/api/datastore/src/filesystem.rs) + +- Key-to-path encoding implementation +- Metadata storage with dot-prefix pattern +- `live/` and `pending/` directory structure +- Percent-encoding for filesystem safety + +[8] [`sources/api/storewolf/README.md`](../sources/api/storewolf/README.md) + +- Data store creation and initialization +- Default settings population from `/etc/storewolf/defaults.toml` + +[9] [`sources/api/thar-be-settings/README.md`](../sources/api/thar-be-settings/README.md) + +- Configuration file rendering from templates +- Service restart mechanism