Make board updates one step: update_board.py wrapper, single-board sync, ESPHome-version guard (#1653)

Author: bdraco

CLAUDE.md

@@ -349,7 +349,17 @@ against legacy behaviour before assuming the simpler version suffices.
   `_split_qualified_key` flips it.
 - **Board catalog** (`definitions/boards/<id>/manifest.yaml`) is
   hand-curated YAML. ~80 popular boards plus generic fallbacks per
-  platform. `script/validate_definitions.py` lints the manifests.
+  platform. `script/validate_definitions.py` lints the manifests. The
+  manifest is the only hand-edited file; after editing one, run
+  `script/update_board.py [board-id]` (auto-detects the edited board) to
+  regenerate its JSON and validate in one step. It wraps
+  `script/sync_boards.py <board-id>` (single board) / `sync_boards.py`
+  (all) + `validate_definitions.py`. Single-board mode refuses unless the
+  installed `esphome` matches the `esphome_version` stamped in
+  `boards.index.json` by the last full sync (betas canonicalized to base, as
+  it rebuilds the shared index from every board); a full sync regenerates
+  everything from the installed esphome and re-stamps it, so it doesn't
+  check. Full contributor workflow: `definitions/README.md`.
 - **Frontend handoff** for the catalog is documented inline in models
   (`ConfigEntry`, `ComponentCatalogEntry`). New `ConfigEntryType` values
   need a frontend update — coordinate.
@@ -611,7 +621,8 @@ When changing the sync script or catalog handling, watch for these:
 | `esphome_device_builder/definitions/boards/<id>/manifest.yaml` | Curated; hand-edited. The body directory is `board_bodies/` (separate from this manifests dir) so the body-swap rmtree can't trample the hand-curated source. |
 | `esphome_device_builder/definitions/platform_capabilities.index.json` | Generated; do not hand-edit. esphome platform metadata the long-lived process reads instead of importing `esphome.components.*` (download routing, wifi-inference no-wifi sets, static download-types). Loaded via `load_platform_capabilities_index`. |
 | `esphome_device_builder/helper_cli.py` (`device-builder-helper`) | Subprocess for `get_download_types` on build-dir-dependent platforms (libretiny/nrf52), so the child imports `esphome.components.<X>`, not the dashboard process. |
-| `script/sync_boards.py` | Regenerates the split board catalog from the manifests |
+| `script/update_board.py` | One-step contributor wrapper: regenerate one board's JSON (`sync_boards.py`) + validate (`validate_definitions.py`). Auto-detects the edited board, or takes an id. |
+| `script/sync_boards.py` | Regenerates the split board catalog from the manifests; stamps the generating `esphome_version` into `boards.index.json`. Takes an optional board id to regenerate just one (single-board mode guards installed `esphome` against that stamp). |
 | `script/sync_components.py` | Regenerates the component catalog + `platform_capabilities.index.json` |
 | `script/check_catalog.py` | Smoke test for popular components |
 | `script/check_import_time.py` | CI guard: fails if `import …device_builder` regresses past `script/import_time_budget.json` (e.g. a fresh eager `esphome.components.*` import) |

README.md

@@ -487,7 +487,9 @@ receiver is paired.
 - **[docs/API.md](docs/API.md)** — every WebSocket command, request/response
   shapes, event types.
 - **[esphome_device_builder/definitions/README.md](esphome_device_builder/definitions/README.md)** —
-  contributor guide for board manifests.
+  board (and component) contributor guide: manifest schema plus the
+  workflow for adding or updating a board (edit the manifest, then run
+  `python script/update_board.py` to regenerate and validate).
 
 ## Contributing
 

esphome_device_builder/definitions/README.md

@@ -2,6 +2,88 @@
 
 This directory contains the board and component definitions used by the ESPHome Device Builder.
 
+## Editing boards: the build pipeline
+
+> **Read this first.** Editing a `manifest.yaml` is only half the job; if you
+> stop there, your change never reaches the dashboard and CI fails.
+
+The `manifest.yaml` files under `boards/<id>/` are the only thing you hand-edit.
+The dashboard does **not** read them at runtime. It reads three generated
+artefacts, all written by `script/sync_boards.py`:
+
+| Artefact | What it is |
+|----------|------------|
+| `boards.index.json` | Slim per-board index; powers the board picker. |
+| `board_bodies/<id>.json` | Full body (hardware, pins, featured components); lazy-loaded when a board is opened. |
+| `featured_components.index.json` | Aggregated featured-components map read once at startup. |
+
+After editing a manifest, run the one-step helper from the project venv:
+
+```bash
+python script/update_board.py            # auto-detects the board you edited
+python script/update_board.py my-board   # or name it explicitly
+```
+
+It regenerates that board's JSON, validates the definitions, and prints the
+files to commit. That is all most contributors need.
+
+Under the hood it runs the two scripts you can also call directly:
+
+```bash
+# regenerate the JSON. Pass a board id to rewrite just that board; omit it
+# to rebuild the whole catalog (~990 body files).
+python script/sync_boards.py my-board
+
+# validate the manifests against the schema and cross-references
+python script/validate_definitions.py
+```
+
+Passing the board id (the folder name under `boards/`) rewrites only that
+board's `board_bodies/<id>.json` and refreshes the two index files, so the diff
+stays scoped to what you edited.
+
+Commit the manifest **and** the regenerated JSON together. **Never hand-edit the
+JSON** (`boards.index.json`, `board_bodies/*.json`); it is overwritten on the
+next sync, and `tests/test_boards_json.py::test_split_artefacts_match_manifests`
+compares every manifest against its generated body and fails CI on any drift.
+
+`validate_definitions.py --check-images` additionally fetches each image URL to
+confirm it resolves (network, opt-in); the consistency test exempts `images`, so
+a broken image URL passes the sync but is caught here.
+
+### Run the sync with the project venv
+
+`sync_boards.py` imports ESPHome: it generates the boards no manifest covers
+straight from your installed ESPHome's board tables, and fills curated boards'
+pin aliases the same way. Run it from the project venv:
+
+```bash
+source .venv/bin/activate    # or call .venv/bin/python directly
+python script/sync_boards.py my-board
+```
+
+Single-board mode (and `update_board.py`) rewrites one body but rebuilds the
+shared index from every board, so the installed ESPHome must match the version
+the committed catalog was generated against (stamped as `esphome_version` in
+`boards.index.json` by the last full sync, betas canonicalized to their base
+release) or their index entries silently drift; it refuses on a mismatch and
+prints the version to install. A full `python script/sync_boards.py` regenerates
+everything against your installed ESPHome and re-stamps that version, so it does
+not check; still run it from the venv so you don't commit catalog-wide changes
+from a different ESPHome.
+
+### Curated vs generated vs imported boards
+
+- **Curated** (most hand-written manifests): a `manifest.yaml` with no `source:`
+  block. Edit freely; this is the normal case.
+- **Generated**: no manifest at all. The board comes from ESPHome's board tables
+  on every sync, with a synthesized pin map and a generic image. To customise
+  one (real pinout, photo, featured components), add a curated manifest at
+  `boards/<esphome.board>/manifest.yaml`.
+- **Imported**: a manifest carrying a `source:` block (e.g.
+  `type: esphome-devices`). It is owned by the importer and regenerated from
+  upstream, so hand edits are overwritten; change the upstream source instead.
+
 ## Adding a Board
 
 Create a new subfolder in `boards/` with a `manifest.yaml`:
@@ -27,20 +109,28 @@ manufacturer: "Acme Corp"
 
 # ESPHome configuration — maps directly to the ESPHome YAML platform block
 esphome:
-  platform: esp32              # esp32, esp8266, rp2040, bk72xx, rtl87xx
+  platform: esp32              # esp32, esp8266, rp2040, bk72xx, rtl87xx, ln882x, nrf52, host
   board: esp32-s3-devkitc-1    # PlatformIO board ID
-  variant: esp32s3             # ESP32 chip variant (omit for esp8266/rp2040)
-  framework: esp-idf           # arduino or esp-idf (omit for platform default)
-  flash_size: 8MB              # 2MB, 4MB, 8MB, 16MB (omit for board default)
+  variant: esp32s3             # ESP32 chip variant only (omit otherwise)
+  framework: esp-idf           # arduino, esp-idf, or zephyr (omit for platform default)
+
+# Hardware specs (all optional)
+hardware:
+  flash_size: 8MB              # 2MB, 4MB, 8MB, 16MB
+  ram_size: 327680             # bytes
+  cpu_frequency: 240MHz
+  connectivity: [wifi, bluetooth]
 
 # Optional metadata
-tags: [esp32-s3, wifi, bluetooth, usb, rgb-led]
-docs_url: "https://example.com/docs"
+tags: [compact, usb-c, rgb-led]   # only the enum values in board.schema.json
+docs_url: "https://esphome.io/components/esp32.html"
+product_url: "https://example.com/my-awesome-board"
 is_generic: false              # true only for generic fallback boards
 
-# Images — URLs or paths relative to this manifest (first = primary)
+# Images; URLs or paths relative to this manifest (first = primary).
+# Prefer a bundled local asset over a hotlinked vendor URL, which can rot.
 images:
-  - "https://example.com/board.png"
+  - "images/board-top.png"
   - "images/pinout.png"
 
 # Pin definitions (see below)
@@ -49,6 +139,11 @@ pins:
     # ...
 ```
 
+`tags` accepts only the values enumerated in
+[`schemas/board.schema.json`](schemas/board.schema.json) (`compact`, `dev-kit`,
+`usb-c`, `rgb-led`, `poe`, ...); platform, variant, and connectivity live in
+their own fields, not in `tags`.
+
 ### Pin Definitions
 
 The pin map is the most valuable part of a board definition. It enables the

esphome_device_builder/definitions/boards.index.json

@@ -23,17 +23,27 @@
 remain the human-editable source of truth; this script is the only
 thing that writes the three artefacts.
 
+Single-board mode (``BOARD_ID``) must run against the same ESPHome the
+rest of the committed catalog was generated against
+(``esphome_schema_version`` in ``components.index.json``), since it
+rebuilds the shared index from every board; it refuses on a mismatch. A
+full sync regenerates everything from the installed ESPHome and is
+internally consistent regardless, so it does not check.
+
 Usage
 -----
 
-    python script/sync_boards.py
+    python script/sync_boards.py              # regenerate every board
+    python script/sync_boards.py BOARD_ID     # regenerate only one board
 """
 
 from __future__ import annotations
 
+import argparse
 import importlib
 import logging
 import re
+import shutil
 import sys
 from dataclasses import replace
 from operator import attrgetter
@@ -811,6 +821,21 @@ def build_catalog() -> BoardCatalogResponse:
 
 def main() -> int:
     logging.basicConfig(level=logging.INFO, format="%(levelname)s %(name)s: %(message)s")
+    parser = argparse.ArgumentParser(
+        description="Regenerate the board catalog JSON from the YAML manifests."
+    )
+    parser.add_argument(
+        "board",
+        nargs="?",
+        help="Board id (the folder name under esphome_device_builder/definitions/boards/) "
+        "to regenerate on its own. Omit to regenerate the whole catalog.",
+    )
+    args = parser.parse_args()
+
+    # Only single-board mode needs the match: it rebuilds the shared index from
+    # every board, so a mismatched esphome drifts the others' index entries.
+    if args.board:
+        _require_matching_esphome()
 
     # Abort the sync on the first bad manifest — partial output here
     # would silently ship a board-shaped hole to every install.
@@ -819,6 +844,10 @@ def main() -> int:
     # ``to_dict`` here already applies the omit_default Configs, so
     # body files and index entries both ship the stripped wire shape.
     full_payloads = [board.to_dict() for board in catalog.boards]
+
+    if args.board:
+        return _emit_single_board(catalog.boards, full_payloads, args.board)
+
     _emit_split_catalog(catalog.boards, full_payloads)
     _emit_featured_components_index(catalog.boards)
 
@@ -832,6 +861,45 @@ def main() -> int:
     return 0
 
 
+def _emit_single_board(
+    boards: list[BoardCatalogEntry], full_payloads: list[dict[str, Any]], board_id: str
+) -> int:
+    """
+    Rewrite one board's body file, then refresh the index and featured map.
+
+    Only the target's ``board_bodies/<id>.json`` is rewritten; the index and
+    featured-components files are full rebuilds whose other entries are
+    byte-identical (the version guard rules out drift), so the diff stays
+    scoped to the edited board. This assumes only *board_id* was edited:
+    naming one board while another's manifest is also dirty rewrites the
+    other's index entry but not its body, which the consistency test flags.
+    """
+    idx = next((i for i, board in enumerate(boards) if board.id == board_id), None)
+    if idx is None:
+        raise SystemExit(
+            f"sync_boards: no board with id {board_id!r}; "
+            f"expected a folder name under {_DEFINITIONS_DIR / 'boards'}"
+        )
+    _emit_body_atomically(full_payloads[idx], board_id)
+    _write_index(full_payloads)
+    _emit_featured_components_index(boards)
+    _LOGGER.info("Regenerated board_bodies/%s.json + refreshed index and featured map", board_id)
+    return 0
+
+
+def _emit_body_atomically(payload: dict[str, Any], board_id: str) -> None:
+    """Write one body file via stage-then-replace so an interrupted write can't truncate it."""
+    staging = _BODIES_DIR.parent / "board_bodies.single"
+    prepare_next_bodies_dir(staging)
+    try:
+        emit_body_with_roundtrip(
+            payload, board_id, staging, BoardCatalogEntry, log_label="Board", sort_keys=True
+        )
+        (staging / f"{board_id}.json").replace(_BODIES_DIR / f"{board_id}.json")
+    finally:
+        shutil.rmtree(staging, ignore_errors=True)
+
+
 def _emit_split_catalog(
     boards: list[BoardCatalogEntry], full_payloads: list[dict[str, Any]]
 ) -> None:
@@ -853,23 +921,87 @@ def _emit_split_catalog(
             sort_keys=True,
         )
 
-    index_payload = {
-        "boards": sorted(
-            (_strip_body_fields(payload) for payload in full_payloads),
-            key=lambda p: p["id"],
-        ),
-    }
     swap_split_catalog_in(
         next_bodies=next_bodies,
         live_bodies=_BODIES_DIR,
-        index_payload=index_payload,
+        index_payload=_index_payload(full_payloads),
         live_index=_INDEX_FILE,
         index_cls=BoardCatalogIndex,
         index_entries_key="boards",
         sort_keys=True,
     )
 
 
+def _index_payload(full_payloads: list[dict[str, Any]]) -> dict[str, Any]:
+    """Build the slim index payload, stamped with the ESPHome it was generated from."""
+    return {
+        "esphome_version": _installed_esphome_version(),
+        "boards": sorted(
+            (_strip_body_fields(payload) for payload in full_payloads),
+            key=lambda p: p["id"],
+        ),
+    }
+
+
+# ESPHome betas/dev builds (``2026.7.0b1``, ``2026.7.0-dev``) share board tables
+# with their base release, so canonicalize to the base for both the stamp and
+# the guard or a beta would false-mismatch its own release.
+_ESPHOME_BASE_VERSION_RE = re.compile(r"^(\d+\.\d+\.\d+)")
+
+
+def _canonical_esphome_version(version: str) -> str:
+    """Drop a prerelease/dev suffix: ``2026.7.0b1`` -> ``2026.7.0``."""
+    match = _ESPHOME_BASE_VERSION_RE.match(version)
+    return match.group(1) if match else version
+
+
+def _installed_esphome_version() -> str:
+    from esphome.const import __version__
+
+    return _canonical_esphome_version(__version__)
+
+
+def _write_index(full_payloads: list[dict[str, Any]]) -> None:
+    """Rewrite ``boards.index.json`` only, leaving the body files untouched."""
+    index_payload = _index_payload(full_payloads)
+    for entry in index_payload["boards"]:
+        BoardCatalogIndex.from_dict(entry)
+    next_index = _INDEX_FILE.with_suffix(".json.next")
+    next_index.write_bytes(
+        orjson.dumps(index_payload, option=orjson.OPT_SORT_KEYS | orjson.OPT_APPEND_NEWLINE)
+    )
+    next_index.replace(_INDEX_FILE)
+
+
+def _require_matching_esphome() -> None:
+    """Abort unless installed ESPHome matches the ``esphome_version`` boards.index.json was built with."""
+    try:
+        expected = orjson.loads(_INDEX_FILE.read_bytes())["esphome_version"]
+    except (OSError, orjson.JSONDecodeError, KeyError):
+        raise SystemExit(
+            f"sync_boards: could not read esphome_version from {_INDEX_FILE}.\n"
+            f"To fix, regenerate the whole catalog first: python script/sync_boards.py"
+        ) from None
+    try:
+        from esphome.const import __version__ as raw_installed
+    except ImportError:
+        raise SystemExit(
+            f"sync_boards: ESPHome is not importable in this interpreter ({sys.executable}).\n"
+            f"To fix, install it into the project venv and re-run:\n"
+            f"    uv pip install 'esphome=={expected}'   # or: pip install 'esphome=={expected}'"
+        ) from None
+    installed = _canonical_esphome_version(raw_installed)
+    if installed != expected:
+        raise SystemExit(
+            f"sync_boards: single-board mode needs ESPHome {expected} (the version "
+            f"boards.index.json was generated with), but {installed} is installed.\n"
+            f"To fix, install the matching version into this venv and re-run:\n"
+            f"    uv pip install 'esphome=={expected}'   # or: pip install 'esphome=={expected}'\n"
+            f"Or regenerate the whole catalog against your installed ESPHome instead:\n"
+            f"    python script/sync_boards.py"
+        )
+
+
 def _emit_featured_components_index(boards: list[BoardCatalogEntry]) -> None:
     """Write the aggregated ``{board_id: list[FeaturedComponent]}`` index.