Tup refactor - status update

.agents/codebase-insights.txt: 
0006-tup-build-staging-area.md: 
tup-build-staging-implementation-plan.md: 
tup-build-staging-status.md: 

Signed-off-by: Tzanko Matev <[email protected]>

Author: tzanko-matev

.agents/codebase-insights.txt

@@ -33,6 +33,8 @@
 - `nix/packages/default.nix` now hides helper bindings like `mkNimBinary` via `builtins.removeAttrs` so `perSystem.packages` exposes only real derivations; flake checks will reject non-package values there.
 - The generated `appimagePayload` launcher (`bin/ct`) now executes `ct_unwrapped` from the same directory instead of `bin/ct_unwrapped`, fixing AppImage runs where `HERE` already points at `bin/`.
 - `ci/build/dev.sh` temporarily moves `src/links` out of the tree before `tup generate` because the generated script writes artifacts directly into `src/`, forcing a full `git clean -xfd src/` afterward.
+- The staged Tup layout now lives under `src/build/`; live `tup` runs and `just build` execute from there, and variant artifacts are produced in `src/build-debug/build/`.
+- Running a script produced by `tup generate` populates `src/build/`; clean the staging trees with `cd src/build && git clean -fx .` and `cd ../build-debug && git clean -fx .` before resuming live `tup`.
 - `!tup_preserve` only mirrors files relative to the directory containing the Tupfile; after moving Tupfiles into `src/build` we copy assets with the new `!cp_preserve` helper so files can originate from the original `src/` tree without confusing tup.
 - `!cp_preserve` replaces symlink sources with fresh links pointing at the original target (using `readlink -f`) so mirrored assets stay valid when their relative paths shift.
 - `ct_wrapper` resolves `ct_paths.json` by honouring `CODETRACER_CT_PATHS` and otherwise searching upward from both the binary location and the current working directory, so it no longer assumes the wrapper lives at a specific depth in the checkout.

0006-tup-build-staging-area.md

@@ -16,11 +16,11 @@ We tried `tup generate --builddir …` to sandbox the outputs, but the option is
 
 We will move the entire Tup configuration into a dedicated staging area at `src/build/`, so that both `tup` FUSE runs and generated scripts create or modify files only inside that subtree while still producing the expected variant outputs (`build-debug/`, etc.).
 
-1. **Establish `src/build/` as the staging tree:** Relocate the root `Tupfile` (providing a thin wrapper in `src/Tupfile` if needed) and every subordinate `Tupfile` under a mirrored directory structure rooted at `src/build/`, while keeping `Tupfile.ini` anchored in `src/` so the Tup root remains the project root. Each rule will reference sources in `../` (or higher) as needed, but outputs and intermediate files stay under `src/build/` by default.
-2. **Centralize path conventions:** Introduce helper variables in `Tuprules.tup` (e.g., `SRC_ROOT`, `BUILD_ROOT`, `VARIANT_ROOT`) so recipes can target `src/build/` during generated executions while still honoring variant directories such as `src/build-debug/` when `CONFIG_DEBUG=1` or when explicit output directories are required.
+1. **Establish `src/build/` as the staging tree:** Relocate the root `Tupfile` and every subordinate `Tupfile` under a mirrored directory structure rooted at `src/build/`, while keeping `Tupfile.ini` anchored in `src/` so the Tup root remains the project root. The staged layout becomes the canonical entry point—invocations now run from `src/build/` without requiring a compatibility shim in `src/Tupfile`. Each rule references sources in `../` (or higher) as needed, but outputs and intermediate files stay under `src/build/` by default.
+2. **Stable path conventions via `SRC_DIR`:** Each staged `Tupfile` declares its own source root with a local `SRC_DIR` variable. This per-file approach is sufficient for the refactor and avoids introducing additional global macros.
 3. **Git hygiene:** Add a scoped `.gitignore` inside `src/build/` that admits `tup.sh`, generated metadata (`*.tup` state, temporary outputs), and any new staging directories while keeping declarative build files tracked.
-4. **Tooling alignment:** Update `ci/build/dev.sh`, `justfile`, and shell/Nix helpers to invoke `tup` from `src/build/`. The CI clean step will narrow to `git clean -xfd src/build/` instead of the whole source tree, and hacks that relocate `src/links` become unnecessary.
-5. **Documentation & onboarding:** Refresh contributor docs to explain the new layout, clarifying that FUSE-based workflows (`tup monitor`) and generated scripts both operate inside `src/build/`, while `tup build-debug` continues to populate `src/build-debug/`.
+4. **Tooling alignment:** Update `ci/build/dev.sh`, `justfile`, and shell/Nix helpers to invoke `tup` from `src/build/`. The CI clean step focuses on `src/build/` and `src/build-debug/build/`, and hacks that relocate `src/links` become unnecessary.
+5. **Documentation & onboarding:** Refresh contributor docs to explain the new layout, clarifying that FUSE-based workflows (`tup monitor`) operate inside `src/build/`, while `tup build-debug` and generated scripts populate `src/build-debug/build/`. Highlight that `tup generate` writes into the staging tree and document the clean-up sequence required before running the live `tup` monitor again (`cd src/build && git clean -fx .`, followed by `cd ../build-debug && git clean -fx .`).
 
 ## Alternatives Considered
 
@@ -30,10 +30,10 @@ We will move the entire Tup configuration into a dedicated staging area at `src/
 
 ## Consequences
 
-- **Positive:** Generated scripts run cleanly without mangling tracked files; CI simplifications; consistent developer experience across environments lacking FUSE; easier to inspect or purge build artifacts via `git clean src/build`.
+- **Positive:** Generated scripts run cleanly without polluting tracked files; CI simplifications; consistent developer experience across environments lacking FUSE; easier to inspect or purge build artifacts via `git clean src/build` and `git clean src/build-debug/build`.
 - **Neutral/Enabling:** Establishes clearer separation between declarative build metadata and source assets, paving the way for additional variants or build caching under `src/build/`.
-- **Negative:** Requires refactoring every `Tupfile` path to account for the new root; risk of path mistakes during migration; temporary churn for developers with pending Tup changes.
-- **Risks & Mitigations:** Misconfigured outputs could still escape the sandbox—mitigate by adding integration tests that run `tup generate` + `tup.sh` and assert the set of touched paths. Variant parity must be verified by running `tup build-debug` before and after the move within CI.
+- **Negative:** Requires refactoring every `Tupfile` path to account for the new root; risk of path mistakes during migration; temporary churn for developers with pending Tup changes. Running a generated script will populate `src/build/`; developers must clean both `src/build/` and `src/build-debug/` before resuming normal `tup` usage.
+- **Risks & Mitigations:** Misconfigured outputs could still escape the sandbox—mitigate by adding integration tests that run `tup generate` + `tup.sh` and assert the set of touched paths. Variant parity must be verified by running `tup build-debug` before and after the move within CI. Document the clean-up sequence after generated runs to avoid confusing residual artifacts.
 
 ## Key Locations
 
@@ -44,14 +44,14 @@ We will move the entire Tup configuration into a dedicated staging area at `src/
 
 ## Implementation Notes
 
-1. Mirror the existing Tup hierarchy under `src/build/`, keeping directory-by-directory `Tupfile`s collocated with their rules while adjusting relative paths for inputs and outputs; retain `src/Tupfile.ini` at the root and add a lightweight `src/Tupfile` shim that delegates into `src/build/Tupfile`.
-2. Define shared path macros (e.g., `SRC_ROOT = $(TUP_CWD)/../../..`) so recipes can reference the actual sources without brittle `../../..` sequences, and expose a `BUILD_ROOT` for staging outputs.
+1. Mirror the existing Tup hierarchy under `src/build/`, keeping directory-by-directory `Tupfile`s collocated with their rules while adjusting relative paths for inputs and outputs; retain `src/Tupfile.ini` at the root and run `tup` directly from `src/build/`.
+2. Keep per-directory `SRC_DIR` variables to reference the original sources without introducing additional macros.
 3. Create a `.gitignore` within `src/build/` that ignores generated scripts, logs, and `tup` state directories but keeps the committed `Tupfile`s tracked.
-4. Update tooling (`ci`, `just`, Nix shells) to execute `tup` from `src/build/`, ensuring variant commands still find `build-debug/tup.config`.
-5. Add CI coverage that runs both `tup generate` + generated script and `tup build-debug` to confirm artifacts remain confined to `src/build/` and `src/build-debug/`.
+4. Update tooling (`ci`, `just`, Nix shells) to execute `tup` from `src/build/`, ensuring variant commands still find `build-debug/tup.config` and write outputs to `src/build-debug/build/`.
+5. Add CI coverage that runs both `tup generate` + generated script and `tup build-debug` to confirm artifacts remain confined to `src/build/` and `src/build-debug/build/`, and document the clean-up requirement after generated runs.
 
 ## Status & Next Steps
 
 - Draft ADR for review (this document).
-- Prototype the directory move on a feature branch to validate path macros and variant compatibility.
-- Once validated, mark this ADR **Accepted** and execute the implementation plan alongside automated regression coverage.
+- Prototype the directory move on a feature branch to validate the staged layout, confirm both `tup` and `tup generate` succeed, and measure the clean-up workflow required after generated runs.
+- Once validated, mark this ADR **Accepted**, land the staging refactor, update the documentation (including the post-`tup.sh` cleaning steps), and complete the implementation plan alongside automated regression coverage.

tup-build-staging-implementation-plan.md

@@ -8,41 +8,41 @@ This plan tracks the work required to implement ADR 0006 (“Stage Tup Builds Un
 
 1. **Create the staging layout**
    - Add `src/build/` with placeholders for every directory that currently hosts a `Tupfile`.
-   - Move the root `Tupfile` and each subordinate `Tupfile` into the mirrored locations (e.g., `src/build/frontend/Tupfile`), leaving `src/Tupfile.ini` anchored at the project root and keeping `Tuprules.tup` wherever is most practical.
-   - Ensure the Git history retains only the relocated files (remove the old copies rather than duplicating them), and provide a `src/Tupfile` shim that delegates into `src/build/Tupfile` so existing entry points keep working.
+   - Move the root `Tupfile` and each subordinate `Tupfile` into the mirrored locations (e.g., `src/build/frontend/Tupfile`), leaving `Tupfile.ini` anchored at the project root and keeping `Tuprules.tup` wherever is most practical.
+   - Ensure the Git history retains only the relocated files (remove the old copies rather than duplicating them). `tup` now runs directly from `src/build/`, so no compatibility shim in `src/Tupfile` is required.
 
 2. **Shared path configuration**
-   - Introduce `SRC_ROOT`, `BUILD_ROOT`, and `VARIANT_ROOT` variables in `src/build/Tuprules.tup`.
-   - Replace hard-coded relative paths (e.g., `../bin/...`) with macros so rules are robust after the move.
-   - Verify all macros expand correctly under both the default staging run and when `CONFIG_*` values switch for variants (`build-debug`).
+   - Retain per-file `SRC_DIR` variables for referencing source directories; no additional global macros (`SRC_ROOT`, `BUILD_ROOT`, `VARIANT_ROOT`) are necessary.
+   - Review staged Tupfiles to confirm outputs route into the staging tree using the existing `SRC_DIR` pattern.
 
 3. **Output confinement**
-   - Adjust commands so intermediate and final outputs default to `$(BUILD_ROOT)` (or a subdirectory) instead of landing inside the source tree.
-   - Confirm resource-copy rules (`!tup_preserve`, `!cp`, etc.) still put assets in the expected staging paths and that downstream consumers read from the new locations.
+   - Ensure commands emit intermediate and final outputs into `src/build/` (for live runs) or `src/build-debug/build/` (for variant builds) instead of the source tree.
+   - Confirm resource-copy rules (`!cp_preserve`, etc.) still put assets in the expected staging paths and that downstream consumers read from the new locations.
 
 ## Part 2 – Preserve Variant Behaviour
 
 1. **Variant path wiring**
    - Audit references to `build-debug` (and other variant directories) to ensure they resolve relative to the new staging root.
-   - Update scripts or rules that assume the old `src/` root when reading `tup.config` or variant outputs.
+   - Update scripts or rules that assume the old `src/build-debug/bin` layout so they target `src/build-debug/build/**` instead.
 
 2. **Regression tests**
    - Run `tup build-debug` in the new layout and compare the directory structure of `src/build-debug/` against a baseline (focus on executables in `bin/`, JS bundles, and resource copies).
-   - Capture discrepancies and update rules or macros until parity is achieved.
+   - Capture discrepancies and update rules until parity is achieved; ensure the `build-debug/build` subtree contains the expected artifacts.
 
 3. **Generated script validation**
-   - Run `tup generate tup.sh` inside `src/build/`, execute the script, and assert that all files it produces stay within `src/build/` (aside from the intentional `build-debug` tree).
+   - Run `tup generate tup.sh` inside `src/build/`, execute the script, and confirm the outputs stay within the staging area (`src/build/`) and `src/build-debug/build/`.
+   - Document the required clean-up before returning to the live monitor: `cd src/build && git clean -fx .` followed by `cd ../build-debug && git clean -fx .`.
    - Add automated checks (e.g., a CI script diffing `git status`) to prevent regressions.
 
 ## Part 3 – Tooling & Automation Updates
 
 1. **CI adjustments**
-   - Simplify `ci/build/dev.sh` to operate entirely within `src/build/`, removing the temporary `src/links` relocation and limiting cleanups to `src/build/` and `src/build-debug/`.
-   - Update any other CI jobs that run `tup` or expect artifacts in `src/`.
+   - Simplify `ci/build/dev.sh` to operate entirely within `src/build/`, removing the temporary `src/links` relocation and limiting cleanups to `src/build/` and `src/build-debug/build/`.
+   - Update any other CI jobs that run `tup` or expect artifacts in `src/`, including the post-`tup.sh` cleaning sequence.
 
 2. **Developer workflows**
-   - Modify `justfile` recipes, Nix shell hooks (`nix/shells/main.nix`, `armShell.nix`), and helper scripts so they `cd src/build` before invoking `tup`.
-   - Double-check that commands such as `tup monitor -a` behave identically in the new location.
+   - Modify `justfile` recipes, Nix shell hooks (`nix/shells/main.nix`, `armShell.nix`), and helper scripts so they `cd src/build` before invoking `tup` and expect outputs in `src/build-debug/build/`.
+   - Double-check that commands such as `tup monitor -a` behave identically in the new location after cleaning.
 
 3. **Git hygiene**
    - Add `src/build/.gitignore` to ignore generated scripts, `.tup` state directories, and transient outputs while keeping committed `Tupfile`s tracked.
@@ -52,14 +52,15 @@ This plan tracks the work required to implement ADR 0006 (“Stage Tup Builds Un
 
 1. **Contributor docs**
    - Update build instructions (`README.md`, `docs/book`, onboarding guides) to reference `src/build/` as the entry point for `tup` workflows.
-   - Highlight the difference between the staging area (`src/build/`) and variant outputs (`src/build-debug/`) to avoid confusion.
+   - Highlight the difference between the staging area (`src/build/`) and variant outputs (`src/build-debug/build/`) to avoid confusion.
+   - Document the clean-up sequence required after running a generated script so developers can return to live `tup` runs without residual artifacts.
 
 2. **Knowledge base**
    - Refresh `.agents/codebase-insights.txt` (and any other internal notes) to capture the new layout.
 
 3. **Post-migration cleanup**
-   - Remove temporary workarounds that are no longer necessary (e.g., scripts assuming outputs land in `src/`).
-   - Monitor developer feedback during the first few sprints and adjust macros or documentation as needed.
+   - Remove temporary workarounds that are no longer necessary (e.g., scripts assuming outputs land in `src/` or `src/build-debug/bin`).
+   - Monitor developer feedback during the first few sprints and adjust documentation or supporting scripts as needed.
 
 ---
 

tup-build-staging-status.md

@@ -1,11 +1,15 @@
 # Tup Build Staging Refactor – Status
 
 ## Completed
-- Step 1: Created the `src/build/` staging layout, moved the root Tupfile/Tuprules plus crate-specific Tupfiles, added the `src/Tupfile` shim, and kept `Tupfile.ini` rooted in `src/`.
+- Relocated the entire Tup graph to `src/build/`, with `Tupfile.ini` still rooted in `src/`; `tup` now runs directly from the staging tree and no `src/Tupfile` shim is needed.
+- Standardized on per-file `SRC_DIR` handling for relocated Tupfiles, added `!cp_preserve` plus `src/build/.gitignore`, and refreshed `.agents/codebase-insights.txt`; no further Tupfile edits are planned.
+- Validated the build: both `tup` and `tup generate` succeed, `just build` / `just build-once` work end-to-end, and their outputs land under `src/build/` or `src/build-debug/build/` as intended.
 
 ## In Progress
-- Step 2: Updating relocated Tupfiles to pull inputs from the original source tree and emit outputs inside `src/build/`. Added `include_rules` to every staged Tupfile, introduced a `!cp_preserve` helper, and rewrote asset copies to use explicit relative paths instead of `!tup_preserve`. Need to validate with `tup` runs (current environment blocks user namespaces) and adjust paths based on the results.
+- Update tooling, scripts, and documentation (CI jobs, `justfile`, Nix shells, onboarding guides) to reference `src/build-debug/build/**` and reflect the new staging entry point.
+- Capture the operational caveat that a generated script populates `src/build/`; developers must run `cd src/build && git clean -fx .` followed by `cd ../build-debug && git clean -fx .` before returning to live `tup` runs.
 
 ## Next
-- Step 2: Validate the updated rules by running `tup` (default + `build-debug`) once user-namespace restrictions are lifted, and fix any remaining path issues that show up.
-- Step 3: Refresh tooling, documentation, and CI scripts to rely on the staging tree and ensure `tup generate` produces artifacts only under `src/build/` and the existing variant directories.
+- Sweep remaining references to `src/build-debug/bin` (arm shell, docs, helper scripts) and switch them to `src/build-debug/build/**`.
+- Document the post-`tup.sh` clean-up sequence across contributor docs and CI scripts, adding automation where possible to enforce a clean staging area.
+- Verify CI and developer tooling run successfully with the updated paths and documented clean-up, then mark ADR 0006 as accepted.