|
| 1 | +# Dalec Architecture |
| 2 | + |
| 3 | +## Goals and Scope |
| 4 | +Dalec provides a declarative workflow for building operating system packages and container images from a single spec. The repository couples a custom [BuildKit](https://github.com/moby/buildkit) frontend with distro-specific build logic, packaging templates, signing, and test automation. This document sketches the high-level architecture so maintainers and contributors understand how the moving parts fit together. |
| 5 | + |
| 6 | +:::note |
| 7 | +This document is a work in progress. |
| 8 | +If you see a mistake or missing information please open an issue or ping us on [Slack](https://cloud-native.slack.com/archives/C09MHVDGMAB). |
| 9 | +::: |
| 10 | + |
| 11 | +## System View |
| 12 | +```mermaid |
| 13 | +flowchart TB |
| 14 | + classDef component fill:#0f172a,stroke:#1e293b,color:#e2e8f0,rx:6,ry:6,text-align:left; |
| 15 | + classDef store fill:#111827,stroke:#1e293b,color:#f8fafc,rx:6,ry:6,text-align:left; |
| 16 | +
|
| 17 | + BKClient["BuildKit client<br/>(docker, buildx)"]:::component |
| 18 | + BKDaemon["Buildkit Service<br/>(dockerd, buildkitd)"]:::component |
| 19 | + Frontend["cmd/frontend (gRPC)<br/>- plugin loader<br/>- BuildMux router<br/>- subrequests"]:::component |
| 20 | + Targets["targets / packaging<br/>- distro configs<br/>- package/image flow"]:::component |
| 21 | + Core["dalec core package<br/>- spec parsing<br/>- source fetching<br/>- signing & tests APIs"]:::component |
| 22 | +
|
| 23 | + BKClient --> BKDaemon |
| 24 | + BKDaemon --> BKClient |
| 25 | + BKDaemon --> Frontend |
| 26 | + Frontend --> BKDaemon |
| 27 | + Frontend --> Targets |
| 28 | + Targets --> Core |
| 29 | + Core --> BKDaemon |
| 30 | +``` |
| 31 | + |
| 32 | +## Runtime Components |
| 33 | +- **BuildKit frontend (`cmd/frontend/main.go`)** |
| 34 | + Runs inside BuildKit, exposes a gRPC service via `grpcclient.RunFromEnvironment`, and handles build requests. Subcommands handle credential helper and test utilities reused in integration flows. |
| 35 | + |
| 36 | +- **Frontend orchestration (`frontend/`)** |
| 37 | + Implements spec loading (`build.go`), request routing (`mux.go`), BuildKit UI integration, signing hooks, and test execution. `BuildMux` acts like an HTTP router: targets register handlers which later receive trimmed build target prefixes. |
| 38 | + |
| 39 | +- **Target plugins (`targets/`, `packaging/`)** |
| 40 | + Target-specific packages (Debian, Ubuntu, RPM variants, Windows) register `gwclient.BuildFunc` handlers via a lightweight plugin framework (`internal/plugins`). Each handler knows how to build distro packages, assemble container images, and, where relevant, system extensions. |
| 41 | + |
| 42 | +- **Tooling commands (`cmd/*`)** |
| 43 | + Helper binaries provide linting, schema generation, BuildKit worker matrices, registry retagging, and log conversion for CI. These commands run outside BuildKit. |
| 44 | + |
| 45 | +- **Shared libraries (`dalec`, `internal`, `sessionutil`, `pkg`)** |
| 46 | + The root Go module exposes the spec model, source handling, cache helpers, packaging utilities, and test support. Internal packages hide pluggable subsystems (`internal/plugins`, `internal/testrunner`) from external consumers. |
| 47 | + |
| 48 | +## Build Orchestration Flow |
| 49 | +1. **BuildKit invokes the frontend** with a build context containing a Dalec spec (the spec is the effective `Dockerfile` for the custom frontend). |
| 50 | +2. **`BuildMux` resolves the target**: exact match, default handler, or prefix routing (e.g. `mariner2/container`). The matched handler sees the shortened target and a `dalec.target` build option that identifies the active spec target stanza. |
| 51 | +3. **Specs load once per build** through `LoadSpec`, which reads the spec from the Docker build context, merges build-arg substitutions (including derived platform arguments), and returns a `*dalec.Spec`. |
| 52 | +4. **`BuildWithPlatform` fans out per platform** requested by the client, invoking a target handler for each architecture. Handler closures return an image reference and optional `dalec.DockerImageSpec` metadata. |
| 53 | +5. **Handlers construct LLB graphs** using helpers from the `dalec` package to fetch sources, apply patches, run build steps, and generate artifacts (packages, root filesystems, signatures). |
| 54 | +6. **Results finalize** via BuildKit: container images are pushed or exported, packages and auxiliary files are returned as BuildKit outputs, and metadata is attached for the client. |
| 55 | + |
| 56 | +## Specification |
| 57 | +- **Spec model (`spec.go`)** |
| 58 | + Defines the declarative schema for packages: metadata, sources, patches, build steps, artifacts, target overrides, signing, tests, caching, image configuration, and changelog entries. YAML/JSON tags drive linting (`cmd/lint`) and schema generation (`cmd/gen-jsonschema`). |
| 59 | + |
| 60 | +- **Sources (`source.go` & `source_*.go`)** |
| 61 | + Support multiple backing stores (context tarball, Git, HTTP archives, Docker images, inline content, nested builds). `SourceOpts` bundles the BuildKit resolver, credential helpers, and context readers. Generators (Go modules, Cargo, Pip) allow producing dependency caches from source trees. |
| 62 | + |
| 63 | +- **Artifacts (`artifacts.go`, `packaging/linux/*`)** |
| 64 | + Represent build outputs such as OS packages, tarballs, and runtime sysext bundles. Targets interpret the spec’s `Artifacts`, `PackageConfig`, and `Image` sections to decide which packaging templates to apply. |
| 65 | + |
| 66 | +- **Caches & accelerators (`cache.go`)** |
| 67 | + Provide declarative cache mounts for generic directories, Go build caches, Rust `sccache`, and Bazel caches. Options propagate platform and distro context into cache keys. |
| 68 | + |
| 69 | +- **Signing (`request.go`, `package_config.go`)** |
| 70 | + `MaybeSign` stitches signing configuration pulled from spec or build context, optionally forwarding to external signing frontends. Build args such as `DALEC_SKIP_SIGNING` and `DALEC_SIGNING_CONFIG_PATH` control behavior. |
| 71 | + |
| 72 | + |
| 73 | +## Target Implementations and Packaging |
| 74 | +- **Plugin discovery (`cmd/frontend/plugin.go`)** |
| 75 | + Auto-loads targets registered through `targets/plugin/init.go` using the containerd `plugin` graph. Only `build-target` plugins are forwarded to the mux. |
| 76 | + |
| 77 | +- **Distro contracts (`targets/linux/distro_handler.go`)** |
| 78 | + Each distro implements a `DistroConfig` interface covering validation, worker image selection, package build, package extraction, container assembly, and test execution. Common helper flows build packages once and reuse artifacts across container and sysext outputs. |
| 79 | + |
| 80 | +- **Packaging templates (`packaging/linux/*`)** |
| 81 | + Go templated assets render control files (Debian `control`, RPM spec fragments, post-install scripts) using spec data. Generated scripts run inside BuildKit worker containers to produce `.deb`, `.rpm`, or sysext tarballs. |
| 82 | + |
| 83 | +- **Windows support (`targets/windows`)** |
| 84 | + Cross-compilation support builds Windows container images and packages while running on Linux workers, handling platform-specific artifact layout. |
| 85 | + |
| 86 | +- **Caching and reuse (`targets/cache.go`)** |
| 87 | + Targets detect pre-built packages supplied via BuildKit inputs and skip rebuilds when available, reducing rebuild cost in multi-stage workflows. |
| 88 | + |
| 89 | +## Testing and Verification |
| 90 | +- **Declarative tests (`Spec.Tests`)** |
| 91 | + Tests attach to specs or per-target overrides. Steps describe commands, expected output, mounts, and file assertions. |
| 92 | + |
| 93 | +- **Execution engine (`frontend/test_runner.go`)** |
| 94 | + Orchestrates containerized test runs using BuildKit’s LLB API, collects output in an embedded virtual filesystem (`frontend/pkg/bkfs`), and reports structured errors with source mapping. |
| 95 | + |
| 96 | +- **Reusable helpers (`internal/testrunner`)** |
| 97 | + Implements subcommands invoked by the frontend (`testrunner.StepRunnerCmdName`, `testrunner.CheckFilesCmdName`) and Go helpers used in unit tests. This keeps test logic identical between spec execution and CLI tooling. |
| 98 | + |
| 99 | +- **Linting and schema checks (`cmd/lint`, `cmd/gen-jsonschema`)** |
| 100 | + Enforce consistent YAML/JSON field usage, tag coverage, and schema drift between docs and code. CI runs these tools alongside `go test`. |
| 101 | + |
| 102 | +## Tooling and Generated Assets |
| 103 | +- **Generation commands** |
| 104 | + - `cmd/gen-jsonschema`: emits the public JSON schema used by docs. |
| 105 | + - `cmd/gen-source-variants`: maintains the `Source` union struct. |
| 106 | +- **Utility commands** |
| 107 | + - `cmd/retagger`: retags container images (used for publishing). |
| 108 | + - `cmd/test2json2gha`: converts Go test JSON to GitHub Actions annotations. |
| 109 | + |
| 110 | +Generated files live under `_output/` (local builds) or `docs/` (published schema & docs). Always run `go generate ./...` after touching schema-affecting files. |
| 111 | + |
| 112 | +## Extensibility Paths |
| 113 | +- **Add a new target** |
| 114 | + Create a package under `targets/<platform>/<distro>`, implement `DistroConfig`, and register it in `targets/plugin/init.go`. Expose worker images, packaging steps, and tests tailored to the new distro. |
| 115 | + |
| 116 | +- **Introduce a new source type or generator** |
| 117 | + Extend the `Source` type with a new field with type name that starts with `Source` add the implementation under `source_<type>.go`, and regenerate `source_generated.go` (`go generate ./...). |
| 118 | + |
| 119 | +- **Build-time integrations** |
| 120 | + BuildKit subrequests handled in `frontend/mux.go` (`frontend.dalec.resolve`, `frontend.dalec.defaultPlatform`) illustrate how to add new metadata endpoints consumable via `docker buildx --print`. |
| 121 | + |
| 122 | +## Repository Layout Reference |
| 123 | +- `cmd/` – entrypoints for the BuildKit frontend and tooling. |
| 124 | +- `frontend/` – build routing, spec resolution, test runner, request helpers. |
| 125 | +- `targets/` – distro-specific build logic and plugin registration. |
| 126 | +- `packaging/` – template assets and helpers for OS package formats. |
| 127 | +- `dalec` root files (`*.go`) – shared spec, sources, cache, artifact logic. |
| 128 | +- `internal/` – plugin wiring and test runner internals not exported publicly. |
| 129 | +- `sessionutil/` – helpers for BuildKit client sessions. |
| 130 | +- `docs/`, `website/` – published documentation and static site source. |
| 131 | +- `test/` – integration suites exercising full BuildKit builds. |
| 132 | + |
| 133 | +--- |
| 134 | + |
| 135 | +This document is intentionally high-level; detailed API references live alongside code in package comments and GoDoc. Use it as a map when navigating the repository or planning new features. |
0 commit comments