fix(cache): a package's object layout must not depend on the consumer (#344) (#345)

* fix(cache): a package's object layout must not depend on the consumer (#344)

A dependency's objects live in the global build cache under a key that
deliberately excludes the consuming project — that exclusion is what makes an
entry shareable across projects. Their LAYOUT inside the entry did not exclude
it: the address was the consumer's build-dir path with `obj/` stripped, and
build-dir object paths are decided by #233's basename disambiguation, whose
census spans the WHOLE build directory. So `compat.zlib`'s compress.o was

    obj/compress.o                          alone
    obj/compat_zlib/zlib-1.3.2/compress.o   beside compat.bzip2

under one key. Whichever project ran second asked the entry for a path the first
had never written, and ninja rejected the graph before running any command:

    ninja: error: '<cache>/…/obj/compress.o', needed by 'obj/compress.o',
           missing and no known rule to make it

one line after the CLI printed `Cached compat.zlib v1.3.2 (15 units)`.
32 of 47 mcpp-index workspace members across three platforms.

#233 (compile edges collided), #240 (link inputs didn't follow the rename) and
this are three products of one machine: a layout decided by a global census.
The fix is not a fourth place to keep in sync — it is to take dependencies out
of the census.

* plan.cppm now emits TWO addresses from ONE derivation: `object` (build dir)
  and `packageObjectRel` (inside a cache entry, a pure function of the owning
  package). Dependency objects go to `obj/<pkg-slug>/<mirrored relDir>/<name>.o`
  unconditionally — cross-package collisions are structurally impossible, so
  dependencies need no census at all. The root project, never cached, keeps its
  flat `obj/<name>.o` and now censuses only its own sources.
* prepare.cppm's `object_cache_path` — the second derivation — is deleted,
  along with its `filename()` fallback, which silently mapped distinct sources
  onto one entry address.
* `is_cached(key)` validated the entry's OWN file list while the caller went on
  to read addresses it computed itself; the two answers were never compared.
  It is now `probe_cached(key, requested)`. Any divergence is a MISS, never a
  build failure, and is reported — a systematic recurrence would otherwise show
  up only as "the cache never hits", with no signal at all.
* Cacheability is judged by where the payload is on disk, not by the
  `sourceKind` label. Multi-version mangling re-anchors a consumer package at
  `target/.mangled/…` and rewrites its sources while keeping that label; it
  stays out of the cache today only because axis F happens to differ.
  Containment is judged LEXICALLY (`path_is_under_any`): stores built out of
  symlinks are ordinary, and canonicalizing drops every such package out of the
  cache silently.
* All-or-nothing per package: one unaddressable unit takes the whole package
  out, rather than leaving it half staged.
* kCacheEpoch 1 → 2 (the artifact layout changed) and `cache verify` now
  reports obj addresses that escape their entry.

Tests: unit ObjectAddress pins "a dependency's addresses are immune to the rest
of the graph"; e2e 184 runs both orderings and asserts ZERO compile edges for
the reused dependency — asserting only "it builds" would be satisfied by a
build that quietly degraded every hit to a miss. e2e 123's assertion is
generalized from "the consumer's main must be renamed" (the shape of #240's
first fix) to "every link input is produced by some edge" (the property).

* fix(build): link through a response file on every platform, not just Windows

Discovered by running the real mcpp-index workspace against the #344 branch:
`opencv-module`, `opencv-module-dnn` and `opencv-module-unifont` all died with

    ninja: fatal: posix_spawn: Argument list too long

naming no edge, no file and no cause. #344 lengthened dependency object paths
(they now carry a per-package directory) and that was enough to cross a ceiling
nothing was watching.

The rule emitter said rsp was needed only where commands spawn through
CreateProcess, and that "POSIX driver-style keeps the inline form: ARG_MAX is
ample". Both halves are wrong. ninja runs `sh -c "<whole command>"` on POSIX, so
the command is a SINGLE argv entry and the limit is MAX_ARG_STRLEN — 32 pages,
128 KiB — not the 2 MiB ARG_MAX anyone would think to check. And it was never
ample: measured on opencv-module, the inline link line was already 56 840 bytes
before this branch, 43% of the ceiling. With #344's paths it reached 161 687.

So: rsp always, every platform, every dialect. clang/gcc drivers, link.exe, GNU
ar and llvm-ar all accept @rspfile, so this is one rule shape instead of two,
and the response file next to the output reads better than a 160 KB command line
anyway. A build system may not have a maximum project size that it discovers by
crashing, and "how long is this command" must not be something anyone carries in
their head when choosing an object path.

The unit test asserted the falsified premise for POSIX ("must NOT use rspfile");
it now asserts the invariant on every platform, plus that `$in` appears exactly
once per rule — as rspfile_content, never inlined into the command.

Note this was invisible to all 18 CI jobs: none of them builds an
opencv-class package. CI has no large-link coverage at all.

Author: Sunrisepeak

.agents/docs/2026-08-03-issue344-cache-object-address-design.md

@@ -3,6 +3,35 @@
 > 本文件追踪 `mcpp-community/mcpp` 公开仓的版本演进。
 > 格式参考 [Keep a Changelog](https://keepachangelog.com/zh-CN/1.1.0/)。
 
+## [2026.8.3.4] — 2026-08-03
+
+### 修复
+
+- **全局 build cache:同一个 key 下第二个消费者必挂 `missing and no known rule to make it`(#344)。** 一个依赖的 `.o` 存在 cache 条目里的**路径**,此前取自消费方 build dir 的相对路径。而 build dir 里的对象布局由 #233 的 basename 消歧决定,消歧的判据是一次**跨越整个 build dir**的普查 —— 也就是说,它取决于消费方还拉了哪些别的包:
+
+  - 同时拉了 `compat.zlib` 与 `compat.bzip2`(两者上游各有一个 `compress.c`)→ `obj/compat_zlib/zlib-1.3.2/compress.o`
+  - 只拉了 `compat.zlib` → `obj/compress.o`
+
+  cache key **刻意不含消费方**(这正是跨工程共享成立的前提),于是两种布局落进同一个条目,后跑的那个工程按自己的布局去取,必然缺一个文件。失败发生在 ninja 的 **graph 加载**阶段 —— 一条命令都还没跑,上一行却刚打印过 `Cached … (15 units)`。mcpp-index 全量 workspace 在三个平台上共 32 个成员因此失败。
+
+  条目内的对象地址现在是**包自身的纯函数**:镜像源文件相对它**自己**包根的路径,不含任何消费方信息。构建目录里,依赖包的对象一律落在 `obj/<pkg-slug>/…` 下 —— 跨包撞名结构性地不可能发生,依赖包因此完全退出消歧普查。根工程(永不入 cache)保持沿用至今的扁平 `obj/<name>.o`。
+
+  #233(编译边撞名)、#240(链接输入未跟改名)与本条是同一台机器的三个产物:**布局由一次全局普查决定**。所以修的不是再补一处同步,而是把依赖包从普查里彻底拿掉。
+
+- **链接/归档命令一律走 response file,不再有「项目大到一定程度就崩」的隐形上限。** 此前只有 Windows(CreateProcess 32 KiB)与 msvc 方言用 rspfile,POSIX 走内联 `$in`,理由写的是「ARG_MAX 很宽裕」。两半都错:ninja 在 POSIX 上是 `sh -c "<整条命令>"`,整条命令是**一个 argv 项**,撞的是 `MAX_ARG_STRLEN`(32 页 = 128 KiB)而不是 2 MiB 的 `ARG_MAX`;而且它从来就不宽裕 —— 实测 mcpp-index 的 `opencv-module`,内联链接行**本来就已经 56840 字节**,占那条无人看守的上限的 43%。
+
+  上面 #344 让依赖对象路径变长(多一层包目录),同一条边到了 161687 字节,于是 ninja 直接 `ninja: fatal: posix_spawn: Argument list too long` —— **不报是哪条边、哪个文件、什么原因**。构建系统不能有一个「靠崩溃才被发现的项目规模上限」,「这条命令有多长」也不应该是选对象路径时需要有人记在脑子里的事。clang/gcc driver、link.exe、GNU ar、llvm-ar 全都认 `@rspfile`,现在全平台一个规则形态。
+
+### 改进
+
+- **cache 条目与本次构建的布局分歧,现在降级为 miss 并明确报告,而不是让 ninja 崩在图加载阶段。** `is_cached` 此前校验的是条目**自述的**文件表,而消费方随后按**自己算的**地址去取 —— 两处独立推导,从不比对。命中判据现在校验「本次实际要读的那批产物」,任何不匹配都只是一次重编。同时新增一行 warning:一个系统性的分歧否则会表现为「cache 永远不命中」而毫无信号,这正是 v2026.7.30.2 之前那个假 `Cached` 骗了三个月的失败模态。
+
+- **可缓存性改判磁盘出处,不再只认标签。** 规则一直写着「无法证明来自不可变的 xpkgs store 就不准入」,但代码实现的是更弱的代理(`sourceKind == "version"`)。多版本共存(mangling)会把消费方包的根重锚到 `<project>/target/.mangled/…` 并**改写其源码**,而标签仍是 `"version"` —— 它今天不出错只靠轴 F 侥幸。现在按包根的实际位置判定。同一批还加了「全有全无」:任何一个单元拿不到与机器无关的条目地址,整个包退出缓存,不留下半 staged 的包(那会表现为三条边之后的 BMI CRC mismatch)。
+
+- **`mcpp cache verify` 现在会报告逃出条目的对象地址。** 让「条目地址必须是包内相对路径」这条不变量可以离线审计,而不是只能通过复现一次双工程构建才看得见。
+
+- **`kCacheEpoch` 1 → 2。** 产物布局变了,旧条目描述的是本版本不会去要的布局。它们本来就会被判为 miss,但让两套布局共用一个目录会让 `cache gc` 的体积统计和 `cache verify` 的输出失去意义。用户侧表现为一次全量重建,无需任何手工步骤。
+
 ## [2026.8.3.3] — 2026-08-03
 
 ### 修复

CHANGELOG.md

@@ -1,6 +1,6 @@
 [package]
 name        = "mcpp"
-version     = "2026.8.3.3"
+version     = "2026.8.3.4"
 description = "Modern C++ build & package management tool"
 license     = "Apache-2.0"
 authors     = ["mcpp-community"]

mcpp.toml

@@ -5,7 +5,16 @@
 //   (the cache root is $MCPP_HOME/build-cache/v1 — see mcpp.home::cache_root)
 //     entry.json                       sentinel + self-description + file list
 //     bmi/<module>.{gcm,pcm}
-//     obj/<relative>.o
+//     obj/<package-internal path>.o
+//
+// The obj address is PACKAGE-INTERNAL: it mirrors the source's path relative to
+// its own package root and contains nothing about the consuming project
+// (mcpp#344). It used to be the consumer's build-dir path with `obj/` stripped,
+// which made the layout depend on which OTHER packages the consumer happened to
+// pull in — #233's basename disambiguation is triggered by a census over the
+// whole build dir — while the key deliberately excludes the consumer. Two
+// consumers then wrote and read incompatible layouts under one key and the
+// second one died in ninja's graph phase.
 //
 // <key16> comes from mcpp.build.cache_key: a per-package Merkle key over the
 // toolchain, the language/dialect settings, the resolved profile, the package's
@@ -75,16 +84,63 @@ struct CacheKey {
     std::filesystem::path objDir()    const { return dir() / "obj"; }
 };
 
-// File names (basenames for BMIs, output-relative paths for objects) belonging
-// to one package's cache entry.
+// One cached object file. The two addresses are deliberately separate
+// (mcpp#344):
+//
+//   cacheRel — where it lives INSIDE the entry (`<entry>/obj/<cacheRel>`).
+//              Must be a pure function of the package, because the key
+//              deliberately excludes the consumer. plan.cppm derives it.
+//   buildRel — where THIS build produces it (`<buildDir>/<buildRel>`).
+//              Consumer-side and therefore not recordable: two consumers of
+//              one entry may legitimately place the same object at different
+//              build-dir paths.
+//
+// Collapsing the two — recording the consumer's path as the entry's address —
+// is exactly what made the second consumer of an entry fail with ninja's
+// "missing and no known rule to make it". Only `cacheRel` ever reaches
+// entry.json; `buildRel` is populate-time input and is empty on read-back.
+struct ObjArtifact {
+    std::string           cacheRel;
+    std::filesystem::path buildRel;
+};
+
+// The artifacts belonging to one package's cache entry: BMI basenames plus the
+// objects above.
 struct DepArtifacts {
-    std::vector<std::string> bmiFiles;
-    std::vector<std::string> objFiles;
+    std::vector<std::string>  bmiFiles;
+    std::vector<ObjArtifact>  objFiles;
 };
 
-// True when entry.json exists, its schema matches, its recorded inputs equal
-// `key.inputs` field for field, and every listed file is present on disk.
-bool is_cached(const CacheKey& key);
+// Why an entry could not serve this build.
+struct CacheProbe {
+    bool ok = false;
+    // Non-empty ONLY when the entry itself validated (schema, key, inputs) but
+    // does not carry the artifacts THIS build asked for. After mcpp#344 that
+    // shape should be unreachable, which is precisely why it must be reported
+    // rather than silently folded into "miss": a systematic recurrence would
+    // otherwise present as "the cache simply never hits", with no signal at
+    // all — the same failure mode as the fake `Cached` that went unnoticed for
+    // three months.
+    std::vector<std::string> layoutMismatch;
+};
+
+// Validate an entry AGAINST WHAT THIS BUILD WILL ACTUALLY READ.
+//
+// A hit requires all of: entry.json exists, its schema matches, its recorded
+// key matches, its recorded inputs equal `key.inputs` field for field, and
+// every artifact in `requested` is both listed by the entry and present on
+// disk. Checking only the entry's OWN file list — which is what this used to
+// do — validates a different question than the one the caller goes on to ask,
+// and the two answers diverged the moment the object layout stopped being a
+// function of the package alone.
+//
+// Anything short of a full match is a MISS. This function must never be the
+// reason a build fails: an unusable entry costs a recompile, and the staging
+// edges that would read it are never emitted.
+CacheProbe probe_cached(const CacheKey& key, const DepArtifacts& requested);
+
+// probe_cached(...).ok
+bool is_cached(const CacheKey& key, const DepArtifacts& requested);
 
 // The artifact list of a validated entry. Does NOT copy anything: the ninja
 // backend stages cached files through its own `stage_file` edges, so that a
@@ -137,12 +193,15 @@ std::optional<nlohmann::json> read_entry(const std::filesystem::path& p) {
     return j;
 }
 
+// Read-back fills `cacheRel` only: `buildRel` is consumer-side and is not — and
+// must not be — recorded in the entry.
 DepArtifacts artifacts_from(const nlohmann::json& j) {
     DepArtifacts a;
     if (auto it = j.find("bmi"); it != j.end() && it->is_array())
         for (auto& v : *it) if (v.is_string()) a.bmiFiles.push_back(v.get<std::string>());
     if (auto it = j.find("obj"); it != j.end() && it->is_array())
-        for (auto& v : *it) if (v.is_string()) a.objFiles.push_back(v.get<std::string>());
+        for (auto& v : *it)
+            if (v.is_string()) a.objFiles.push_back({v.get<std::string>(), {}});
     return a;
 }
 
@@ -199,21 +258,39 @@ std::filesystem::path cached_obj_path(const CacheKey& key, std::string_view rel)
     return key.objDir() / std::filesystem::path(std::string(rel));
 }
 
-bool is_cached(const CacheKey& key) {
+CacheProbe probe_cached(const CacheKey& key, const DepArtifacts& requested) {
+    CacheProbe probe;
     auto j = read_entry(key.entryFile());
-    if (!j) return false;
-    if (j->value("schema", 0) != kEntrySchema) return false;
-    if (j->value("key", std::string{}) != key.keyHex) return false;
+    if (!j) return probe;
+    if (j->value("schema", 0) != kEntrySchema) return probe;
+    if (j->value("key", std::string{}) != key.keyHex) return probe;
     auto it = j->find("inputs");
-    if (it == j->end() || !inputs_match(*it, key.inputs)) return false;
+    if (it == j->end() || !inputs_match(*it, key.inputs)) return probe;
+
+    // The entry itself is valid. From here on, every remaining check is about
+    // whether it holds what THIS build is going to read.
+    auto recorded = artifacts_from(*j);
+    std::set<std::string> haveBmi(recorded.bmiFiles.begin(), recorded.bmiFiles.end());
+    std::set<std::string> haveObj;
+    for (auto& o : recorded.objFiles) haveObj.insert(o.cacheRel);
 
-    auto arts = artifacts_from(*j);
     std::error_code ec;
-    for (auto& g : arts.bmiFiles)
-        if (!std::filesystem::exists(cached_bmi_path(key, g), ec)) return false;
-    for (auto& o : arts.objFiles)
-        if (!std::filesystem::exists(cached_obj_path(key, o), ec)) return false;
-    return true;
+    for (auto& g : requested.bmiFiles) {
+        if (!haveBmi.contains(g)
+            || !std::filesystem::exists(cached_bmi_path(key, g), ec))
+            probe.layoutMismatch.push_back(g);
+    }
+    for (auto& o : requested.objFiles) {
+        if (!haveObj.contains(o.cacheRel)
+            || !std::filesystem::exists(cached_obj_path(key, o.cacheRel), ec))
+            probe.layoutMismatch.push_back(o.cacheRel);
+    }
+    probe.ok = probe.layoutMismatch.empty();
+    return probe;
+}
+
+bool is_cached(const CacheKey& key, const DepArtifacts& requested) {
+    return probe_cached(key, requested).ok;
 }
 
 std::expected<DepArtifacts, std::string> resolve_cached(const CacheKey& key) {
@@ -250,7 +327,6 @@ populate_from(const CacheKey& key,
     std::filesystem::create_directories(cacheObj, ec);
 
     auto projectBmi = projectTargetDir / key.bmiDirName;
-    auto projectObj = projectTargetDir / "obj";
 
     for (auto& g : arts.bmiFiles) {
         auto from = projectBmi / g;
@@ -263,15 +339,20 @@ populate_from(const CacheKey& key,
                 "populate bmi '{}': {}", g, ec.message()));
         }
     }
+    // Read from `buildRel`, write at `cacheRel`. These are NOT the same path in
+    // general (mcpp#344): the build-dir layout partitions objects by package,
+    // the entry's layout is package-internal, and a source that sits outside its
+    // package root is re-anchored for the entry. Deriving one from the other
+    // here is what this split exists to prevent.
     for (auto& o : arts.objFiles) {
-        auto from = projectObj / o;
-        if (!std::filesystem::exists(from)) {
+        auto from = projectTargetDir / o.buildRel;
+        if (o.buildRel.empty() || !std::filesystem::exists(from)) {
             return std::unexpected(std::format(
                 "expected build output missing: {}", from.string()));
         }
-        if (!copy_one(from, cached_obj_path(key, o), ec)) {
+        if (!copy_one(from, cached_obj_path(key, o.cacheRel), ec)) {
             return std::unexpected(std::format(
-                "populate obj '{}': {}", o, ec.message()));
+                "populate obj '{}': {}", o.cacheRel, ec.message()));
         }
     }
 
@@ -289,7 +370,13 @@ populate_from(const CacheKey& key,
     j["tag"]      = key.manifestTag;
     j["inputs"]   = key.inputs;
     j["bmi"]      = arts.bmiFiles;
-    j["obj"]      = arts.objFiles;
+    // Only the entry-internal addresses. Recording the consumer's build path
+    // here is mcpp#344 in one line.
+    {
+        auto objs = nlohmann::json::array();
+        for (auto& o : arts.objFiles) objs.push_back(o.cacheRel);
+        j["obj"] = std::move(objs);
+    }
     j["accessed"] = now_iso8601();
     return write_entry(key.entryFile(), j);
 }

src/bmi_cache.cppm

@@ -177,9 +177,24 @@ void check_pkg_files(Entry& e, const nlohmann::json& j) {
     if (auto it = j.find("obj"); it != j.end() && it->is_array()) {
         for (auto& v : *it) {
             if (!v.is_string()) continue;
-            if (missing(e.dir / "obj" / v.get<std::string>())) {
+            auto rel = v.get<std::string>();
+            // mcpp#344: an entry's obj addresses must be package-internal —
+            // downward, relative, no drive letter. An address that escapes the
+            // entry is one that was derived from some consumer's build tree,
+            // which is precisely the defect that made two consumers of one key
+            // disagree about the layout. Report it here so the invariant is
+            // auditable offline rather than only observable as a build that
+            // dies in ninja's graph phase.
+            if (rel.empty() || rel.starts_with("/") || rel.starts_with("..")
+                || rel.find(':') != std::string::npos) {
                 e.complete = false;
-                e.problem = std::format("missing obj/{}", v.get<std::string>());
+                e.problem = std::format(
+                    "obj address is not package-internal: '{}'", rel);
+                return;
+            }
+            if (missing(e.dir / "obj" / rel)) {
+                e.complete = false;
+                e.problem = std::format("missing obj/{}", rel);
                 return;
             }
         }

src/bmi_cache/maintenance.cppm

@@ -64,7 +64,13 @@ export namespace mcpp::build::cache_key {
 // Deliberately NOT the mcpp release number: folding the whole version in
 // orphaned every entry on every release, including plain C object files whose
 // validity has nothing to do with mcpp's version.
-inline constexpr int kCacheEpoch = 1;
+// 2 (mcpp#344): the artifact layout changed. An entry's obj addresses are now
+// package-internal instead of "the first consumer's build-dir path minus
+// `obj/`", so entries written by an older mcpp describe a layout this one does
+// not ask for. They would all miss anyway (probe_cached compares the REQUESTED
+// artifacts), but sharing a directory between two layouts makes `cache gc`'s
+// size accounting and `cache verify`'s output meaningless.
+inline constexpr int kCacheEpoch = 2;
 
 // Axes A/B/C — identical for every package in one build, computed once.
 struct BuildAxes {

src/build/cache_key.cppm

@@ -749,15 +749,33 @@ std::string emit_ninja_string(const BuildPlan& plan) {
     // link/archive command; revisit the first-match replace if a dialect
     // ever grows another).
     //
-    // rsp is used when the command spawns through CreateProcess (32 KiB
-    // command-line ceiling): always for the separate-linker msvc dialect,
-    // and on Windows for driver-style too (#247 — ffmpeg/opencv-class
-    // packages link thousands of objects; clang/gcc drivers and GNU/llvm ar
-    // all accept @rspfile). POSIX driver-style keeps the inline form
-    // byte-identical: ARG_MAX is ample and the plain command is easier to
-    // reproduce by hand.
+    // rsp is used ALWAYS, on every platform and every dialect. The objects of
+    // one link edge are unbounded — an ecosystem package like opencv or ffmpeg
+    // contributes thousands — and every way of spawning a command has a ceiling:
+    //
+    //   Windows   CreateProcess, 32 KiB command line (#247)
+    //   POSIX     ninja spawns `sh -c "<whole command>"`, so the command is a
+    //             SINGLE argv entry and hits MAX_ARG_STRLEN — 32 pages, 128 KiB
+    //             — long before the 2 MiB ARG_MAX anyone would think to check.
+    //
+    // This used to read "POSIX keeps the inline form; ARG_MAX is ample and the
+    // plain command is easier to reproduce by hand". Both halves were wrong.
+    // ARG_MAX is the wrong limit, and it was never ample: measured on
+    // mcpp-index's opencv-module, the inline link line was already 56 840 bytes
+    // — 43% of a ceiling nothing was watching. mcpp#344 lengthened dependency
+    // object paths (they now carry a per-package directory) and the same edge
+    // reached 161 687 bytes, at which point ninja dies with
+    //
+    //     ninja: fatal: posix_spawn: Argument list too long
+    //
+    // naming no edge, no file and no cause. A build system may not have a
+    // maximum project size that it discovers by crashing, and "how long is this
+    // command" must not be a thing anyone has to keep in their head when
+    // choosing an object path. clang/gcc drivers, link.exe, GNU ar and llvm-ar
+    // all accept @rspfile, so there is one rule shape everywhere; the response
+    // file sits next to the output and `cat`ing it beats reading a 160 KB line.
     {
-        const bool useRsp = separateLinker || mcpp::platform::is_windows;
+        constexpr bool useRsp = true;
         auto link_rule = [&](std::string_view name, std::string cmd,
                              std::string_view desc) {
             append(std::format("rule {}\n", name));