Turn on Cross-Module Incremental Dependencies!

Cross-Module incremental dependencies are a new experimental mode of the Swift driver and frontend. Through a tight partnership between the two, we enable the driver to have far greater visibility into the dependency structure of a Swift module.

Rather than invent a new model, we have chosen to extend the existing incremental compilation model that works for a single module to multiple modules. To do this, we need the frontend to emit Swift dependencies in a form the driver can consume. We could emit these metadata in the form of an extra supplementary output that summarizes the contents of a generated module. However, this approach comes with a number of downsides:

- It requires additional integration with the build system
- It assumes swiftmodule files will be consumed directly from the build directory; they are not
- It incorrectly assumes a swiftmodule has but one interface. Taken in aggregate, a swiftmodule directory has one interface *per triple*

Given this, the approach we take here is to encode these dependencies directly into the swiftmodule file itself. When frontends load these souped-up incremental swiftmodule files, they record in their own swiftdeps files that they depend on an incremental swiftmodule. Upon the next build, the driver is then able to read that module file, extract the swiftdeps information from it, and use it to influence the way it schedules jobs.

The sum total is that we neatly extend the intra-module case of incremental builds to the inter-module case by treating swiftmodule inputs not as opaque entities, but as "big ol' flat Swift files" that just export an interface like any other Swift file within the module. As a further optimization, and because clients literally cannot observe this aspect of the incremental build, we only serialize the provides (the "defs" side of a "use-def" edge) when emitting swiftdeps metadata into swiftmodule files.

rdar://69595010

Author: CodaFi

include/swift/Option/Options.td

@@ -573,6 +573,7 @@ def experimental_cxx_stdlib :
 
 def enable_experimental_cross_module_incremental_build :
   Flag<["-"], "enable-experimental-cross-module-incremental-build">,
+  Flags<[FrontendOption]>,
   HelpText<"(experimental) Enable cross-module incremental build metadata and "
            "driver scheduling">;
 

lib/Serialization/SerializedModuleLoader.cpp

@@ -973,13 +973,6 @@ SerializedModuleLoaderBase::loadModule(SourceLoc importLoc,
                   isFramework, isSystemModule)) {
     return nullptr;
   }
-  if (dependencyTracker) {
-    // Don't record cached artifacts as dependencies.
-    StringRef DepPath = moduleInputBuffer->getBufferIdentifier();
-    if (!isCached(DepPath)) {
-      dependencyTracker->addDependency(DepPath, /*isSystem=*/false);
-    }
-  }
 
   assert(moduleInputBuffer);
 
@@ -997,6 +990,18 @@ SerializedModuleLoaderBase::loadModule(SourceLoc importLoc,
   } else {
     M->setFailedToLoad();
   }
+
+  if (dependencyTracker && file) {
+    auto DepPath = file->getFilename();
+    // Don't record cached artifacts as dependencies.
+    if (!isCached(DepPath)) {
+      if (M->hasIncrementalInfo()) {
+        dependencyTracker->addIncrementalDependency(DepPath);
+      } else {
+        dependencyTracker->addDependency(DepPath, /*isSystem=*/false);
+      }
+    }
+  }
   return M;
 }
 

test/Incremental/CrossModule/Inputs/linear/A.json

@@ -0,0 +1,11 @@
+{
+  "A.swift": {
+    "object": "./A.o",
+    "swift-dependencies": "./A.swiftdeps",
+    "swiftmodule": "./A~partial.swiftmodule",
+    "swiftdoc": "./A.swiftdoc",
+  },
+  "": {
+    "swift-dependencies": "./A~buildrecord.swiftdeps"
+  }
+}

test/Incremental/CrossModule/Inputs/linear/A.swift

@@ -0,0 +1 @@
+import B

test/Incremental/CrossModule/Inputs/linear/B.json

@@ -0,0 +1,11 @@
+{
+  "B.swift": {
+    "object": "./B.o",
+    "swift-dependencies": "./B.swiftdeps",
+    "swiftmodule": "./B~partial.swiftmodule",
+    "swiftdoc": "./B.swiftdoc",
+  },
+  "": {
+    "swift-dependencies": "./B~buildrecord.swiftdeps"
+  }
+}

test/Incremental/CrossModule/Inputs/linear/B.swift

@@ -0,0 +1,5 @@
+import C
+
+public func fromB() {
+  return fromC()
+}

test/Incremental/CrossModule/Inputs/linear/C.json

@@ -0,0 +1,11 @@
+{
+  "C.swift": {
+    "object": "./C.o",
+    "swift-dependencies": "./C.swiftdeps",
+    "swiftmodule": "./C~partial.swiftmodule",
+    "swiftdoc": "./C.swiftdoc",
+  },
+  "": {
+    "swift-dependencies": "./C~buildrecord.swiftdeps"
+  }
+}

test/Incremental/CrossModule/Inputs/linear/C.swift

@@ -0,0 +1,7 @@
+#if OLD
+public func fromC() {}
+#elseif NEW
+public func fromC(parameter: Int = 0) {}
+#else
+#error("test must define one of NEW or OLD macros")
+#endif

test/Incremental/CrossModule/Inputs/transitive/A.json

@@ -0,0 +1,11 @@
+{
+  "A.swift": {
+    "object": "./A.o",
+    "swift-dependencies": "./A.swiftdeps",
+    "swiftmodule": "./A~partial.swiftmodule",
+    "swiftdoc": "./A.swiftdoc",
+  },
+  "": {
+    "swift-dependencies": "./A~buildrecord.swiftdeps"
+  }
+}

test/Incremental/CrossModule/Inputs/transitive/A.swift

@@ -0,0 +1,9 @@
+import B
+import C
+
+public func use() {
+  fromB()
+  #if USEC
+  fromC()
+  #endif
+}