Add %p to LLVM_PROFILE_FILE pattern when running tests with coverage

The current setting for `LLVM_PROFILE_PATH`, used for code coverage,
leads to corrupt profile data when tests are run in parallel or when
writing "exit tests" with Swift Testing. This also results in the `swift
test --enable-code-coverage` command to fail.

The `LLVM_PROFILE_PATH` environment variable is used by the runtime to
write raw profile files, which are then processed when the test command
finishes to produce the coverage results as JSON. The variable supports
several pattern variables[^1], including `%Nm`, which is currently set,
and is documented to create a pool of files that the runtime will handle
synchronisation of. This is fine for parallelism within the process but
will not work across different processes. SwiftPM uses multiple
invocations of the same binary for parallel testing and users may also
fork processes within their tests, which is now a required workflow when
using _exit tests_ with Swift Testing, which will fork the process
internally. Furthermore, the current setting for this variable uses only
`%m` (which implies `N=1`), which makes it even more likely that
processes will stomp over each other when writing the raw profile data.

We can see a discussion of this happening in practice in #8893.

The variable also supports `%p`[^1], which will expand to produce a
per-process path for the raw profile, which is probably what we want
here, since Swift PM is combining all the profiles in the configured
directory.

Add %p to LLVM_PROFILE_FILE pattern when running tests with coverage.

- Running tests write coverage raw profile data to their own per-process
file pool.
- Running tests in parallel with code coverage no longer risks
corrupting coverage data.
- Running exit tests no longer risks corrupting coverage data.
- Fixes #8893.

---

```swift
// file: ReproTests.swift

import Testing
import struct Foundation.URL
import Darwin
import Glibc

@suite(.serialized) struct Suite {
    static func updateLLVMProfilePath() {
        let key = "LLVM_PROFILE_FILE"
        let profrawExtension = "profraw"
        guard let previousValueCString = getenv(key) else { return }
        let previousValue = String(cString: previousValueCString)
        let previousPath = URL(filePath: previousValue)
        guard previousPath.pathExtension == profrawExtension else { return }
        guard !previousPath.lastPathComponent.contains("%p") else { return }
        let newPath = previousPath.deletingPathExtension().appendingPathExtension("%p").appendingPathExtension(profrawExtension)
        let newValue = newPath.path(percentEncoded: false)
        print("Replacing \(key)=\(previousValue) with \(key)=\(newValue)")
        setenv(key, newValue, 1)
    }

    @test func testA() async {
        Self.updateLLVMProfilePath()
        await #expect(processExitsWith: .success) { Subject.a() }
    }

    @test func testB() async {
        Self.updateLLVMProfilePath()
        await #expect(processExitsWith: .success) { Subject.b() }
    }
}
```

```swift
// file: Subject.swift

struct Subject {
    static func a() { _ = "a" }
    static func b() { _ = "a" }
}
```

Running with just one test results in one per-process profile and 50%
coverage, as expected.

```console
% swift test --enable-code-coverage --filter Suite.testa
...
◇ Test run started.
↳ Testing Library Version: 6.2 (9ebfc4ebbb2840d)
↳ Target Platform: aarch64-unknown-linux-gnu
◇ Suite Suite started.
◇ Test testa() started.
Replacing LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.profraw with LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.%p.profraw
✔ Test testa() passed after 0.018 seconds.
✔ Suite Suite passed after 0.018 seconds.
✔ Test run with 1 test in 1 suite passed after 0.018 seconds.
```

```console
% ls -1 .build/debug/codecov/
default.profdata
repro-exit-tests-coverage-corruption.json
'Swift Testing12847901981426048528_0.15828.profraw'
'Swift Testing12847901981426048528_0.profraw'
XCTest12847901981426048528_0.profraw

% cat .build/debug/codecov/repro-exit-tests-coverage-corruption.json  | jq '.data[].files[] | select(.filename == "/pwd/Tests/ReproTests/Subject.swift").summary.functions'
{
  "count": 2,
  "covered": 1,
  "percent": 50
}
```

Running the other test also results in one per-process profile and 50%
coverage, as expected.

```console
% swift test --enable-code-coverage --filter Suite.testb
...
◇ Test run started.
↳ Testing Library Version: 6.2 (9ebfc4ebbb2840d)
↳ Target Platform: aarch64-unknown-linux-gnu
◇ Suite Suite started.
◇ Test testb() started.
Replacing LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.profraw with LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.%p.profraw
✔ Test testb() passed after 0.017 seconds.
✔ Suite Suite passed after 0.017 seconds.
✔ Test run with 1 test in 1 suite passed after 0.017 seconds.
```

```console
% ls -1 .build/debug/codecov/
default.profdata
repro-exit-tests-coverage-corruption.json
'Swift Testing12847901981426048528_0.15905.profraw'
'Swift Testing12847901981426048528_0.profraw'
XCTest12847901981426048528_0.profraw

% cat .build/debug/codecov/repro-exit-tests-coverage-corruption.json  | jq '.data[].files[] | select(.filename == "/pwd/Tests/ReproTests/Subject.swift").summary.functions'
{
  "count": 2,
  "covered": 1,
  "percent": 50
}
```

Running both tests results in two per-process profile and 100% coverage,
after merge.

```console
% swift test --enable-code-coverage --filter Suite.testa --filter Suite.testb
...
◇ Test run started.
↳ Testing Library Version: 6.2 (9ebfc4ebbb2840d)
↳ Target Platform: aarch64-unknown-linux-gnu
◇ Suite Suite started.
◇ Test testa() started.
Replacing LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.profraw with LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.%p.profraw
✔ Test testa() passed after 0.016 seconds.
◇ Test testb() started.
✔ Test testb() passed after 0.015 seconds.
✔ Suite Suite passed after 0.033 seconds.
✔ Test run with 2 tests in 1 suite passed after 0.033 seconds.
```

```console
% ls -1 .build/debug/codecov/
default.profdata
repro-exit-tests-coverage-corruption.json
'Swift Testing12847901981426048528_0.15981.profraw'
'Swift Testing12847901981426048528_0.15988.profraw'
'Swift Testing12847901981426048528_0.profraw'
XCTest12847901981426048528_0.profraw

% cat .build/debug/codecov/repro-exit-tests-coverage-corruption.json  | jq '.data[].files[] | select(.filename == "/pwd/Tests/ReproTests/Subject.swift").summary.functions'
{
  "count": 2,
  "covered": 2,
  "percent": 100
}
```

[^1]:
https://clang.llvm.org/docs/SourceBasedCodeCoverage.html#running-the-instrumented-program

(cherry picked from commit 5e566d4)

Author: simonjbeaumont

Sources/Commands/Utilities/TestingSupport.swift

@@ -197,12 +197,25 @@ enum TestingSupport {
         if buildParameters.testingParameters.enableCodeCoverage {
             // Defines the path at which the profraw files will be written on test execution.
             //
-            // `%m` will create a pool of profraw files and append the data from
-            // each execution in one of the files. This doesn't matter for serial
-            // execution but is required when the tests are running in parallel as
-            // SwiftPM repeatedly invokes the test binary with the test case name as
-            // the filter.
-            let codecovProfile = buildParameters.buildPath.appending(components: "codecov", "\(library)%m.profraw")
+            // `%Nm` will create a pool of N profraw files and append the data from each execution
+            // in one of the files. The runtime takes care of selecting a raw profile from the pool,
+            // locking it, and updating it before the program exits. If N is not specified, it is
+            // inferred to be 1.
+            //
+            // This is fine for parallel execution within a process, but for parallel tests, SwiftPM
+            // repeatedly invokes the test binary with the testcase name as the filter and the
+            // locking cannot be enforced by the runtime across the process boundaries.
+            //
+            // It's also possible that tests themselves will fork (e.g. for exit tests provided by
+            // Swift Testing), which will inherit the environment of the parent process, and so
+            // write to the same file, leading to profile data corruption.
+            //
+            // For these reasons, we unilaterally also add a %p, which will cause uniquely named
+            // files per process.
+            //
+            // These are all merged using `llvm-profdata merge` once the outer test command has
+            // completed.
+            let codecovProfile = buildParameters.buildPath.appending(components: "codecov", "\(library)%m.%p.profraw")
             env["LLVM_PROFILE_FILE"] = codecovProfile.pathString
         }
         #if !os(macOS)

Sources/_InternalTestSupport/SkippedTestSupport.swift

@@ -58,6 +58,17 @@ extension Trait where Self == Testing.ConditionTrait {
             #endif
         }
     }
+
+    /// Skip test if compiler is older than 6.2.
+    public static var requireSwift6_2: Self {
+        enabled("This test requires Swift 6.2, or newer.") {
+            #if compiler(>=6.2)
+            true
+            #else
+            false
+            #endif
+        }
+    }
 }
 
 extension Trait where Self == Testing.Bug {

Tests/IntegrationTests/SwiftPMTests.swift

@@ -201,4 +201,93 @@ private struct SwiftPMTests {
             }
         }
     }
+
+    @Test(.requireSwift6_2)
+    func testCodeCoverageMergedAcrossSubprocesses() async throws {
+        try await withTemporaryDirectory { tmpDir in
+            let packagePath = tmpDir.appending(component: "test-package-coverage")
+            try localFileSystem.createDirectory(packagePath)
+            try await executeSwiftPackage(packagePath, extraArgs: ["init", "--type", "empty"])
+            try await executeSwiftPackage(packagePath, extraArgs: ["add-target", "--type", "test", "ReproTests"])
+            try localFileSystem.writeFileContents(
+                AbsolutePath(validating: "Tests/ReproTests/Subject.swift", relativeTo: packagePath),
+                string: """
+                struct Subject {
+                    static func a() { _ = "a" }
+                    static func b() { _ = "b" }
+                }
+                """
+            )
+            try localFileSystem.writeFileContents(
+                AbsolutePath(validating: "Tests/ReproTests/ReproTests.swift", relativeTo: packagePath),
+                string: """
+                import Testing
+                import class Foundation.ProcessInfo
+                @Suite struct Suite {
+                    @Test func testProfilePathCanary() throws {
+                        let pattern = try #require(ProcessInfo.processInfo.environment["LLVM_PROFILE_FILE"])
+                        #expect(pattern.hasSuffix(".%p.profraw"))
+                    }
+                    @Test func testA() async { await #expect(processExitsWith: .success) { Subject.a() } }
+                    @Test func testB() async { await #expect(processExitsWith: .success) { Subject.b() } }
+                }
+                """
+            )
+            let expectedCoveragePath = try await executeSwiftTest(packagePath, extraArgs: ["--show-coverage-path"]).stdout.trimmingCharacters(in: .whitespacesAndNewlines)
+            try await executeSwiftTest(packagePath, extraArgs: ["--enable-code-coverage", "--disable-xctest"])
+            let coveragePath = try AbsolutePath(validating: expectedCoveragePath)
+
+            // Check the coverage path exists.
+            #expect(localFileSystem.exists(coveragePath))
+
+            // This resulting coverage file should be merged JSON, with a schema that valiades against this subset.
+            struct Coverage: Codable {
+                var data: [Entry]
+                struct Entry: Codable {
+                    var files: [File]
+                    struct File: Codable {
+                        var filename: String
+                        var summary: Summary
+                        struct Summary: Codable {
+                            var functions: Functions
+                            struct Functions: Codable {
+                                var count, covered: Int
+                                var percent: Double
+                            }
+                        }
+                    }
+                }
+            }
+            let coverageJSON = try localFileSystem.readFileContents(coveragePath)
+            let coverage = try JSONDecoder().decode(Coverage.self, from: Data(coverageJSON.contents))
+
+            // Check for 100% coverage for Subject.swift, which should happen because the per-PID files got merged.
+            let subjectCoverage = coverage.data.first?.files.first(where: { $0.filename.hasSuffix("Subject.swift") })
+            #expect(subjectCoverage?.summary.functions.count == 2)
+            #expect(subjectCoverage?.summary.functions.covered == 2)
+            #expect(subjectCoverage?.summary.functions.percent == 100)
+
+            // Check the directory with the coverage path contains the profraw files.
+            let coverageDirectory = coveragePath.parentDirectory
+            let coverageDirectoryContents = try localFileSystem.getDirectoryContents(coverageDirectory)
+
+            // SwiftPM uses an LLVM_PROFILE_FILE that ends with ".%p.profraw", which we validated in the test above.
+            // Let's first check all the files have the expected extension.
+            let profrawFiles = coverageDirectoryContents.filter { $0.hasSuffix(".profraw") }
+
+            // Then check that %p expanded as we expected: to something that plausibly looks like a PID.
+            for profrawFile in profrawFiles {
+                let shouldBePID = try #require(profrawFile.split(separator: ".").dropLast().last)
+                #expect(Int(shouldBePID) != nil)
+            }
+
+            // Group the files by binary identifier (have a different prefix, before the per-PID suffix).
+            let groups = Dictionary(grouping: profrawFiles) { path in path.split(separator: ".").dropLast(2) }.values
+
+            // Check each group has 3 files: one per PID (the above suite has 2 exit tests => 2 forks => 3 PIDs total).
+            for binarySpecificProfrawFiles in groups {
+                #expect(binarySpecificProfrawFiles.count == 3)
+            }
+        }
+    }
 }