Add %p to LLVM_PROFILE_FILE pattern when running tests with coverage #8894

simonjbeaumont · 2025-07-04T09:42:47Z

Motivation

The current setting for LLVM_PROFILE_PATH, used for code coverage, leads to corrupt profile data when tests are run in parallel or when writing "exit tests" with Swift Testing. This also results in the swift test --enable-code-coverage command to fail.

The LLVM_PROFILE_PATH environment variable is used by the runtime to write raw profile files, which are then processed when the test command finishes to produce the coverage results as JSON. The variable supports several pattern variables¹, including %Nm, which is currently set, and is documented to create a pool of files that the runtime will handle synchronisation of. This is fine for parallelism within the process but will not work across different processes. SwiftPM uses multiple invocations of the same binary for parallel testing and users may also fork processes within their tests, which is now a required workflow when using exit tests with Swift Testing, which will fork the process internally. Furthermore, the current setting for this variable uses only %m (which implies N=1), which makes it even more likely that processes will stomp over each other when writing the raw profile data.

We can see a discussion of this happening in practice in #8893.

The variable also supports %p¹, which will expand to produce a per-process path for the raw profile, which is probably what we want here, since Swift PM is combining all the profiles in the configured directory.

Modifications

Add %p to LLVM_PROFILE_FILE pattern when running tests with coverage.

Result

Running tests write coverage raw profile data to their own per-process file pool.
Running tests in parallel with code coverage no longer risks corrupting coverage data.
Running exit tests no longer risks corrupting coverage data.
Fixes Multiple exit tests result in corrupt coverage profdata, which leads to test command failure, even with .serialized #8893.

Appendix: Demonstrating the merging of per-process profiles

// file: ReproTests.swift

import Testing
import struct Foundation.URL
#if canImport(Darwin)
import Darwin
#elseif canImport(Glibc)
import Glibc
#endif

@Suite(.serialized) struct Suite {
    static func updateLLVMProfilePath() {
        let key = "LLVM_PROFILE_FILE"
        let profrawExtension = "profraw"
        guard let previousValueCString = getenv(key) else { return }
        let previousValue = String(cString: previousValueCString)
        let previousPath = URL(filePath: previousValue)
        guard previousPath.pathExtension == profrawExtension else { return }
        guard !previousPath.lastPathComponent.contains("%p") else { return }
        let newPath = previousPath.deletingPathExtension().appendingPathExtension("%p").appendingPathExtension(profrawExtension)
        let newValue = newPath.path(percentEncoded: false)
        print("Replacing \(key)=\(previousValue) with \(key)=\(newValue)")
        setenv(key, newValue, 1)
    }

    @Test func testA() async {
        Self.updateLLVMProfilePath()
        await #expect(processExitsWith: .success) { Subject.a() }
    }

    @Test func testB() async {
        Self.updateLLVMProfilePath()
        await #expect(processExitsWith: .success) { Subject.b() }
    }
}

// file: Subject.swift

struct Subject {
    static func a() { _ = "a" }
    static func b() { _ = "a" }
}

Running with just one test results in one per-process profile and 50% coverage, as expected.

% swift test --enable-code-coverage --filter Suite.testa
...
◇ Test run started.
↳ Testing Library Version: 6.2 (9ebfc4ebbb2840d)
↳ Target Platform: aarch64-unknown-linux-gnu
◇ Suite Suite started.
◇ Test testa() started.
Replacing LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.profraw with LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.%p.profraw
✔ Test testa() passed after 0.018 seconds.
✔ Suite Suite passed after 0.018 seconds.
✔ Test run with 1 test in 1 suite passed after 0.018 seconds.

% ls -1 .build/debug/codecov/
default.profdata
repro-exit-tests-coverage-corruption.json
'Swift Testing12847901981426048528_0.15828.profraw'
'Swift Testing12847901981426048528_0.profraw'
XCTest12847901981426048528_0.profraw

% cat .build/debug/codecov/repro-exit-tests-coverage-corruption.json  | jq '.data[].files[] | select(.filename == "/pwd/Tests/ReproTests/Subject.swift").summary.functions'
{
  "count": 2,
  "covered": 1,
  "percent": 50
}

Running the other test also results in one per-process profile and 50% coverage, as expected.

% swift test --enable-code-coverage --filter Suite.testb
...
◇ Test run started.
↳ Testing Library Version: 6.2 (9ebfc4ebbb2840d)
↳ Target Platform: aarch64-unknown-linux-gnu
◇ Suite Suite started.
◇ Test testb() started.
Replacing LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.profraw with LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.%p.profraw
✔ Test testb() passed after 0.017 seconds.
✔ Suite Suite passed after 0.017 seconds.
✔ Test run with 1 test in 1 suite passed after 0.017 seconds.

% ls -1 .build/debug/codecov/
default.profdata
repro-exit-tests-coverage-corruption.json
'Swift Testing12847901981426048528_0.15905.profraw'
'Swift Testing12847901981426048528_0.profraw'
XCTest12847901981426048528_0.profraw

% cat .build/debug/codecov/repro-exit-tests-coverage-corruption.json  | jq '.data[].files[] | select(.filename == "/pwd/Tests/ReproTests/Subject.swift").summary.functions'
{
  "count": 2,
  "covered": 1,
  "percent": 50
}

Running both tests results in two per-process profile and 100% coverage, after merge.

% swift test --enable-code-coverage --filter Suite.testa --filter Suite.testb
...
◇ Test run started.
↳ Testing Library Version: 6.2 (9ebfc4ebbb2840d)
↳ Target Platform: aarch64-unknown-linux-gnu
◇ Suite Suite started.
◇ Test testa() started.
Replacing LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.profraw with LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.%p.profraw
✔ Test testa() passed after 0.016 seconds.
◇ Test testb() started.
✔ Test testb() passed after 0.015 seconds.
✔ Suite Suite passed after 0.033 seconds.
✔ Test run with 2 tests in 1 suite passed after 0.033 seconds.

% ls -1 .build/debug/codecov/
default.profdata
repro-exit-tests-coverage-corruption.json
'Swift Testing12847901981426048528_0.15981.profraw'
'Swift Testing12847901981426048528_0.15988.profraw'
'Swift Testing12847901981426048528_0.profraw'
XCTest12847901981426048528_0.profraw

% cat .build/debug/codecov/repro-exit-tests-coverage-corruption.json  | jq '.data[].files[] | select(.filename == "/pwd/Tests/ReproTests/Subject.swift").summary.functions'
{
  "count": 2,
  "covered": 2,
  "percent": 100
}

https://clang.llvm.org/docs/SourceBasedCodeCoverage.html#running-the-instrumented-program ↩ ↩²

dschaefer2 · 2025-07-04T19:32:44Z

@swift-ci please test

dschaefer2 · 2025-07-04T19:35:04Z

Thanks Si! Can you add some simple tests here to make sure the attributes in the path resolve correctly?

simonjbeaumont · 2025-07-04T21:21:58Z

@dschaefer2 sure thing. I couldn't see any tests that check for the coverage output, but I maybe missed them? Could you point me at them if they exist or, if they don't, suggest the best place to add the tests (don't super know my way around in SwiftPM).

The PR description shows the impact of running with %p with an out of tree workaround.

simonjbeaumont · 2025-07-07T09:35:37Z

@dschaefer2 I added a test in 5290b2b. I couldn't see anywhere where there were already tests for how the coverage files got exported, merged, or validated, so I decided to put them in integration tests.

dschaefer2 · 2025-07-07T14:26:46Z

@dschaefer2 I added a test in 5290b2b. I couldn't see anywhere where there were already tests for how the coverage files got exported, merged, or validated, so I decided to put them in integration tests.

Nice. Thanks!

dschaefer2 · 2025-07-07T14:26:57Z

@swift-ci please test

dschaefer2 · 2025-07-07T14:27:09Z

@swift-ci please test windows

simonjbeaumont · 2025-07-08T16:48:50Z

@dschaefer2 I notice the Linux jobs are still running with Swift 6.1, is that expected?

[2025-07-07T14:37:31.549Z] ◇ Test run started.
[2025-07-07T14:37:31.549Z] ↳ Testing Library Version: 6.1 (43b6f88e2f2712e)

— https://ci.swift.org/job/swift-package-manager-self-hosted-Linux-smoke-test/6515/consoleText

I will update the test to either skip on pre-6.2 (because it uses exit tests) or to try and validate some other way. But just checking that's what we expect.

I have validated locally that they do pass with the 6.2 nightly Docker image.

simonjbeaumont · 2025-07-08T16:52:22Z

@swift-ci please test

bkhouri · 2025-07-09T04:17:56Z

I notice the Linux jobs are still running with Swift 6.1, is that expected?

@simonjbeaumont : This is currently expected. The self-hosted pipeline run using a released version of the toolchain, while the "Some Test" execute, what I cal, a pseudo-toolchain build - that is, executes the swiftlang/swift/utils/build-script and swiftlang/swift/utils/build-windows-toolchain scripts.

simonjbeaumont · 2025-07-09T07:42:06Z

@swift-ci please test windows

simonjbeaumont · 2025-07-09T08:58:51Z

Windows CI appears to be upset even getting going.

[2025-07-09T07:43:29.049Z] jenkins.util.io.CompositeIOException: Unable to delete 'C:\Users\swift-ci\jenkins\workspace\pr-swiftpm-windows-self-hosted'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 204 additional exceptions)
[2025-07-09T07:43:29.052Z] [Pipeline] }
[2025-07-09T07:43:29.067Z] [Pipeline] // timeout
[2025-07-09T07:43:29.074Z] [Pipeline] }
[2025-07-09T07:43:29.093Z] [Pipeline] // node
[2025-07-09T07:43:29.101Z] [Pipeline] End of Pipeline
[2025-07-09T07:43:29.565Z] Setting status of 3b48cda0e86e4e5df5e1d2ac2b600694866526fb to FAILURE with url https://ci-external.swift.org/job/pr-swiftpm-windows-self-hosted/1492/ and message: 'Failed
[2025-07-09T07:43:29.565Z]  '
[2025-07-09T07:43:29.565Z] Using context: Swift Test Windows Platform (self hosted) (Swift 6.1)
[2025-07-09T07:43:29.809Z] Finished: FAILURE

bkhouri · 2025-07-09T17:56:40Z

@swift-ci test self hosted windows

bkhouri · 2025-07-10T15:58:54Z

Sources/_InternalTestSupport/SwiftTesting+TraitConditional.swift

@@ -58,4 +58,15 @@ extension Trait where Self == Testing.ConditionTrait {
            #endif
        }
    }
+
+    /// Skip test if compiler is older than 6.2.


praise: Thanks for creating a trait for this.

bkhouri · 2025-07-10T16:03:04Z

Tests/IntegrationTests/SwiftPMTests.swift

+            let coverage = try JSONDecoder().decode(Coverage.self, from: Data(coverageJSON.contents))
+
+            // Check for 100% coverage for Subject.swift, which should happen because the per-PID files got merged.
+            let subjectCoverage = coverage.data.first?.files.first(where: { $0.filename.hasSuffix("Subject.swift") })


suggestion: use #require() Swift Testing api to ensure a value is set.

e.g.:

let firstCoverageData = try #require(coverage.data.first) let subjectCoverage = try #require(firstCoverageData.files.first(where: { $0.filename.hasSuffix("Subject.swift") })) #expect(subjectCoverage.summary.functions.count == 2) #expect(subjectCoverage.summary.functions.covered == 2) #expect(subjectCoverage.summary.functions.percent == 100)

Yeah, we could do that. If the values not there it'll result in the #expect calls comparing against nil so they'll fail gracefully enough.

Do you want me to make this change before we land the PR?

bkhouri · 2025-07-10T16:04:47Z

Tests/IntegrationTests/SwiftPMTests.swift

+
+            // Check for 100% coverage for Subject.swift, which should happen because the per-PID files got merged.
+            let subjectCoverage = coverage.data.first?.files.first(where: { $0.filename.hasSuffix("Subject.swift") })
+            #expect(subjectCoverage?.summary.functions.count == 2)


thought: I'm not certain we should be validating the contents of the coverage data in this test as SwiftPM has no control over this.

I actually think SwiftPM does control this. It's SwiftPM that takes the LLVM raw profile data, in .profraw files, merges it into a .profdata file, and then creates a JSON file to represent the coverage.

The JSON file has a known schema which is API so we can rely on it and the test has been constructed above specifically so we can test that there's 100% coverage of a file, when split across two tests that run in separate processes.

bkhouri · 2025-07-10T16:11:13Z

This fixes #8893

@suite

…wiftlang#8894) ## Motivation The current setting for `LLVM_PROFILE_PATH`, used for code coverage, leads to corrupt profile data when tests are run in parallel or when writing "exit tests" with Swift Testing. This also results in the `swift test --enable-code-coverage` command to fail. The `LLVM_PROFILE_PATH` environment variable is used by the runtime to write raw profile files, which are then processed when the test command finishes to produce the coverage results as JSON. The variable supports several pattern variables[^1], including `%Nm`, which is currently set, and is documented to create a pool of files that the runtime will handle synchronisation of. This is fine for parallelism within the process but will not work across different processes. SwiftPM uses multiple invocations of the same binary for parallel testing and users may also fork processes within their tests, which is now a required workflow when using _exit tests_ with Swift Testing, which will fork the process internally. Furthermore, the current setting for this variable uses only `%m` (which implies `N=1`), which makes it even more likely that processes will stomp over each other when writing the raw profile data. We can see a discussion of this happening in practice in swiftlang#8893. The variable also supports `%p`[^1], which will expand to produce a per-process path for the raw profile, which is probably what we want here, since Swift PM is combining all the profiles in the configured directory. ## Modifications Add %p to LLVM_PROFILE_FILE pattern when running tests with coverage. ## Result - Running tests write coverage raw profile data to their own per-process file pool. - Running tests in parallel with code coverage no longer risks corrupting coverage data. - Running exit tests no longer risks corrupting coverage data. - Fixes swiftlang#8893. --- ## Appendix: Demonstrating the merging of per-process profiles ```swift // file: ReproTests.swift import Testing import struct Foundation.URL #if canImport(Darwin) import Darwin #elseif canImport(Glibc) import Glibc #endif @suite(.serialized) struct Suite { static func updateLLVMProfilePath() { let key = "LLVM_PROFILE_FILE" let profrawExtension = "profraw" guard let previousValueCString = getenv(key) else { return } let previousValue = String(cString: previousValueCString) let previousPath = URL(filePath: previousValue) guard previousPath.pathExtension == profrawExtension else { return } guard !previousPath.lastPathComponent.contains("%p") else { return } let newPath = previousPath.deletingPathExtension().appendingPathExtension("%p").appendingPathExtension(profrawExtension) let newValue = newPath.path(percentEncoded: false) print("Replacing \(key)=\(previousValue) with \(key)=\(newValue)") setenv(key, newValue, 1) } @test func testA() async { Self.updateLLVMProfilePath() await #expect(processExitsWith: .success) { Subject.a() } } @test func testB() async { Self.updateLLVMProfilePath() await #expect(processExitsWith: .success) { Subject.b() } } } ``` ```swift // file: Subject.swift struct Subject { static func a() { _ = "a" } static func b() { _ = "a" } } ``` Running with just one test results in one per-process profile and 50% coverage, as expected. ```console % swift test --enable-code-coverage --filter Suite.testa ... ◇ Test run started. ↳ Testing Library Version: 6.2 (9ebfc4ebbb2840d) ↳ Target Platform: aarch64-unknown-linux-gnu ◇ Suite Suite started. ◇ Test testa() started. Replacing LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.profraw with LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.%p.profraw ✔ Test testa() passed after 0.018 seconds. ✔ Suite Suite passed after 0.018 seconds. ✔ Test run with 1 test in 1 suite passed after 0.018 seconds. ``` ```console % ls -1 .build/debug/codecov/ default.profdata repro-exit-tests-coverage-corruption.json 'Swift Testing12847901981426048528_0.15828.profraw' 'Swift Testing12847901981426048528_0.profraw' XCTest12847901981426048528_0.profraw % cat .build/debug/codecov/repro-exit-tests-coverage-corruption.json | jq '.data[].files[] | select(.filename == "/pwd/Tests/ReproTests/Subject.swift").summary.functions' { "count": 2, "covered": 1, "percent": 50 } ``` Running the other test also results in one per-process profile and 50% coverage, as expected. ```console % swift test --enable-code-coverage --filter Suite.testb ... ◇ Test run started. ↳ Testing Library Version: 6.2 (9ebfc4ebbb2840d) ↳ Target Platform: aarch64-unknown-linux-gnu ◇ Suite Suite started. ◇ Test testb() started. Replacing LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.profraw with LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.%p.profraw ✔ Test testb() passed after 0.017 seconds. ✔ Suite Suite passed after 0.017 seconds. ✔ Test run with 1 test in 1 suite passed after 0.017 seconds. ``` ```console % ls -1 .build/debug/codecov/ default.profdata repro-exit-tests-coverage-corruption.json 'Swift Testing12847901981426048528_0.15905.profraw' 'Swift Testing12847901981426048528_0.profraw' XCTest12847901981426048528_0.profraw % cat .build/debug/codecov/repro-exit-tests-coverage-corruption.json | jq '.data[].files[] | select(.filename == "/pwd/Tests/ReproTests/Subject.swift").summary.functions' { "count": 2, "covered": 1, "percent": 50 } ``` Running both tests results in two per-process profile and 100% coverage, after merge. ```console % swift test --enable-code-coverage --filter Suite.testa --filter Suite.testb ... ◇ Test run started. ↳ Testing Library Version: 6.2 (9ebfc4ebbb2840d) ↳ Target Platform: aarch64-unknown-linux-gnu ◇ Suite Suite started. ◇ Test testa() started. Replacing LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.profraw with LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.%p.profraw ✔ Test testa() passed after 0.016 seconds. ◇ Test testb() started. ✔ Test testb() passed after 0.015 seconds. ✔ Suite Suite passed after 0.033 seconds. ✔ Test run with 2 tests in 1 suite passed after 0.033 seconds. ``` ```console % ls -1 .build/debug/codecov/ default.profdata repro-exit-tests-coverage-corruption.json 'Swift Testing12847901981426048528_0.15981.profraw' 'Swift Testing12847901981426048528_0.15988.profraw' 'Swift Testing12847901981426048528_0.profraw' XCTest12847901981426048528_0.profraw % cat .build/debug/codecov/repro-exit-tests-coverage-corruption.json | jq '.data[].files[] | select(.filename == "/pwd/Tests/ReproTests/Subject.swift").summary.functions' { "count": 2, "covered": 2, "percent": 100 } ``` [^1]: https://clang.llvm.org/docs/SourceBasedCodeCoverage.html#running-the-instrumented-program (cherry picked from commit 5e566d4)

Add %p to LLVM_PROFILE_FILE pattern when running tests with coverage

0b3beec

simonjbeaumont requested review from jakepetroules, dschaefer2, shawnhyam, bripeticca, plemarquand, owenv, bkhouri, cmcgee1024 and daveyc123 as code owners July 4, 2025 09:42

dschaefer2 added the needs tests This change needs test coverage label Jul 4, 2025

Add integration test

5290b2b

simonjbeaumont added 2 commits July 8, 2025 15:23

fixup: Fix JSON schema and typo in comment

bca1638

Only run exit tests on 6.2+

3b48cda

bkhouri approved these changes Jul 10, 2025

View reviewed changes

bkhouri merged commit 5e566d4 into swiftlang:main Jul 10, 2025
6 checks passed

simonjbeaumont mentioned this pull request Jul 10, 2025

[6.2] Add %p to LLVM_PROFILE_FILE pattern when running tests with coverage (#8894) #8919

Closed

bkhouri mentioned this pull request Jul 10, 2025

[6.2] Add %p to LLVM_PROFILE_FILE pattern when running tests with coverage #8918

Merged

Add %p to LLVM_PROFILE_FILE pattern when running tests with coverage #8894

Add %p to LLVM_PROFILE_FILE pattern when running tests with coverage #8894

Uh oh!

Conversation

simonjbeaumont commented Jul 4, 2025

Motivation

Modifications

Result

Appendix: Demonstrating the merging of per-process profiles

Footnotes

Uh oh!

dschaefer2 commented Jul 4, 2025

Uh oh!

dschaefer2 commented Jul 4, 2025

Uh oh!

simonjbeaumont commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simonjbeaumont commented Jul 7, 2025

Uh oh!

dschaefer2 commented Jul 7, 2025

Uh oh!

dschaefer2 commented Jul 7, 2025

Uh oh!

dschaefer2 commented Jul 7, 2025

Uh oh!

simonjbeaumont commented Jul 8, 2025

Uh oh!

simonjbeaumont commented Jul 8, 2025

Uh oh!

bkhouri commented Jul 9, 2025

Uh oh!

simonjbeaumont commented Jul 9, 2025

Uh oh!

simonjbeaumont commented Jul 9, 2025

Uh oh!

bkhouri commented Jul 9, 2025

Uh oh!

bkhouri Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

bkhouri Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

simonjbeaumont Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

bkhouri Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

simonjbeaumont Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

bkhouri commented Jul 10, 2025

Uh oh!

Uh oh!

Uh oh!

simonjbeaumont commented Jul 4, 2025 •

edited

Loading