Skip to content

Commit d5d611e

Browse files
grynspangrynspan
committed
[ST-NNNN] Conditionally saving attachments
1 parent 1c75af3 commit d5d611e

File tree

1 file changed

+351
-0
lines changed

1 file changed

+351
-0
lines changed

‎proposals/testing/NNNN-conditionally-saving-attachments.md‎

Lines changed: 351 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,351 @@
1+
# Conditionally saving attachments
2+
3+
* Proposal: [ST-NNNN](NNNN-conditionally-saving-attachments.md)
4+
* Authors: [Jonathan Grynspan](https://github.com/grynspan)
5+
* Review Manager: TBD
6+
* Status: **Awaiting review**
7+
* Bug: [rdar://138921461](rdar://138921461)
8+
* Implementation: [swiftlang/swift-testing#1319](https://github.com/swiftlang/swift-testing/pull/1319)
9+
* Review: ([pitch](https://forums.swift.org/...))
10+
11+
## Introduction
12+
13+
In [ST-0009](0009-attachments.md), we introduced [attachments](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0009-attachments.md)
14+
to Swift Testing in Swift 6.2. This feature allows you to create a file
15+
containing data relevant to a test. This proposal covers introducing new API to
16+
Swift Testing to allow test authors to control whether or not a test's
17+
attachments should be saved or discarded.
18+
19+
> [!NOTE]
20+
> In this proposal, **recording** an attachment means calling
21+
> [`Attachment.record()`](https://developer.apple.com/documentation/testing/attachment/record(_:named:sourcelocation:)).
22+
> When you record an attachment, it is stored in an implementation-defined
23+
> temporary location (such as memory or `/tmp/`) until the current test function
24+
> returns. Swift Testing then determines if the attachment should be **saved**,
25+
> i.e. written to persistent storage such as a file on disk or an Xcode test
26+
> report.
27+
28+
## Motivation
29+
30+
In **XCTest** on Apple platforms, you can specify the *lifetime* of an
31+
attachment by setting the [`lifetime`](https://developer.apple.com/documentation/xctest/xctattachment/lifetime-swift.property)
32+
property of an `XCTAttachment` object. This property lets you avoid serializing
33+
and saving an attachment if its data won't be necessary (e.g. "if the test
34+
passes, don't bother saving this attachment.")
35+
36+
It is especially useful for test authors working in CI environments to be able
37+
to control whether or not their attachments are saved, especially when those
38+
attachments are large (on the order of hundreds of megabytes or more).
39+
Persistent storage may come with a real-world monetary cost, or may be limited
40+
such that their CI jobs run out of space for new attachments too quickly.
41+
42+
The initial implementation of the attachments feature did not include this
43+
functionality, but we listed it as a future direction in [ST-0009](0009-attachments.md).
44+
We understand the utility of [`XCTAttachment.lifetime`](https://developer.apple.com/documentation/xctest/xctattachment/lifetime-swift.property)
45+
and want to bring an analogous interface to Swift Testing.
46+
47+
## Proposed solution
48+
49+
I propose introducing a new test trait that can be applied to a suite or test
50+
function. This trait can be configured with a condition that determines whether
51+
or not attachments should be saved.
52+
53+
## Detailed design
54+
55+
A new trait type is added:
56+
57+
```swift
58+
/// A type that defines a condition which must be satisfied for the testing
59+
/// library to save attachments recorded by a test.
60+
///
61+
/// To add this trait to a test, use one of the following functions:
62+
///
63+
/// - ``Trait/savingAttachments(if:)``
64+
///
65+
/// By default, the testing library saves your attachments as soon as you call
66+
/// ``Attachment/record(_:named:sourceLocation:)``. You can access saved
67+
/// attachments after your tests finish running:
68+
///
69+
/// - When using Xcode, you can access attachments from the test report.
70+
/// - When using Visual Studio Code, the testing library saves attachments to
71+
/// `.build/attachments` by default. Visual Studio Code reports the paths to
72+
/// individual attachments in its Tests Results panel.
73+
/// - When using Swift Package Manager's `swift test` command, you can pass the
74+
/// `--attachments-path` option. The testing library saves attachments to the
75+
/// specified directory.
76+
///
77+
/// If you add an instance of this trait type to a test, any attachments that
78+
/// test records are stored in memory until the test finishes running. The
79+
/// testing library then evaluates the instance's condition and, if the
80+
/// condition is met, saves the attachments.
81+
public struct AttachmentSavingTrait: TestTrait, SuiteTrait, TestScoping {
82+
/// A type that describes the conditions under which the testing library
83+
/// will save attachments.
84+
///
85+
/// You can pass instances of this type to ``Trait/savingAttachments(if:)``.
86+
public struct Condition: Sendable {
87+
/// The testing library saves attachments if the test passes.
88+
public static var testPasses: Self { get }
89+
90+
/// The testing library saves attachments if the test fails.
91+
public static var testFails: Self { get }
92+
93+
/// The testing library saves attachments if the test records a matching
94+
/// issue.
95+
///
96+
/// - Parameters:
97+
/// - issueMatcher: A function to invoke when an issue occurs that is used
98+
/// to determine if the testing library should save attachments for the
99+
/// current test.
100+
///
101+
/// - Returns: An instance of ``AttachmentSavingTrait/Condition`` that
102+
/// evaluates `issueMatcher`.
103+
public static func testRecordsIssue(
104+
matching issueMatcher: @escaping @Sendable (_ issue: Issue) async throws -> Bool
105+
) -> Self
106+
}
107+
}
108+
109+
extension Trait where Self == AttachmentSavingTrait {
110+
/// Constructs a trait that tells the testing library to only save attachments
111+
/// if a given condition is met.
112+
///
113+
/// - Parameters:
114+
/// - condition: A condition which, when met, means that the testing library
115+
/// should save attachments that the current test has recorded. If the
116+
/// condition is not met, the testing library discards the test's
117+
/// attachments when the test ends.
118+
/// - sourceLocation: The source location of the trait.
119+
///
120+
/// - Returns: An instance of ``AttachmentSavingTrait`` that evaluates the
121+
/// closure you provide.
122+
///
123+
/// By default, the testing library saves your attachments as soon as you call
124+
/// ``Attachment/record(_:named:sourceLocation:)``. You can access saved
125+
/// attachments after your tests finish running:
126+
///
127+
/// - When using Xcode, you can access attachments from the test report.
128+
/// - When using Visual Studio Code, the testing library saves attachments to
129+
/// `.build/attachments` by default. Visual Studio Code reports the paths to
130+
/// individual attachments in its Tests Results panel.
131+
/// - When using Swift Package Manager's `swift test` command, you can pass
132+
/// the `--attachments-path` option. The testing library saves attachments
133+
/// to the specified directory.
134+
///
135+
/// If you add this trait to a test, any attachments that test records are
136+
/// stored in memory until the test finishes running. The testing library then
137+
/// evaluates `condition` and, if the condition is met, saves the attachments.
138+
public static func savingAttachments(
139+
if condition: Self.Condition,
140+
sourceLocation: SourceLocation = #_sourceLocation
141+
) -> Self
142+
143+
/// Constructs a trait that tells the testing library to only save attachments
144+
/// if a given condition is met.
145+
///
146+
/// - Parameters:
147+
/// - condition: A closure that contains the trait's custom condition logic.
148+
/// If this closure returns `true`, the trait tells the testing library to
149+
/// save attachments that the current test has recorded. If this closure
150+
/// returns `false`, the testing library discards the test's attachments
151+
/// when the test ends. If this closure throws an error, the testing
152+
/// library records that error as an issue and discards the test's
153+
/// attachments.
154+
/// - sourceLocation: The source location of the trait.
155+
///
156+
/// - Returns: An instance of ``AttachmentSavingTrait`` that evaluates the
157+
/// closure you provide.
158+
///
159+
/// By default, the testing library saves your attachments as soon as you call
160+
/// ``Attachment/record(_:named:sourceLocation:)``. You can access saved
161+
/// attachments after your tests finish running:
162+
///
163+
/// - When using Xcode, you can access attachments from the test report.
164+
/// - When using Visual Studio Code, the testing library saves attachments
165+
///  to `.build/attachments` by default. Visual Studio Code reports the paths
166+
///  to individual attachments in its Tests Results panel.
167+
/// - When using Swift Package Manager's `swift test` command, you can pass
168+
/// the `--attachments-path` option. The testing library saves attachments
169+
/// to the specified directory.
170+
///
171+
/// If you add this trait to a test, any attachments that test records are
172+
/// stored in memory until the test finishes running. The testing library then
173+
/// evaluates `condition` and, if the condition is met, saves the attachments.
174+
public static func savingAttachments(
175+
if condition: @autoclosure @escaping @Sendable () throws -> Bool,
176+
sourceLocation: SourceLocation = #_sourceLocation
177+
) -> Self
178+
179+
/// Constructs a trait that tells the testing library to only save attachments
180+
/// if a given condition is met.
181+
///
182+
/// - Parameters:
183+
/// - condition: A closure that contains the trait's custom condition logic.
184+
/// If this closure returns `true`, the trait tells the testing library to
185+
/// save attachments that the current test has recorded. If this closure
186+
/// returns `false`, the testing library discards the test's attachments
187+
/// when the test ends. If this closure throws an error, the testing
188+
/// library records that error as an issue and discards the test's
189+
/// attachments.
190+
/// - sourceLocation: The source location of the trait.
191+
///
192+
/// - Returns: An instance of ``AttachmentSavingTrait`` that evaluates the
193+
/// closure you provide.
194+
///
195+
/// By default, the testing library saves your attachments as soon as you call
196+
/// ``Attachment/record(_:named:sourceLocation:)``. You can access saved
197+
/// attachments after your tests finish running:
198+
///
199+
/// - When using Xcode, you can access attachments from the test report.
200+
/// - When using Visual Studio Code, the testing library saves attachments
201+
///  to `.build/attachments` by default. Visual Studio Code reports the paths
202+
///  to individual attachments in its Tests Results panel.
203+
/// - When using Swift Package Manager's `swift test` command, you can pass
204+
/// the `--attachments-path` option. The testing library saves attachments
205+
/// to the specified directory.
206+
///
207+
/// If you add this trait to a test, any attachments that test records are
208+
/// stored in memory until the test finishes running. The testing library then
209+
/// evaluates `condition` and, if the condition is met, saves the attachments.
210+
public static func savingAttachments(
211+
if condition: @escaping @Sendable () async throws -> Bool,
212+
sourceLocation: SourceLocation = #_sourceLocation
213+
) -> Self
214+
}
215+
```
216+
217+
This trait can then be added to a test function using one of the listed factory
218+
functions. If added to a test suite, it is recursively applied to the test
219+
functions in that suite.
220+
221+
If multiple traits of this type are added to a test (directly or indirectly),
222+
then _all_ their conditions must be met for the test's attachments to be saved.
223+
This behavior is consistent with existing API such as [`ConditionTrait`](https://developer.apple.com/documentation/testing/conditiontrait).
224+
225+
### Default behavior
226+
227+
If a test has no instance of `AttachmentSavingTrait` applied to it, then the
228+
test host environment determines whether or not attachments from that test are
229+
saved:
230+
231+
- When using Xcode, your test plan determines if attachments are saved by
232+
default.
233+
- When using Visual Studio Code, attachments are saved to a temporary directory
234+
inside `./.build` by default.
235+
- When using `swift test`, you must pass `--attachments-path` to enable saving
236+
attachments.
237+
238+
For example, a test author may wish to only save attachments from a particular
239+
test if the test fails:
240+
241+
```swift
242+
@Test(.savingAttachments(if: .testFails))
243+
func `All vowels are green`() {
244+
var vowels = "AEIOU"
245+
if Bool.random() {
246+
vowels += "Y"
247+
}
248+
Attachment.record(vowels)
249+
#expect(vowels.allSatisfy { $0.color == .green })
250+
}
251+
```
252+
253+
Or a test author may wish to save attachments if an issue is recorded in a
254+
specific file:
255+
256+
```swift
257+
extension Issue {
258+
var inCriticalFile: Bool {
259+
guard let sourceLocation else { return false }
260+
return sourceLocation.fileID.hasSuffix("Critical.swift")
261+
}
262+
}
263+
264+
@Test(.savingAttachments(if: .testRecordsIssue { $0.inCriticalFile })
265+
func `Ideas taste tremendous`() { ... }
266+
```
267+
268+
If a test author wants to conditionally save attachments based on some outside
269+
state, there are overloads of `savingAttachments(if:)` that take an autoclosure
270+
or explicit closure:
271+
272+
```swift
273+
@Test(
274+
.savingAttachments(if: CommandLine.arguments.contains("--save--attachments")),
275+
.savingAttachments { try await CI.current.storage.available >= 500.MB }
276+
)
277+
func `The fandango is especially grim today`() { ... }
278+
```
279+
280+
## Source compatibility
281+
282+
This change is additive.
283+
284+
## Integration with supporting tools
285+
286+
Tools that consume the JSON event stream Swift Testing produces and which
287+
already observe the `.valueAttached` event will generally not need to change.
288+
When this trait is applied to a test, those events will be delivered later than
289+
they would be without it, but still before `.testCaseEnded` for the current test
290+
case.
291+
292+
Two new properties, `preferredName` and `bytes`, are added to the JSON structure
293+
describing an attachment when the stream's schema version is `"6.3"` or higher:
294+
295+
- `preferredName` is a string and contains (perhaps unsurprisingly) the test
296+
author's preferred filename for the attachment. For more information about
297+
this property, see [`Attachment.preferredName`](https://developer.apple.com/documentation/testing/attachment/preferredname).
298+
- `bytes`, if present, contains the serialized representation of the attachment
299+
as either a Base64-encoded string or an array of integers (one per byte). If
300+
the existing `path` property is set, this property is optional and may be
301+
excluded. (While this property may seem redundant, it is possible for a tool
302+
to consume the JSON event stream without also setting the attachments
303+
directory path, in which case the `bytes` property is necessary to recover the
304+
attachment's serialized representation.)
305+
306+
<!-- TODO: BNF for these properties -->
307+
308+
## Future directions
309+
310+
- We may wish to augment `Issue` or other types/concepts in Swift Testing to
311+
allow associating attachments with them rather than with the current test.
312+
This would likely take the form of an additional argument to
313+
[`Issue.record()`](https://developer.apple.com/documentation/testing/issue/record(_:sourcelocation:)).
314+
315+
- Test authors may still need more fine-grained control over whether individual
316+
attachments in a test should be saved. Our expectation (no pun intended) here
317+
is that per-test granularity will be sufficient for the majority of test
318+
authors. For those test authors who need more fine-grained control, we may
319+
want to add an argument of type `AttachmentSavingTrait.Condition` to
320+
[`Attachment.record()`](https://developer.apple.com/documentation/testing/attachment/record(_:named:sourcelocation:))
321+
or, alternatively, allow for applying the `AttachmentSavingTrait` trait to a
322+
local scope. (Locally-scoped traits are another area we're looking at for a
323+
future proposal.)
324+
325+
- There is interest in augmenting [`ConditionTrait`](https://developer.apple.com/documentation/testing/conditiontrait)
326+
to allow for boolean operations on them. It would make sense to add such
327+
functionality to `AttachmentSavingTrait` too.
328+
329+
## Alternatives considered
330+
331+
- Directly mapping XCTest's [`lifetime`](https://developer.apple.com/documentation/xctest/xctattachment/lifetime-swift.property)
332+
property to Swift Testing. Swift Testing presents the opportunity to improve
333+
upon this interface in ways that don't map cleanly to Objective-C.
334+
335+
- Adding a `shouldBeSaved: Bool` property to [`Attachment`](https://developer.apple.com/documentation/testing/attachment).
336+
While developers can create an instance of [`Attachment`](https://developer.apple.com/documentation/testing/attachment)
337+
before calling [`Attachment.record()`](https://developer.apple.com/documentation/testing/attachment/record(_:sourcelocation:)),
338+
it is typically more ergonomic to pass the attachable value directly. Thus a
339+
property on [`Attachment`](https://developer.apple.com/documentation/testing/attachment)
340+
is less accessible than other alternatives.
341+
342+
- Adding a `save: Bool` parameter to [`Attachment.record()`](https://developer.apple.com/documentation/testing/attachment/record(_:named:sourcelocation:)).
343+
In our experience, it's frequently the case that a test author wants to
344+
conditionally save an attachment based on whether a test fails (or some other
345+
external factor) and won't know if an attachment should be saved until after
346+
they've created it.
347+
348+
## Acknowledgments
349+
350+
Thanks to the team for their feedback on this proposal and to the Swift
351+
community for their continued interest in Swift Testing!

0 commit comments

Comments
 (0)