Attachment lifetimes (#1319)

This PR introduces a new experimental trait, `.savingAttachments(if:)`,
that can be used to control whether a test's attachments are saved or
not.

XCTest has API around the
[`XCTAttachment.Lifetime`](https://developer.apple.com/documentation/xctest/xctattachment/lifetime-swift.enum)
enumeration that developers can use to control whether attachments are
saved to a test report in Xcode. This enumeration has two cases:

```objc
/*
 * Attachment will be kept regardless of the outcome of the test.
 */
XCTAttachmentLifetimeKeepAlways = 0,

/*
 * Attachment will only be kept when the test fails, and deleted otherwise.
 */
XCTAttachmentLifetimeDeleteOnSuccess = 1
```

I've opted to implement something a bit more granular. A developer can
specify `.savingAttachments(if: .testFails)` and `.savingAttachments(if:
.testPasses)` or can call some custom function of their own design like
`runningInCI` or `hasPlentyOfFloppyDiskSpace`. The default behaviour if
this trait is not used is to always save attachments, which is
equivalent to `XCTAttachmentLifetimeKeepAlways`.
`XCTAttachmentLifetimeDeleteOnSuccess` is, in effect, equivalent to
`.savingAttachments(if: .testFails)`, but I hope reads a bit more
clearly in context.

Here's a usage example:

```swift
@test(.savingAttachments(if: .testFails))
func `best test ever`() {
  Attachment.record("...") // only saves to the test report or to disk if the
                           // next line is uncommented.
  // Issue.record("sadness")
}
```

I've taken the opportunity to update existing documentation for
`Attachment` and `Attachable` to try to use more consistent language: a
test records an attachment and then the testing library saves it
(somewhere). I'm sure I've missed some spots, so please point them out
if you see them.

Resolves rdar://138921461.

### Checklist:

- [x] Code and documentation should follow the style of the [Style
Guide](https://github.com/apple/swift-testing/blob/main/Documentation/StyleGuide.md).
- [x] If public symbols are renamed or modified, DocC references should
be updated.

Author: grynspan

Sources/Testing/Attachments/Attachable.swift

@@ -8,14 +8,15 @@
 // See https://swift.org/CONTRIBUTORS.txt for Swift project authors
 //
 
-/// A protocol describing a type that can be attached to a test report or
-/// written to disk when a test is run.
+private import _TestingInternals
+
+/// A protocol describing a type whose instances can be recorded and saved as
+/// part of a test run.
 ///
 /// To attach an attachable value to a test, pass it to ``Attachment/record(_:named:sourceLocation:)``.
 /// To further configure an attachable value before you attach it, use it to
 /// initialize an instance of ``Attachment`` and set its properties before
-/// passing it to ``Attachment/record(_:sourceLocation:)``. An attachable
-/// value can only be attached to a test once.
+/// passing it to ``Attachment/record(_:sourceLocation:)``.
 ///
 /// The testing library provides default conformances to this protocol for a
 /// variety of standard library types. Most user-defined types do not need to
@@ -36,8 +37,8 @@ public protocol Attachable: ~Copyable {
   /// an attachment.
   ///
   /// The testing library uses this property to determine if an attachment
-  /// should be held in memory or should be immediately persisted to storage.
-  /// Larger attachments are more likely to be persisted, but the algorithm the
+  /// should be held in memory or should be immediately saved. Larger
+  /// attachments are more likely to be saved immediately, but the algorithm the
   /// testing library uses is an implementation detail and is subject to change.
   ///
   /// The value of this property is approximately equal to the number of bytes
@@ -66,13 +67,12 @@ public protocol Attachable: ~Copyable {
   /// - Throws: Whatever is thrown by `body`, or any error that prevented the
   ///   creation of the buffer.
   ///
-  /// The testing library uses this function when writing an attachment to a
-  /// test report or to a file on disk. The format of the buffer is
-  /// implementation-defined, but should be "idiomatic" for this type: for
-  /// example, if this type represents an image, it would be appropriate for
-  /// the buffer to contain an image in PNG format, JPEG format, etc., but it
-  /// would not be idiomatic for the buffer to contain a textual description of
-  /// the image.
+  /// The testing library uses this function when saving an attachment. The
+  /// format of the buffer is implementation-defined, but should be "idiomatic"
+  /// for this type: for example, if this type represents an image, it would be
+  /// appropriate for the buffer to contain an image in PNG format, JPEG format,
+  /// etc., but it would not be idiomatic for the buffer to contain a textual
+  /// description of the image.
   ///
   /// @Metadata {
   ///   @Available(Swift, introduced: 6.2)
@@ -91,9 +91,8 @@ public protocol Attachable: ~Copyable {
   /// - Returns: The preferred name for `attachment`.
   ///
   /// The testing library uses this function to determine the best name to use
-  /// when adding `attachment` to a test report or persisting it to storage. The
-  /// default implementation of this function returns `suggestedName` without
-  /// any changes.
+  /// when saving `attachment`. The default implementation of this function
+  /// returns `suggestedName` without any changes.
   ///
   /// @Metadata {
   ///   @Available(Swift, introduced: 6.2)

Sources/Testing/Attachments/AttachableWrapper.swift

@@ -8,9 +8,8 @@
 // See https://swift.org/CONTRIBUTORS.txt for Swift project authors
 //
 
-/// A protocol describing a type that can be attached to a test report or
-/// written to disk when a test is run and which contains another value that it
-/// stands in for.
+/// A protocol describing a type whose instances can be recorded and saved as
+/// part of a test run and which contains another value that it stands in for.
 ///
 /// To attach an attachable value to a test, pass it to ``Attachment/record(_:named:sourceLocation:)``.
 /// To further configure an attachable value before you attach it, use it to

Sources/Testing/Attachments/Attachment.swift

@@ -13,11 +13,24 @@ private import _TestingInternals
 /// A type describing values that can be attached to the output of a test run
 /// and inspected later by the user.
 ///
-/// Attachments are included in test reports in Xcode or written to disk when
-/// tests are run at the command line. To create an attachment, you need a value
-/// of some type that conforms to ``Attachable``. Initialize an instance of
-/// ``Attachment`` with that value and, optionally, a preferred filename to use
-/// when writing to disk.
+/// To create an attachment, you need a value of some type that conforms to
+/// ``Attachable``. Initialize an instance of ``Attachment`` with that value
+/// and, optionally, a preferred filename to use when saving the attachment. To
+/// record the attachment, call ``Attachment/record(_:sourceLocation:)``.
+/// Alternatively, pass your attachable value directly to ``Attachment/record(_:named:sourceLocation:)``.
+///
+/// By default, the testing library saves your attachments as soon as you call
+/// ``Attachment/record(_:sourceLocation:)`` or
+/// ``Attachment/record(_:named:sourceLocation:)``. You can access saved
+/// attachments after your tests finish running:
+///
+/// - When using Xcode, you can access attachments from the test report.
+/// - When using Visual Studio Code, the testing library saves attachments to
+///   `.build/attachments` by default. Visual Studio Code reports the paths to
+///   individual attachments in its Tests Results panel.
+/// - When using Swift Package Manager's `swift test` command, you can pass the
+///   `--attachments-path` option. The testing library saves attachments to the
+///   specified directory.
 ///
 /// @Metadata {
 ///   @Available(Swift, introduced: 6.2)
@@ -41,16 +54,17 @@ public struct Attachment<AttachableValue> where AttachableValue: Attachable & ~C
   /// Storage for ``attachableValue-7dyjv``.
   private var _storage: Storage
 
-  /// The path to which the this attachment was written, if any.
+  /// The path to which the this attachment was saved, if any.
   ///
   /// If a developer sets the ``Configuration/attachmentsPath`` property of the
   /// current configuration before running tests, or if a developer passes
   /// `--attachments-path` on the command line, then attachments will be
-  /// automatically written to disk when they are attached and the value of this
-  /// property will describe the path where they were written.
+  /// automatically saved when they are attached and the value of this property
+  /// will describe the paths where they were saved. A developer can use the
+  /// ``AttachmentSavingTrait`` trait type to defer or skip saving attachments.
   ///
-  /// If no destination path is set, or if an error occurred while writing this
-  /// attachment to disk, the value of this property is `nil`.
+  /// If no destination path is set, or if an error occurred while saving this
+  /// attachment, the value of this property is `nil`.
   @_spi(ForToolsIntegrationOnly)
   public var fileSystemPath: String?
 
@@ -62,8 +76,7 @@ public struct Attachment<AttachableValue> where AttachableValue: Attachable & ~C
   /// Storage for ``preferredName``.
   fileprivate var _preferredName: String?
 
-  /// A filename to use when writing this attachment to a test report or to a
-  /// file on disk.
+  /// A filename to use when saving this attachment.
   ///
   /// The value of this property is used as a hint to the testing library. The
   /// testing library may substitute a different filename as needed. If the
@@ -106,9 +119,9 @@ extension Attachment where AttachableValue: ~Copyable {
   /// - Parameters:
   ///   - attachableValue: The value that will be attached to the output of the
   ///     test run.
-  ///   - preferredName: The preferred name of the attachment when writing it to
-  ///     a test report or to disk. If `nil`, the testing library attempts to
-  ///     derive a reasonable filename for the attached value.
+  ///   - preferredName: The preferred name of the attachment to use when saving
+  ///     it. If `nil`, the testing library attempts to generate a reasonable
+  ///     filename for the attached value.
   ///   - sourceLocation: The source location of the call to this initializer.
   ///     This value is used when recording issues associated with the
   ///     attachment.
@@ -248,11 +261,11 @@ extension Attachment where AttachableValue: Sendable & ~Copyable {
   ///
   /// When `attachableValue` is an instance of a type that does not conform to
   /// the [`Sendable`](https://developer.apple.com/documentation/swift/sendable)
-  /// protocol, the testing library encodes it as data immediately. If
-  /// `attachableValue` throws an error when the testing library attempts to
-  /// encode it, the testing library records that error as an issue in the
-  /// current test and does not write the attachment to the test report or to
-  /// persistent storage.
+  /// protocol, the testing library calls its ``Attachable/withUnsafeBytes(for:_:)``
+  /// immediately and records a copy of the resulting buffer instead. If
+  /// `attachableValue` throws an error when the testing library calls its
+  /// ``Attachable/withUnsafeBytes(for:_:)`` function, the testing library
+  /// records that error as an issue in the current test.
   ///
   /// @Metadata {
   ///   @Available(Swift, introduced: 6.2)
@@ -273,18 +286,18 @@ extension Attachment where AttachableValue: Sendable & ~Copyable {
   ///
   /// - Parameters:
   ///   - attachableValue: The value to attach.
-  ///   - preferredName: The preferred name of the attachment when writing it to
-  ///     a test report or to disk. If `nil`, the testing library attempts to
-  ///     derive a reasonable filename for the attached value.
+  ///   - preferredName: The preferred name of the attachment to use when saving
+  ///     it. If `nil`, the testing library attempts to generate a reasonable
+  ///     filename for the attached value.
   ///   - sourceLocation: The source location of the call to this function.
   ///
   /// When `attachableValue` is an instance of a type that does not conform to
   /// the [`Sendable`](https://developer.apple.com/documentation/swift/sendable)
-  /// protocol, the testing library encodes it as data immediately. If
-  /// `attachableValue` throws an error when the testing library attempts to
-  /// encode it, the testing library records that error as an issue in the
-  /// current test and does not write the attachment to the test report or to
-  /// persistent storage.
+  /// protocol, the testing library calls its ``Attachable/withUnsafeBytes(for:_:)``
+  /// immediately and records a copy of the resulting buffer instead. If
+  /// `attachableValue` throws an error when the testing library calls its
+  /// ``Attachable/withUnsafeBytes(for:_:)`` function, the testing library
+  /// records that error as an issue in the current test.
   ///
   /// This function creates a new instance of ``Attachment`` and immediately
   /// attaches it to the current test.
@@ -308,11 +321,11 @@ extension Attachment where AttachableValue: ~Copyable {
   ///
   /// When `attachableValue` is an instance of a type that does not conform to
   /// the [`Sendable`](https://developer.apple.com/documentation/swift/sendable)
-  /// protocol, the testing library encodes it as data immediately. If
-  /// `attachableValue` throws an error when the testing library attempts to
-  /// encode it, the testing library records that error as an issue in the
-  /// current test and does not write the attachment to the test report or to
-  /// persistent storage.
+  /// protocol, the testing library calls its ``Attachable/withUnsafeBytes(for:_:)``
+  /// immediately and records a copy of the resulting buffer instead. If
+  /// `attachableValue` throws an error when the testing library calls its
+  /// ``Attachable/withUnsafeBytes(for:_:)`` function, the testing library
+  /// records that error as an issue in the current test.
   ///
   /// @Metadata {
   ///   @Available(Swift, introduced: 6.2)
@@ -332,18 +345,18 @@ extension Attachment where AttachableValue: ~Copyable {
   ///
   /// - Parameters:
   ///   - attachableValue: The value to attach.
-  ///   - preferredName: The preferred name of the attachment when writing it to
-  ///     a test report or to disk. If `nil`, the testing library attempts to
-  ///     derive a reasonable filename for the attached value.
+  ///   - preferredName: The preferred name of the attachment to use when saving
+  ///     it. If `nil`, the testing library attempts to generate a reasonable
+  ///     filename for the attached value.
   ///   - sourceLocation: The source location of the call to this function.
   ///
   /// When `attachableValue` is an instance of a type that does not conform to
   /// the [`Sendable`](https://developer.apple.com/documentation/swift/sendable)
-  /// protocol, the testing library encodes it as data immediately. If
-  /// `attachableValue` throws an error when the testing library attempts to
-  /// encode it, the testing library records that error as an issue in the
-  /// current test and does not write the attachment to the test report or to
-  /// persistent storage.
+  /// protocol, the testing library calls its ``Attachable/withUnsafeBytes(for:_:)``
+  /// immediately and records a copy of the resulting buffer instead. If
+  /// `attachableValue` throws an error when the testing library calls its
+  /// ``Attachable/withUnsafeBytes(for:_:)`` function, the testing library
+  /// records that error as an issue in the current test.
   ///
   /// This function creates a new instance of ``Attachment`` and immediately
   /// attaches it to the current test.
@@ -372,10 +385,9 @@ extension Attachment where AttachableValue: ~Copyable {
   /// - Throws: Whatever is thrown by `body`, or any error that prevented the
   ///   creation of the buffer.
   ///
-  /// The testing library uses this function when writing an attachment to a
-  /// test report or to a file on disk. This function calls the
-  /// ``Attachable/withUnsafeBytes(for:_:)`` function on this attachment's
-  /// ``attachableValue-2tnj5`` property.
+  /// The testing library uses this function when saving an attachment. This
+  /// function calls the ``Attachable/withUnsafeBytes(for:_:)`` function on this
+  /// attachment's ``attachableValue-2tnj5`` property.
   ///
   /// @Metadata {
   ///   @Available(Swift, introduced: 6.2)
@@ -404,16 +416,16 @@ extension Attachment where AttachableValue: ~Copyable {
   /// is derived from the value of the ``Attachment/preferredName`` property.
   ///
   /// If you pass `--attachments-path` to `swift test`, the testing library
-  /// automatically uses this function to persist attachments to the directory
-  /// you specify.
+  /// automatically uses this function to save attachments to the directory you
+  /// specify.
   ///
   /// This function does not get or set the value of the attachment's
   /// ``fileSystemPath`` property. The caller is responsible for setting the
   /// value of this property if needed.
   ///
-  /// This function is provided as a convenience to allow tools authors to write
-  /// attachments to persistent storage the same way that Swift Package Manager
-  /// does. You are not required to use this function.
+  /// This function is provided as a convenience to allow tools authors to save
+  /// attachments the same way that Swift Package Manager does. You are not
+  /// required to use this function.
   @_spi(ForToolsIntegrationOnly)
   public borrowing func write(toFileInDirectoryAtPath directoryPath: String) throws -> String {
     try write(
@@ -505,9 +517,9 @@ extension Configuration {
   /// - Returns: Whether or not to continue handling the event.
   ///
   /// This function is called automatically by ``handleEvent(_:in:)``. You do
-  /// not need to call it elsewhere. It automatically persists the attachment
+  /// not need to call it elsewhere. It automatically saves the attachment
   /// associated with `event` and modifies `event` to include the path where the
-  /// attachment was stored.
+  /// attachment was saved.
   func handleValueAttachedEvent(_ event: inout Event, in eventContext: borrowing Event.Context) -> Bool {
     guard let attachmentsPath else {
       // If there is no path to which attachments should be written, there's
@@ -519,9 +531,9 @@ extension Configuration {
       preconditionFailure("Passed the wrong kind of event to \(#function) (expected valueAttached, got \(event.kind)). Please file a bug report at https://github.com/swiftlang/swift-testing/issues/new")
     }
     if attachment.fileSystemPath != nil {
-      // Somebody already persisted this attachment. This isn't necessarily a
-      // logic error in the testing library, but it probably means we shouldn't
-      // persist it again. Suppress the event.
+      // Somebody already saved this attachment. This isn't necessarily a logic
+      // error in the testing library, but it probably means we shouldn't save
+      // it again. Suppress the event.
       return false
     }
 

Sources/Testing/CMakeLists.txt

@@ -100,6 +100,7 @@ add_library(Testing
   Test+Discovery.swift
   Test+Discovery+Legacy.swift
   Test+Macro.swift
+  Traits/AttachmentSavingTrait.swift
   Traits/Bug.swift
   Traits/Comment.swift
   Traits/Comment+Macro.swift

Sources/Testing/Events/Event.swift

@@ -191,6 +191,16 @@ public struct Event: Sendable {
   /// The instant at which the event occurred.
   public var instant: Test.Clock.Instant
 
+#if DEBUG
+  /// Whether or not this event was deferred.
+  ///
+  /// A deferred event is handled significantly later than when was posted.
+  ///
+  /// We currently use this property in our tests, but do not expose it as API
+  /// or SPI. We can expose it in the future if tools need it.
+  var wasDeferred: Bool = false
+#endif
+
   /// Initialize an instance of this type.
   ///
   /// - Parameters:

Sources/Testing/Running/Configuration+EventHandling.swift

@@ -23,18 +23,6 @@ extension Configuration {
     var contextCopy = copy context
     contextCopy.configuration = self
     contextCopy.configuration?.eventHandler = { _, _ in }
-
-#if !SWT_NO_FILE_IO
-    if case .valueAttached = event.kind {
-      var eventCopy = copy event
-      guard handleValueAttachedEvent(&eventCopy, in: contextCopy) else {
-        // The attachment could not be handled, so suppress this event.
-        return
-      }
-      return eventHandler(eventCopy, contextCopy)
-    }
-#endif
-
     return eventHandler(event, contextCopy)
   }
 }

Sources/Testing/Running/Runner.RuntimeState.swift

@@ -47,9 +47,19 @@ extension Runner {
       return
     }
 
-    configuration.eventHandler = { [eventHandler = configuration.eventHandler] event, context in
+    configuration.eventHandler = { [oldEventHandler = configuration.eventHandler] event, context in
+#if !SWT_NO_FILE_IO
+      var event = copy event
+      if case .valueAttached = event.kind {
+        guard let configuration = context.configuration,
+              configuration.handleValueAttachedEvent(&event, in: context) else {
+          // The attachment could not be handled, so suppress this event.
+          return
+        }
+      }
+#endif
       RuntimeState.$current.withValue(existingRuntimeState) {
-        eventHandler(event, context)
+        oldEventHandler(event, context)
       }
     }
   }