Controlling function definition visibility in clients

proposals/nnnn-definition-visibility.md

@@ -15,7 +15,7 @@ On the other hand, making the function definition available to the caller means
 
 The `@inlinable` attribute introduced in [SE-0193](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0193-cross-module-inlining-and-specialization.md) provides the ability to explicitly make the definition of a function available for callers. It ensures that the definition can be specialized, inlined, or otherwise used in clients to produce better code. However, it also compiles the definition into the module's binary so that the caller can choose to call it directly without emitting a copy of the definition. The `@inlinable` attribute has very little to do with inlining per se; rather, it's about visibility of the definition.
 
-The Swift compiler has optimizations that make some function definitions implicitly available across modules. The primary one is cross-module-optimization (CMO), which is enabled by default in release builds with the Swift Package Manager. A more aggressive form of cross-module optimization is used in [Embedded Swift](https://github.com/swiftlang/swift-evolution/blob/main/visions/embedded-swift.md), where it is necessary to (for example) ensure that all generic functions and types get specialized.
+This proposal provides explicit control over whether a function (1) generates a callable symbol in a binary and (2) makes its definition available for callers outside the module to be used for specialization, inlining, or other optimizations.
 
 ## Motivation
 
@@ -25,6 +25,63 @@ The Embedded Swift compilation model, in particular its use of aggressive cross-
 
 The `@inlinable` attribute provides explicit permission to the compiler to expose the definition of a function to its callers. However, the examples above illustrate that more control over when a function definition is emitted into a binary is needed for certain cases.
 
+### Existing controls for symbols and exposing function definitions
+
+The Swift language model itself mostly avoids defining what symbols are emitted into the binary when compiling code. However, there are some places in the language where the presence of a symbol in the final binary has been implied:
+
+* `@c` declarations ([SE-0495](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0495-cdecl.md)) and `@objc` classes need to produce symbols that can be referenced by compilers for the C and Objective-C languages, respectively.
+* The `@main` attribute ([SE-0281](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0281-main-attribute.md)) needs to produce a symbol that is known to the operating system's loader as an entry point.
+* The `@section` and `@used` attributes ([SE-0492](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0492-section-control.md)) imply that the compiler should produce a symbol.
+
+Similarly, whether the definition of a function is available to callers or not is mostly outside of the realm of the language. However, it has been touched on by several language features:
+
+* Library Evolution ([SE-0260](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0260-library-evolution.md)) explicitly ensures that clients cannot see the definition of a function within another module that was compiled with library evolution. 
+* The `@inlinable` attribute ([SE-0193](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0193-cross-module-inlining-and-specialization.md)) explicitly allows clients to see the definition of a function across modules. This attribute became particularly important with Library Evolution (above), which normally prevents clients from seeing the definition of a function.
+
+These are relatively indirect ways in which one can state whether a symbol should be generated for a function and whether a function's definition can be used by clients outside of the module. 
+
+### Effect of compiler optimizations
+
+Outside of those constraints on the interpretation of the language, the Swift compiler and build systems have an enormous amount of flexibility as to when to emit symbols and when to make the definition of functions available to clients. Various optimizations and compilation flags can affect both of these decisions. For example:
+
+* Incremental compilation (typical of debug builds) allows the compiler to avoid emitting symbols for `fileprivate` and `private` functions if they aren't needed elsewhere in the file, for example because all of their uses have been inlined (or there were no uses).
+* Whole-module optimization (WMO) allows the definitions of `internal` , `fileprivate`, and `private` functions to be available to other source files in the same module. The compiler may choose not to emit symbols for `internal`, `fileprivate`, or `private` entities at all if they aren't needed. (For example, because they've been inlined into all callers)
+* Cross-module optimization (CMO) allows the definitions of functions to be made available to clients in other modules. The "conservative" form of CMO, which has been enabled by the Swift Package Manager since Swift 5.8, does this primarily for `public` functions. A more aggressive form of cross-module optimization can also make the definitions of `internal`, `fileprivate`, or `private` entities available to clients (for the compiler's use only!).
+* [Embedded Swift](https://github.com/swiftlang/swift-evolution/blob/main/visions/embedded-swift.md) relies on WMO and the aggressive CMO described above. It will also avoid emitting symbols to binaries unless they appear to be needed, which helps reduce code size. It is also necessary, because Embedded Swift cannot create symbols for certain 
+
+The same Swift source code may very well be compiled in a number of different ways at different times: debug builds often use incremental compilation, release builds generally use WMO and conservative CMO, and an embedded build would use the more aggressive CMO. The differences in symbol availability and the use of function definitions by clients don't generally matter. It is expected that the default behavior may shift over time: for example, the build system might enable progressively more aggressive CMO to improve performance.
+
+This proposal provides a mechanism to explicitly state the intent to emit symbols or provide the function definition to clients independent of the compilation mode, optimization settings, or language features (from the prior section) that infer these properties. This can be important, for example, when some external system expects certain symbols to be present, but the compiler might not choose to emit the symbol in some cases.
+
+### Implementation hiding
+
+When the definition of a function is not available to clients, it can make use of declarations that are not available to those clients. For example, it can use `internal` or `private` declarations from the same module or file, respectively, that have not been marked `@usableFromInline`. It can also use declarations imported from other modules that were imported using an `internal` or `private` import ([SE-0409](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0409-access-level-on-imports.md)). 
+
+Although it was left to a [future direction in SE-0409](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0409-access-level-on-imports.md#hiding-dependencies-for-non-resilient-modules), implementation hiding can be used to avoid transitive dependencies on modules. For example, given the following setup:
+
+```swift
+// module A
+public func f() { }
+
+// module B
+@_implementationOnly internal import A
+
+public func g() {
+  f()
+}
+
+// module C
+import B
+
+func h() {
+  g()
+}
+```
+
+Module B makes use of module A only in its implementation, to call the function `A.f`. When module C imports module B, it conceptually does not need to know about module A. However, whether is true in practice depends on how the code is compiled: if `B` is built with library evolution enabled, then `C` does not need to know about `A`. If the modules are built with Embedded Swift, the definition of `B.g()` will be available to module `C`, so `C` will have to know about `A`.
+
+This can present a code portability problem for Embedded Swift. The proposed attribute that allows one to hide the definition of a function can help ensure that specific implementations stay hidden, making it possible to avoid transitive dependencies. It is by no means a complete solution: see the commentary about the effect of type layout on transitive dependencies in [SE-0409](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0409-access-level-on-imports.md). However, it is a practical solution for improving portability of Swift code across the different compilation modes.
+
 ## Proposed solution
 
 This proposal introduces a new attribute `@exported` that provides the required control over the ability of clients to make use of the callable interface or the definition of a particular function (or both). The `@exported` attribute takes one or both of the following arguments in parentheses:
@@ -36,13 +93,37 @@ The existing `@inlinable` for public symbols is subsumed by `@export(interface,
 
 ## Detailed design
 
-`@export` that includes the `implementation` argument inherits all of the restrictions as `@inlinable` that are outlined in SE-0193, for example, the definition itself can only reference public entities or those that are themselves `@usableFromInline`. 
+`@export` that includes the `implementation` argument inherits all of the restrictions as `@inlinable` that are outlined in [SE-0193](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0193-cross-module-inlining-and-specialization.md), for example, the definition itself can only reference public entities or those that are themselves `@usableFromInline`. 
 
 `@export` that includes `interface` always produces a symbol in the object file.
 
 `@export` cannot be used without arguments.
 
-## Relationship to `@inline(always)` / `@inline(never)`
+### Relationship to access control
+
+The `@export` attribute is orthogonal to access control, because the visibility of a declaration for the programmer (`public`, `internal`, etc.) can be different from the visibility of its definition from the compiler's perspective, depending on what compiler optimizations are being used. For example, consider the following two modules:
+
+```swift
+// module A
+private func secret() { /* ... */ }
+
+public func f() {
+  secret()
+}
+
+// module B
+import A
+
+func g() {
+  f()
+}
+```
+
+Module B cannot call the function `secret` under any circumstance. However, with aggressive CMO or Embedded Swift, the compiler will still make the definition available when compiling `B`, which can be used to (for example) inline both `f()` and `secret` into the body of `g`. 
+
+If this behavior is not desired, the `secret` function could be marked as `@export(interface)` to ensure that it is compiled to a symbol that it usable from outside of module A. It is still `private`, meaning that it still cannot be referenced by source code outside of that file.
+
+### Relationship to `@inline(always)` / `@inline(never)`
 
 The `@inline(always)` attribute [under discussion now](https://forums.swift.org/t/pitch-inline-always-attribute/82040) instructs the compiler to inline the function definition. The existing `@inline(never)` prevents the compiler from inlining the function. These have an effect on the heuristics the compiler's optimizer uses to decide when to inline. That's a matter of policy, but it does not impact whether a binary provides a definition for the given symbol that other callers can use. The notion of inlining is orthogonal to that of definition visibility and symbol availability.
 
@@ -54,6 +135,28 @@ The following table captures the ways in which these attributes interact.
 | `@export(interface, implementation)` | Always inlined everywhere; a symbol exists that could only be used by non-Swift clients. | Never inlined; callers may emit their own definitions or may call the definition in the function's module. |
 | `@export(interface)`                 | Always inlined within the function's module; a symbol exists for callers outside the function's module. | Never inlined; callers may call the definition in the function's module. Use this to fully encapsulate a function definition so that it can be replaced at link time without affecting any other code. |
 
+### Embedded Swift limitations
+
+Embedded Swift depends on "monomorphizing" all generic functions, meaning that the compiler needs to produce a specialize with concrete generic arguments for every use in the program. It is not possible to emit a single generic implementation that works for all generic arguments. This requires the definition to be available for any module that might create a specialization:
+
+```swift
+// module A
+private func secretGeneric<T>(_: T) { }
+
+public func fGeneric<T>(_ value: T) {
+  secretGeneric(T)
+}
+
+// module B
+struct MyType { }
+
+func h() {
+  fGeneric(MyType()) // must specialize fGeneric<MyType> and secretGeneric<MyType>
+}
+```
+
+This means that generic functions are incompatible with `@export(interface)`, because there is no way to export a generic interface without the implementation.
+
 ## Source compatibility
 
 Introduces a new attribute. This could cause a source-compatibility problem with an attached macro of the same name, but otherwise has no impact.