Skip to content

Commit 3f984e6

Browse files
jckarterjckarter
authored
Merge pull request #1203 from maxovtsin/NNNN-opt-in-reflection-metadata
[Proposal] Opt-In Reflection metadata
2 parents 01a916c + d969b4e commit 3f984e6

File tree

1 file changed

+246
-0
lines changed

1 file changed

+246
-0
lines changed

‎proposals/0379-opt-in-reflection-metadata.md

Lines changed: 246 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,246 @@
1+
2+
# Swift Opt-In Reflection Metadata
3+
4+
* Proposal: [SE-0379](0379-opt-in-reflection-metadata.md)
5+
* Authors: [Max Ovtsin](https://github.com/maxovtsin)
6+
* Review Manager: [Joe Groff](https://github.com/jckarter)
7+
* Status: **Active review (November 30...December 14, 2022)**
8+
* Implementation: [apple/swift#34199](https://github.com/apple/swift/pull/34199)
9+
10+
## Introduction
11+
12+
This proposal seeks to increase the safety, efficiency, and secrecy of Swift Reflection Metadata by improving the existing mechanism and providing the opportunity to express a requirement on Reflection Metadata in APIs that consume it.
13+
14+
Swift-evolution thread: [Discussion thread topic for that proposal](https://forums.swift.org/t/pitch-3-opt-in-reflection-metadata/58852)
15+
16+
17+
## Motivation
18+
19+
There are two main kinds of Swift metadata emitted by the compiler:
20+
21+
1. Core Metadata (type metadata records, nominal type descriptors, etc).
22+
2. Reflection metadata (reflection metadata field descriptors).
23+
24+
Core metadata must constantly be emitted and may only be stripped if provenly not used. (This kind of metadata isn't affected by this proposal.)
25+
On the other hand, reflection metadata contains optional information about declaration fields - their names and references to their types.
26+
The language's runtime features don't use this metadata, and the emission may be skipped if such types aren't passed to reflection-consuming APIs.
27+
28+
29+
Currently, there is no way to selectively enable the emission of reflectable metadata for a type or understand if an API consumes reflection metadata under the hood.
30+
Moreover, compiler's flags exist that allow to completely disable emission.
31+
32+
A developer has two ways right now - either
33+
1. To just in case enable Reflection in full.
34+
2. To try to guess which used APIs consume Reflection, and enable it only for modules that are users of such APIs.
35+
36+
Both of those options have flaws. The first one leads to exsessive contribution of reflection metadta to binary size and might affects the secrecy of generated code.
37+
The second one isn't safe because many APIs are black boxes if the guess is wrong, an app might behave not as expected at runtime.
38+
39+
Furthermore, APIs can use Reflection Metadata differently. Some like `print`, `debugPrint`, and `dump` will still work with disabled reflection, but the output will be limited.
40+
Others, like SwiftUI, rely on it and won't work correctly if the reflection metadata is missing.
41+
While the former can benefit as well, the main focus of this proposal is on the latter.
42+
43+
A developer can mistakenly turn off Reflection Metadata for a Swift module and won't be warned at compile-time if APIs that consume reflection are used by that module.
44+
An app with such a module won't behave as expected at runtime which may be challenging to notice and track down such bugs back to Reflection.
45+
For instance, SwiftUI implementation uses reflection metadata from user modules to trigger the re-rendering of the view hierarchy when a state has changed.
46+
If for some reason a user module was compiled with metadata generation disabled, changing the state won't trigger that behavior and will cause inconsistency
47+
between state and representation which will make such API less safe since it becomes a runtime issue rather than a compile-time one.
48+
49+
On the other hand, excessive Reflection metadata may be preserved in a binary even if not used, because there is currently no way to statically determine its usage.
50+
There was an attempt to limit the amount of unused reflection metadata by improving its stripability by the Dead Code Elimination LLVM pass, but in many cases,
51+
it’s still preserved in the binary because it’s referenced by Full Type Metadata which prevents Reflection Metadata from stripping.
52+
This unnecessarily increases the binary size and may simplify reverse-engineering.
53+
54+
Introducing a static compilation check can help to solve both of mentioned issues by adding to the language a way to express the requirement to have Reflection metadata at runtime.
55+
56+
57+
## Proposed solution
58+
59+
Teaching the Type-checker and IRGen to ensure Reflection metadata is preserved in a binary if reflection-consuming APIs are used, will help to move the problem from runtime to compile time.
60+
61+
To achieve that, a new marker protocol `Reflectable` will be introduced. Firstly, APIs developers will gain an opportunity to express a dependency on Reflection Metadata through a generic requirement of their functions, which will make such APIs safer.
62+
Secondly, during IRGen, the compiler will be able to selectively emit Reflection symbols for the types that explicitly conform to the `Reflectable` protocol, which will reduce the overhead from reflection symbols for cases when reflection is emitted but not consumed.
63+
64+
### Case Study 1:
65+
66+
SwiftUI:
67+
```swift
68+
protocol SwiftUI.View: Reflectable {}
69+
class NSHostingView<Content> where Content : View {
70+
init(rootView: Content) { ... }
71+
}
72+
```
73+
User module:
74+
```swift
75+
import SwiftUI
76+
77+
struct SomeModel {}
78+
79+
struct SomeView: SwiftUI.View {
80+
var body: some View {
81+
Text("Hello, World!")
82+
.frame(...)
83+
}
84+
}
85+
86+
window.contentView = NSHostingView(rootView: SomeView())
87+
```
88+
Reflection metadata for `SomeView` will be emitted because it implicitly conforms to `Reflectable` protocol, while for `SomeModel` Reflection metadata won't be emitted. If the user module gets compiled with the reflection metadata disabled, the compiler will emit an error.
89+
90+
91+
### Case Study 2:
92+
93+
Framework:
94+
```swift
95+
public func foo<T: Reflectable>(_ t: T) { ... }
96+
```
97+
User module:
98+
```swift
99+
struct Bar: Reflectable {}
100+
foo(Bar())
101+
```
102+
Reflection metadata for `Bar` will be emitted because it explicitly conforms to Reflectable protocol. Without conformance to Reflectable, an instance of type Bar can't be used on function `foo`. If the user module gets compiled with the reflection metadata disabled, the compiler will emit an error.
103+
104+
105+
### Conditional and Force casts (`as? Reflectable`, `as! Reflectable`, `is Reflectable`)
106+
107+
We also propose to allow conditional and force casts to the `Reflectable` protocol, which would succeed only if Reflection Metadata related to a type is available at runtime. This would allow developers to explicitly check if reflection metadata is present and based on that fact branch the code accordingly.
108+
109+
```swift
110+
public func conditionalUse<T>(_ t: T) {
111+
if let _t = t as? Reflectable { // Consume Reflection metadata
112+
} else { // Back to default implementation }
113+
}
114+
115+
public func forceUse<T>(_ t: T) {
116+
debugPrint(t as! Reflectable) // Will crash if reflection metadata isn't available
117+
}
118+
119+
public func testIsReflectable<T>(_ t: T) -> Bool {
120+
return t is Reflectable // returns True if reflection is available
121+
}
122+
```
123+
124+
### Behavior change for Swift 6
125+
126+
Starting with Swift 6, we propose to enable Opt-In mode by default, to make the user experience consistent and safe.
127+
However, if full reflection isn't enabled with a new flag (`-enable-full-reflection-metadata`), the emission of reflection metadata will be skipped for all types that don't conform to the `Reflectable` protocol.
128+
This may cause changes in the behavior of the code that wasn't audited to conform to Reflectable and uses reflection-consuming APIs.
129+
130+
For instance, stdlib's APIs like `dump`, `debugPrint`, `String(describing:)` will be returning limited output.
131+
Library authors will have to prepare their APIs for Swift 6 and introduce generic requirements on `Reflectable` in their APIs.
132+
133+
We also propose to deprecate the compiler's options that can lead to missing reflection - `-reflection-metadata-for-debugger-only` and `-disable-reflection-metadata` and starting with Swift 6, ignore these arguments in favor of the default opt-in mode.
134+
135+
136+
### Stdlib behavior changes
137+
138+
In Swift `Mirror(reflecting:)` is the only official way to access Reflection metadata, all other APIs are using it under the hood.
139+
We intentionally do not propose adding a Reflectable constraint on Mirror type, because it would impose restrictions on those developers who still don't want to require it and consume Reflection optionally.
140+
If the presence of reflection metadata is mandatory, the requirement on Reflectable protocol should be expressed in the signatures of calling functions.
141+
142+
143+
## Detailed design
144+
145+
To decide when to emit reflection metadata IRGen will check the conformance of a type to the `Reflectable` protocol and if the type conforms, IRGen will emit reflection symbols.
146+
147+
Conformance to Reflectable should be only allowed at type declarations level, to avoid confusing behavior, when a developer adds conformance on an imported from another module type that doesn't have reflection enabled.
148+
149+
Transitive conformance to Reflectable should be allowed to give API authors an opportunity to hide reflection logic from APIs users as implementation details.
150+
151+
```swift
152+
// Library
153+
public protocol Foo: Reflectable {}
154+
public func consume<T: Foo>(_ t: T) {}
155+
156+
// User
157+
struct Bar: Foo {} // Reflection is emitted
158+
consume(Bar())
159+
```
160+
161+
### Changes for debugging
162+
163+
Since Reflection metadata might be used by the debugger, we propose to always keep that metadata
164+
if full emission of debugging information is enabled (with `-gdwarf-types` or `-g` flags).
165+
However, such Reflection metadata won't be accessible through the nominal type descriptor
166+
which will allow to avoid inconsistencies between APIs' outputs in Release and Debug modes.
167+
168+
### Changes in flags
169+
170+
To handle behavior change between Swift pre-6 and 6, we can introduce a new upcoming feature,
171+
which will allow to enable Opt-In mode explicitly for Swift pre-6 with `-enable-upcoming-feature OptInReflection` and will set this mode by default in Swift 6.
172+
173+
A new flag `-enable-full-reflection-metadata` will also have to be introduced to allow developers to enable reflection in full if they desire in Swift 6 and later.
174+
175+
For Swift 6, flags `-disable-reflection-metadata` and `-emit-reflection-for-debugger` will be a no-op, to ensure the reflection metadata is always available when needed.
176+
177+
1. Reflection Disabled (`-disable-reflection-metadata` and `-reflection-metadata-for-debugger-only`)
178+
- For Swift pre-6 emit Reflection metadata only if full debugging information is enabled.
179+
- If there is a type in a module conforming to `Reflectable`, the compiler will emit an error.
180+
- A no-op in Swift 6 and later (Opt-in mode is enabled by default).
181+
182+
2. Opt-In Reflection (`-enable-upcoming-feature OptInReflection`)
183+
- If debugging is disabled, emit only for types that conform to `Reflectable`.
184+
- Will emit reflection in full for all types if debugging is enabled.
185+
- For Swift pre-6 will require an explicit flag, for Swift 6 will be enabled by default.
186+
187+
3. Fully enabled (`-enable-full-reflection-metadata`)
188+
- Always emit reflection metadata for all types regardless of debugging information level.
189+
- Conformance to Reflectable will be synthesized for all types to allow usage of reflection-consuming APIs.
190+
- Current default level for Swift pre-6.
191+
192+
Introducing a new flag to control the feature will allow us to safely roll it out and avoid breakages of the existing code. For those modules that get compiled with fully enabled metadata, nothing will change (all symbols will stay present). For modules that have the metadata disabled, but are consumers of reflectable API, the compiler will emit the error enforcing the guarantee.
193+
194+
### Casts implementation
195+
196+
Casting might be a good way to improve the feature's ergonomics because currently there is no way to check if reflection is available at runtime. (`Mirror.children.count` doesn't really help because it doesn't distinguish between the absence of reflection metadata and the absence of fields on a type)
197+
198+
To implement this feature, we propose to introduce a new runtime function `swift_reflectableCast`, and emit a call to it instead of `swift_dynamicCast`during IRGen if Reflectable is a target type.
199+
200+
Because of the fact that the compiler emits a call to that function at compile-time, all casts must be statically visible.
201+
All other cases like implicit conversion to `Reflectable` must be banned.
202+
This could be done at CSSimplify, when a new conversion constraint is introduced between a type variable and `Reflectable` type, the compiler will emit an error.
203+
204+
```swift
205+
func cast<T, U>(_ x: U) -> T {
206+
return x as! T
207+
}
208+
let a = cast(1) as Reflectable // expression can't be implicitly converted to Reflectable; use 'as? Reflectable' or 'as! Reflectable' instead
209+
let b: Reflectable = cast(1) // expression can't be implicitly converted to Reflectable; use 'as? Reflectable' or 'as! Reflectable' instead
210+
```
211+
Some diagnostics and optimizations will also have to be disabled even if conformance is statically visible to the compiler because all casts will have to go through the runtime call.
212+
213+
**Availability checks**
214+
Since reflectable casting will require a new runtime function, it should be gated by availability checks. If a deployment target is lower than supported, an error will be emitted. However, it might be possible to embed new runtime functions into a compatibility library for back deployment.
215+
216+
217+
## Source compatibility
218+
219+
The change won’t break source compatibility in versions prior to Swift 6, because of the gating by the new flag.
220+
If as proposed, it’s enabled by default in Swift 6, the code with types that has not been audited to conform to the `Reflectable` protocol will fail to compile if used with APIs that consume the reflection metadata.
221+
222+
223+
## Effect on ABI stability
224+
225+
`Reflectable` is a marker protocol, which doesn't have a runtime representation, has no requirements and doesn't affect ABI.
226+
227+
## Effect on API resilience
228+
229+
This proposal has no effect on API resilience.
230+
231+
## Alternatives considered
232+
233+
Dead Code Elimination and linker optimisations were also considered as a way to reduce the amount of present Reflection metadata in release builds.
234+
The optimiser could use a conformance to a `Reflectable` protocol as a hint about what reflection metadata should be preserved.
235+
However, turned out it was quite challenging to statically determine all usages of Reflection metadata even with hints.
236+
237+
It was also considered to use an attribute `@reflectable` on nominal type declaration to express the requirement to have reflection metadata, however, a lot of logic had to be re-implemented outside of type-checker to ensure all guarantees are fulfilled.
238+
239+
## Future directions
240+
241+
Currently, there is only one kind of Reflection Metadata - Field Descriptor Metadata. In the future, it is possible that other kinds will be added (e.g methods, computed properties, etc) `Reflectable` should be able to cover all of them.
242+
If this proposal is approved, it will become easier and more native to migrate Codable to the usage of Reflection metadata for encoding/decoding logic instead of autogenerating code at compile time.
243+
244+
## Acknowledgments
245+
246+
Thanks to [Joe Groff](https://github.com/jckarter) for various useful pieces of advice, general help along the way, and for suggesting several useful features like Reflectable casts!

0 commit comments

Comments
 (0)