Skip to content

Commit 71fcddd

Browse files
al45tairal45tair
authored
Merge pull request #2827 from swiftlang/revert-2826-revert-2825-binary-library-artifacts
[SE-0482] Binary Static Library Artifacts Proposal
2 parents 2b38692 + cba1748 commit 71fcddd

File tree

1 file changed

+165
-0
lines changed

1 file changed

+165
-0
lines changed

‎proposals/0482-swiftpm-static-library-binary-target-non-apple-platforms.md

Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
# Binary Static Library Dependencies
2+
3+
* Proposal: [SE-0482](0482-swiftpm-static-library-binary-target-non-apple-platforms.md)
4+
* Authors: [Daniel Grumberg](https://github.com/daniel-grumberg), [Max Desiatov](https://github.com/MaxDesiatov), [Franz Busch](https://github.com/FranzBusch)
5+
* Review Manager: [Kuba Mracek](https://github.com/kubamracek)
6+
* Status: **Active Review (2 May...15 May, 2005)**
7+
8+
<!--- *During the review process, add the following fields as needed:* --->
9+
10+
* Implementation: [swiftlang/swift-package-manager#6967](https://github.com/swiftlang/swift-package-manager/pull/6967) [swiftlang/swift-package-manager#8605](https://github.com/swiftlang/swift-package-manager/pull/8605)
11+
* Bugs: [Swift Package Manger Issue](https://github.com/swiftlang/swift-package-manager/issues/7035)
12+
13+
<!---
14+
* Decision Notes: [Rationale](https://forums.swift.org/), [Additional Commentary](https://forums.swift.org/)
15+
* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/...commit-ID.../proposals/NNNN-filename.md)
16+
* Previous Proposal: [SE-XXXX](XXXX-filename.md)
17+
--->
18+
19+
20+
## Introduction
21+
22+
Swift continues to grow as a cross-platform language supporting a wide variety of use cases from [programming embedded device](https://www.swift.org/blog/embedded-swift-examples/) to [server-side development](https://www.swift.org/documentation/server/) across a multitude of [operating systems](https://www.swift.org/documentation/articles/static-linux-getting-started.html).
23+
However, currently SwiftPM supports linking against binary dependencies on Apple platforms only.
24+
This proposal aims to make it possible to provide static library dependencies exposing a C interface on non-Apple platforms.
25+
26+
Swift-evolution thread:
27+
28+
## Motivation
29+
30+
The Swift Package Manager’s [`binaryTarget` type](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0272-swiftpm-binary-dependencies.md) lets packages vend libraries that either cannot be built in Swift Package Manager for technical reasons,
31+
or for which the source code cannot be published for legal or other reasons.
32+
33+
In the current version of SwiftPM, binary targets support the following:
34+
35+
* Libraries in an Xcode-oriented format called XCFramework, and only for Apple platforms, introduced in [SE-0272](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0272-swiftpm-binary-dependencies.md).
36+
* Executables through the use of artifact bundles introduced in [SE-0305](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0305-swiftpm-binary-target-improvements.md).
37+
38+
We aim here to bring a subset of the XCFramework capabilities to non-Apple platforms in a safe way.
39+
40+
While this proposal is specifically focused on binary static library dependencies without unexpected unresolved external symbols on non-Apple platforms,
41+
it tries to do so in a way that will not prevent broader future support for static libraries and dynamically linked libraries.
42+
43+
## Proposed solution
44+
45+
This proposal extends artifact bundles introduced by [SE-0305](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0305-swiftpm-binary-target-improvements.md) to include a new kind of artifact type to represent a binary library dependency: `staticLibrary`.
46+
The artifact manifest would encode the following information for each variant:
47+
48+
* The static library to pass to the linker.
49+
On Apple and Linux platforms, this would be `.a` files and on Windows it would be a `.lib` file.
50+
* Enough information to be able to use the library's API in the packages source code,
51+
i.e., headers and module maps for libraries exporting a C-based interface.
52+
53+
Additionnaly, we propose the addition of an auditing tool that can validate the library artifact is safe to use across the Linux-based platforms supported by the Swift project.
54+
Such a tool would ensure that people do not accidentally distribute artifacts that require dependencies that are not met on the various deployment platforms.
55+
56+
## Detailed design
57+
58+
This section describes the changes to artifact bundle manifests in detail, the semantic impact of the changes on SwiftPM's build infrastructure, and describes the operation of the auditing tool.
59+
60+
### Artifact Manifest Semantics
61+
62+
The artifact manifest JSON format for a static library is described below:
63+
64+
```json
65+
{
66+
"schemaVersion": "1.0",
67+
"artifacts": {
68+
"<identifier>": {
69+
"version": "<version number>",
70+
"type": "staticLibrary",
71+
"variants": [
72+
{
73+
"path": "<relative-path-to-library-file>",
74+
"headerPaths": ["<relative-path-to-header-directory-1>, ...],
75+
"moduleMapPath": "<path-to-module-map>",
76+
"supportedTriples": ["<triple1>", ... ],
77+
},
78+
...
79+
]
80+
},
81+
...
82+
}
83+
}
84+
```
85+
86+
The additions are:
87+
88+
* The `staticLibrary` artifact `type` that indicates this binary artifact is not an executable but rather a static library to link against.
89+
* The `headerPaths` field specifies directory paths relative to the root of the artifact bundle that contain the header interfaces to the static library.
90+
These are forwarded along to the swift compiler (or the C compiler) using the usual search path arguments.
91+
Each of these directories can optionally contain a `module.modulemap` file that will be used for importing the API into Swift code.
92+
* The optional `moduleMapPath` field specifies a custom module map to use if the header paths do not contain the module definitions or to provide custom overrides.
93+
94+
As with executable binary artifacts, the `path` field represents the relative path to the binary from the root of the artifact bundle,
95+
and the `supportedTriples` field provides information about the target triples supported by this variant.
96+
97+
An example artifact might look like:
98+
```json
99+
{
100+
"schemaVersion": "1.0",
101+
"artifacts": {
102+
"my-artifact": {
103+
"type": "staticLibrary",
104+
"version": "1.0.0",
105+
"variants": [
106+
{
107+
"path": "artifact.a",
108+
"headerPaths": ["include"],
109+
"supportedTriples": ["aarch64-unknown-linux-gnu"]
110+
}
111+
]
112+
}
113+
}
114+
}
115+
```
116+
117+
### Auditing tool
118+
119+
Without proper auditing it would be very easy to provide binary static library artifacts that call into unresolved external symbols that are not available on the runtime platform, e.g., due to missing linkage to a system dynamic library.
120+
121+
We propose the introduction of a new tool that can validate the "safety" of a binary library artifact across the platforms it supports and the corresponding runtime environment.
122+
123+
In this proposal we restrict ourselves to static libraries that do not have any external dependencies beyond the C standard library and runtime.
124+
To achieve this we need to be able to detect validate this property across the three object file formats used in static libraries on our supported platforms: [Mach-O](https://developer.apple.com/library/archive/documentation/Performance/Conceptual/CodeFootprint/Articles/MachOOverview.html#//apple_ref/doc/uid/20001860-BAJGJEJC) on Apple platforms, [ELF](https://refspecs.linuxfoundation.org/elf/elf.pdf) on Linux-based platforms, and [COFF](https://learn.microsoft.com/en-us/windows/win32/debug/pe-format) on Windows.
125+
All three formats express references to external symbols as _relocations_ which reside in a single section of each object file.
126+
127+
We propose adding the `llvm-objdump` to the toolchain to provide the capability to inspect relocations across all three supported object file formats. The tool would use `llvm-objdump` every object file in the static library and construct a complete list of symbols defined and referenced across the entire library.
128+
Additionally, the tool would construct a simple C compiler invocation to derive to generate a default linker invocation.
129+
This would be used to derive the libraries linked by default in a C program, these libraries would then be scanned to contribute to the list of defined symbols.
130+
The tool would then check that the referenced symbols list is a subset of the set of defined symbols and emit an error otherwise.
131+
132+
This would be sufficient to guarantee that all symbols from the static library would be available at runtime for statically linked executables or for ones running on the build host.
133+
To ensure maximum runtime compatibility we would also provide a Linux-based Docker image that uses the oldest supported `glibc` for a given Swift version.
134+
As `glibc` is backwards compatible, a container running the audit on a given static library would ensure that the version of `glibc` on any runtime platform would be compatible with the binary artifact.
135+
This strategy as been succesfully employed in the Python community with [`manylinux`](https://peps.python.org/pep-0513/).
136+
137+
## Security
138+
139+
This proposal brings the security implications outlined in [SE-0272](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0272-swiftpm-binary-dependencies.md#security) to non-Apple platforms,
140+
namely that a malicious attacker having access to both the server hosting the artifact and the git repository that vends the Package Manifest could provide a malicious library.
141+
Users should exercise caution when onboarding binary dependencies.
142+
143+
## Impact on existing packages
144+
145+
No current package should be affected by this change since this is only an additive change in enabling SwiftPM to use binary target library dependencies on non-Apple platforms.
146+
147+
## Future directions
148+
149+
### Support Swift static libraries
150+
151+
To do this we would extend the static library binary artifact manifest to provide a `.swiftinterface` file that can be consumed by the Swift compiler to import the Swift APIs.
152+
Additionally we would extend the auditing tool to validate the usage of Swift standard library and runtime symbols, e.g., from `libSwiftCore`.
153+
154+
### Extend binary compatibility guarantees
155+
156+
This proposal limits itself to providing facilities for binary compatibility only with the C standard library and runtime.
157+
In the future we could provide a system to allow binary artifact distributors to specify additional linkage dependencies for their binary artifacts.
158+
These would be used to customize the operation of the audit tool and perform automatic linking of them in any client target that depends on the binary artifact, in the same way [CMake](https://cmake.org/cmake/help/v4.0/prop_tgt/INTERFACE_LINK_LIBRARIES.html) propagates link dependencies transitively.
159+
160+
### Add support for dynamically linked dependencies
161+
162+
On Windows dynamic linking requires an _import library_ which is a small static library that contains stubs for symbols exported by the dynamic library.
163+
These stubs are roughly equivalent to a PLT entry in an ELF executable, but are generated during the build of the dynamic library and must be provided to clients of the library for linking purposes.
164+
Similarly on Linux and Apple platforms binary artifact maintainers may wish to provide a dynamic library stub to improve link performance.
165+
To support these use cases the library binary artifact manifest schema could be extended to provide facilities to provide both a link-time and runtime dependency.

0 commit comments

Comments
 (0)