-
Notifications
You must be signed in to change notification settings - Fork 55
Swiftly proxies #155
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Swiftly proxies #155
Changes from 27 commits
e4efbeb
3b5c404
956256f
375df29
ed68a9e
c745f4c
560236d
ae29e88
ce0d27a
66dd459
2df7357
8976bba
3caab66
31f4327
16caf80
b2aa165
7504dd3
0e7b661
43d620a
2e3c59d
9dd640c
5e615ea
10a0856
6d6050e
6989796
52d081f
2f6d701
cb0e923
bb36de0
7269128
35cb5c5
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -4,21 +4,28 @@ This document contains the high level design of swiftly. Not all features have b | |
|
||
## Index | ||
|
||
- [Swiftly's purpose](#swiftlys-purpose) | ||
- [Installation of swiftly](#installation-of-switly) | ||
- [Linux](#linux) | ||
- [Installation of swiftly](#installation-of-swiftly) | ||
- [Installation of a Swift toolchain](#installation-of-a-swift-toolchain) | ||
- [macOS](#macos) | ||
- [Installation of swiftly](#installation-of-swiftly-1) | ||
- [Installation of a Swift toolchain](#installation-of-a-swift-toolchain-1) | ||
- [Interface](#interface) | ||
- [Toolchain names and versions](#toolchain-names-and-versions) | ||
- [Commands](#commands) | ||
- [Toolchain selection](#toolchain-selection) | ||
- [Detailed design](#detailed-design) | ||
- [Implementation sketch - Core](#implementation-sketch---core) | ||
- [Implementation sketch - Ubuntu 20.04](#implementation-sketch---ubuntu-2004) | ||
- [Implementation sketch - macOS](#implementation-sketch---macos) | ||
- [`config.json` schema](#configjson-schema) | ||
|
||
## Swiftly's purpose | ||
|
||
Swiftly helps you to easily install different Swift toolchains locally on your account. It also provides a single path where you can run the tools in the currently selected toolchain. Toolchain selection is [configurable](#toolchain-selection) using different mechanisms. | ||
|
||
Note that swiftly is *not* a virtual toolchain in itself since there are cases where it cannot behave as a self-contained Swift toolchain. For example, there can be external dependencies on specific files, such as headers or libraries. There are far too many files that change between toolchain versions to be managed by swiftly. Also, for long-lived processes, there is no way to gracefully restart them without help from the client. | ||
|
||
## Installation of swiftly | ||
|
||
The installation of swiftly is divided into two phases: delivery and initialization. Delivery of the swiftly binary can be accomplished using different methods: | ||
|
@@ -60,7 +67,7 @@ A simple setup for managing the toolchains could look like this: | |
|
||
The toolchains (i.e. the contents of a given Swift download tarball) would be contained in the toolchains directory, each named according to the major/minor/patch version. `config.json` would contain any required metadata (e.g. the latest Swift version, which toolchain is selected, etc.). If pulling in Foundation to use `JSONEncoder`/`JSONDecoder` (or some other JSON tool) would be a problem, we could also use something simpler. | ||
|
||
The `~/.local/bin` directory would include symlinks pointing to the `bin` directory of the "active" toolchain, if any. | ||
The `~/.local/bin` directory would include symlinks pointing to swiftly itself. When the proxies binaries are executed swiftly proxies them to the requested toolchain, or the default. | ||
|
||
This is all very similar to how rustup does things, but I figure there's no need to reinvent the wheel here. | ||
|
||
|
@@ -78,7 +85,7 @@ The contents of `~/Library/Application Support/swiftly` would look like this: | |
– env | ||
``` | ||
|
||
Instead of downloading tarballs containing the toolchains and storing them directly in `~/.local/share/swiftly/toolchains`, we instead install Swift toolchains to `~/Library/Developer/Toolchains` via the `.pkg` files provided for download at swift.org. To select a toolchain for use, we update the symlinks at `~/Library/Application Support/swiftly/bin` to point to the desired toolchain in `~/Library/Developer/Toolchains`. In the env file, we’ll contain a line that looks like `export PATH="$HOME/Library/Application Support/swiftly:$PATH"`, so the version of swift being used will automatically always be from the active toolchain. `config.json` will contain version information about the selected toolchain as well as its actual location on disk. | ||
Instead of downloading tarballs containing the toolchains and storing them directly in `~/.local/share/swiftly/toolchains`, we instead install Swift toolchains to `~/Library/Developer/Toolchains` via the `.pkg` files provided for download at swift.org. In the env file, we’ll add a line that looks like `export PATH="$HOME/Library/Application Support/swiftly:$PATH"`, so that swiftly can proxy toolchain commands to the requested toolchain, or default. `config.json` will contain version information about the selected toolchain as well as its actual location on disk. | ||
|
||
This scheme works for ensuring the version of Swift used on the command line can be controlled, but it doesn’t affect the active toolchain used by Xcode, which uses its own mechanisms for that. Xcode, if it is installed, can find the toolchains installed by swiftly. | ||
|
||
|
@@ -138,6 +145,14 @@ Installing a specific snapshot from a swift version development branch | |
|
||
`swiftly install 5.5-snapshot-2022-1-28` | ||
|
||
##### Installing the version from the `.swift-version` file | ||
|
||
A package could have a ".swift-version" file that specifies the recommended toolchain version. A swiftly install with no version will search for a version file and install that version. | ||
|
||
`swiftly install` | ||
|
||
If no ".swift-version" file can be found then the installation fails indicating that it couldn't fine the file. | ||
|
||
#### uninstall | ||
|
||
Uninstalling versions of Swift should be in a similar form to install. Uninstalling a toolchain that is currently “in use” (see the “use” command section below) will cause swiftly to use the latest Swift release toolchain that is installed. If none are, the latest snapshot will be used. If no snapshots are installed either, then a message will be printed indicating that all Swift versions are uninstalled. | ||
|
@@ -178,7 +193,7 @@ To list all the versions of swift installed on your system | |
|
||
#### use | ||
|
||
“Using” a toolchain sets it as the active toolchain, meaning it will be the one found via $PATH and invoked via `swift` commands executed in the shell. Only a single toolchain can be used at a given time. Using a toolchain doesn’t uninstall anything; it only updates symlinks so that the requested toolchain can be found by the shell. | ||
“Using” a toolchain sets it as the default toolchain, meaning it will be the default one that is used when running toolchain commands from the shell. Only a single toolchain can be the default at a given time and location. Using a toolchain doesn’t uninstall anything; it only updates the configuration. | ||
|
||
To use the toolchain associated with the most up-to-date Swift version, the “latest” version can be specified: | ||
|
||
|
@@ -208,6 +223,10 @@ To use the latest installed main snapshot, leave off the date: | |
|
||
`swiftly use main-snapshot` | ||
|
||
The use subcommand also supports `.swift-version` files. If a ".swift-version" file is present in the current working directory, or an ancestory directory, then swiftly will update that file with the new version to use. This can be a useful feature for a team to share and align on toolchain versions with git. As a special case, if swiftly could not find a version file, but it could find a Package.swift file it will create a new version file for you in the package and set that to the requested toolchain version. | ||
|
||
Note: The `.swift-version` file mechanisms can be overridden using the `--global-default` flag so that your swiftly installation's default toolchain can be set explicitly. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would rather the default is the other way. ie There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think that there are advantage of having this local as the default. It encourages the sharing of the toolchain revision that the team is working with through discovery of this feature of swiftly. Also, this helps to tie into one of the values of using swiftly itself, which is how easily and flexibly you can switch toolchains. In terms of precedence, git defaults to local over global for its configuration. I'm curious what you think are the pitfalls of this approach. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah I realised as I was writing this, there is previous art that defaults to local. It is more a preference on my behalf and how I'd use swiftly There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
|
||
#### update | ||
|
||
Update replaces a given toolchain with a later version of that toolchain. For a stable release, this means updating to a later patch version. For snapshots, this means updating to the most recently available snapshot. | ||
|
@@ -266,6 +285,58 @@ This command checks to see if there are new versions of `swiftly` itself and upg | |
|
||
`swiftly self-update` | ||
|
||
### Toolchain selection | ||
|
||
Swiftly will create a set of symbolic links in its SWIFTLY_BIN_DIR during installation that point to the swiftly binary itself for each of the common toolchain commands, such as swift, swiftc, clang, etc. This mechanism will allows swiftly to proxy those command invocations to a selected toolchain at the time of invocation. A toolchain can be selected in these ways in order of precedence: | ||
|
||
* The presence of a .swift-version file in the current working directory, or ancestor directory, with the required toolchain version | ||
* The swiftly default (in-use) toolchain set in the swftly config.json by `swiftly install` or `swiftly use` commands | ||
|
||
If swiftly cannot find an installed toolchain that matches the selection then it fails with an error and instructions how to use `swiftly install` to satisfy the selection next time. | ||
|
||
#### Resolve selected toolchain | ||
|
||
For cases where the physical toolchain must be located, such as references specific header files, or shared libraries that are not proxied by swiftly there is a method to resolve the currently selected toolchain to its physical location using `swiftly use`. | ||
|
||
``` | ||
swiftly use --print-location | ||
``` | ||
|
||
This command will provide the full path to the directory where the selected toolchain is installed to standard output if such a toolchain exists. An external tool can directly navigate to the resources that it requires. For external tools that manage long-lived processes from the toolchain, such as the language server, and lldb, this command can be used in a poll to detect cases where the processes should be restarted. | ||
|
||
#### Run with a selected toolchain | ||
|
||
There are cases where you might want to run an arbitrary command using a selected toolchain. An example could be that you want to build something with CMake. | ||
|
||
``` | ||
# CMake | ||
swiftly run cmake -G ninja | ||
swiftly run ninja build | ||
|
||
# Autoconf | ||
swiftly run ./configure | ||
swiftly run make | ||
``` | ||
|
||
Swiftly adjusts certain environment variables, such as prefixing the PATH to the selected toolchain directory, and setting the CC and CXX variables to the locations of clang and clang++ in those toolchains so that the build tools use them. If you want to explicitly specify a toolchain for the command you can do that with a selector notation like this: | ||
|
||
|
||
``` | ||
swiftly run swift build +5.10.1 # Runs swift build with the 5.10.1 toolchain | ||
``` | ||
|
||
A few notes about the '+' prefix. First, if a literal '+' prefix should be sent directly to the tool as an argument then it is escaped by doubling it with '++'. An argument with only '++' is ignored entirely, and any additional arguments are sent directly to the command without any further inspection of their prefixes. This is analogous to the special '--' token that certain argument parsers accept so that they don't interpret anything following that token as command flags or options. | ||
|
||
If the selected toolchain is not installed then swiftly will exit with a message indicating that you need to run `swiftly install x.y.z` to install it. | ||
|
||
``` | ||
# Use the latest main snapshot toolchain and run 'swift build' to build the package with it. | ||
swiftly run swift build +main-snapshot | ||
|
||
# Generate makefiles with the latest released Swift toolchain | ||
swiftly run +latest cmake -G "Unix Makefile" | ||
swiftly run +latest make | ||
``` | ||
|
||
## Detailed Design | ||
|
||
Swiftly itself will be a SPM project consisting of several executable products, one per supported platform, and all of these will share the core module that handles argument parsing, printing help information, and dispatching commands. Each platform’s executable will be built to statically link the stdlib so that they can be run without having installed Swift first. | ||
|
@@ -457,15 +528,7 @@ https://download.swift.org/swift-5.5.1-release/ubuntu1604/swift-5.5.1-RELEASE/sw | |
$ tar -xf <URL> --directory ~/.local/share/swiftly/toolchains | ||
``` | ||
|
||
It also updates `config.json` to include this toolchain as the latest for the provided version. If installing a new patch release toolchain, the now-outdated one can be deleted (e.g. `5.5.0` can be deleted when `5.5.1` is installed). | ||
|
||
Finally, the use implementation executes the following to update the link: | ||
|
||
``` | ||
$ ln -s ~/.local/share/swiftly/toolchains/<toolchain>/usr/bin/swift ~/.local/bin/swift | ||
``` | ||
|
||
It also updates `config.json` to include this version as the currently selected one. | ||
It also updates `config.json` to include this toolchain as the latest for the provided version. If installing a new patch release toolchain, the now-outdated one can be deleted (e.g. `5.5.0` can be deleted when `5.5.1` is installed). The `config.json` is updated to include this version as the currently selected (default) one. | ||
|
||
### Implementation Sketch - macOS | ||
|
||
|
@@ -481,18 +544,13 @@ https://download.swift.org/swift-<version>-RELEASE/xcode/swift-<version>-RELEASE | |
|
||
`config.json` is then updated to include this toolchain as the latest for the provided version. | ||
|
||
Finally, the use implementation executes the following to update the link: | ||
|
||
``` | ||
$ ln -s ~/Library/Developer/Toolchains/<toolchain name> ~/.swiftly/active-toolchain | ||
``` | ||
|
||
It also updates `config.json` to include this version as the currently selected one. | ||
It also updates `config.json` to include this version as the currently selected (default) one. | ||
|
||
### `config.json` Schema | ||
|
||
``` | ||
{ | ||
"version": "<version of swiftly that created/updated this config.json file>", | ||
"platform": { | ||
"namePretty": <OS name pretty printed>, | ||
"fullName": <OS name used in toolchain file name>, | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO we ought to be using
~/.swiftly/bin
for this. Swiftly doesn't own that directory and while it's fine to install theswiftly
command itself there (assuming it's being installed per-user rather than system-wide), I think it's very bad form installing "proxies" in there.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, I've changed this to put the proxies and swiftly itself into its own managed directory by default, which is
~/.local/share/swiftly/bin
. This matches what we do on macOS, except there we use a more typical macOS location~/Library/Application Support/swiftly/bin
.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
~/.local/bin
was originally chosen according to the XDG Base Directories specification, which is something of a standard for directories like this. While swiftly doesn't own~/.local/bin
, it is the designated place to drop per-user executables according to that spec. Why would it be bad form to put our symlinks there?https://specifications.freedesktop.org/basedir-spec/latest/
https://wiki.archlinux.org/title/XDG_Base_Directory
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The concern is that the user might have swift toolchain, and other items (clang, lld, lldb, etc.) existing in the ~/.local/bin directory and swiftly will attempt to manage those.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are really two reasons for this; the first is that, as @cmcgee1024 says, you might actually have Swift installed in
~/.local/bin
, in which case the proxies would then overwrite your Swift executables, which would be wrong. The second is that when using tools likeswiftly
, there are situations where you want to bypass the proxies, and it's easiest to do that if they're in a separatebin
directory all of their own (since whether you're using the proxies or not is dependent on whether thatbin
directory appears in yourPATH
).There is precedent here;
pyenv
,rbenv
et al also put their shims (which we seem to be calling proxies) into a separatebin
directory for the same reasons.Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think
swiftly
itself belongs in~/.local/bin
or/usr/local/bin
or wherever the user installed it. It's just the proxies that should live elsewhere IMO. // @cmcgee1024There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@al45tair swiftly is aware of two locations at the moment. There is the swiftly home dir, where its configuration lives, and the swiftly bin dir where all of the binaries that it manages live. I'm hesitant to add another location to the list.
In the Linux case
~/.local/bin
doesn't tend to be on the user's path, so swiftly finds itself adding it as part of augmenting the user's profile, being mindful of collisions with other packages that might attempt to do the same, or even the user themselves. There's no similar common CLT bin directory at a user level for macOS, so the swiftly bin directory is~/Application Support/swiftly/bin
as the default. There's a far less likelihood of a path collision where something else that adds swiftly's managed bin directory. I think it makes sense to keep all of the binaries that swiftly manages (potentially itself too), separate from a shared binary location on all platforms as the default. The user can customize the paths if they want in the installer to change this, but then they are taking on more responsibility for the possibility of collisions.Swiftly itself can be installed by a system package manager where it will probably be installed in one of the usual locations like
/usr/bin
or/usr/local/bin
. The latter is where swiftly is installed using the macOS pkg file. The package manager manages these paths and the user's profile likely includes them in their path without extra handling by swiftly. In this case the swiftly bin directory is just the proxies that swiftly manages since it can't manage its own binary.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't have a strong objection to that, I suppose, but it does mean that you can't use
swiftly
without having the proxies in your path if you've letswiftly
install itself. At least, unless you make a symlink from~/.local/bin
or similar.