Merge branch 'main' of github.com:cmcgee1024/swiftly into add_new_linux_platforms

Author: cmcgee1024

Documentation/SwiftlyDocs.docc/swiftly-cli-reference.md

@@ -23,7 +23,7 @@ swiftly [--version] [--help]
 Install a new toolchain.
 
 ```
-swiftly install <version> [--use] [--token=<token>] [--verify] [--post-install-file=<post-install-file>] [--version] [--help]
+swiftly install <version> [--use] [--verify] [--post-install-file=<post-install-file>] [--version] [--help]
 ```
 
 **version:**
@@ -59,14 +59,6 @@ Likewise, the latest snapshot associated with a given development branch can be
 *Mark the newly installed toolchain as in-use.*
 
 
-**--token=\<token\>:**
-
-*A GitHub authentication token to use for any GitHub API requests.*
-
-This is useful to avoid GitHub's low rate limits. If an installation
-fails with an "unauthorized" status code, it likely means the rate limit has been hit.
-
-
 **--verify:**
 
 *Verify the toolchain's PGP signature before proceeding with installation.*
@@ -120,9 +112,9 @@ Likewise, the available toolchains associated with a given minor version can be
 The installed snapshots for a given devlopment branch can be listed by specifying the branch as the selector:
 
     $ swiftly list-available main-snapshot
-    $ swiftly list-available 6.0-snapshot
+    $ swiftly list-available x.y-snapshot
 
-Note that listing available snapshots before 6.0 is unsupported.
+Note that listing available snapshots before the latest release (major and minor number) is unsupported.
 
 
 **--version:**

Package.swift

@@ -2,9 +2,6 @@
 
 import PackageDescription
 
-let ghApiCacheResources = (1...16).map { Resource.embedInCode("gh-api-cache/swift-tags-page\($0).json") }
-let ghApiCacheExcludedResources = (17...27).map { "gh-api-cache/swift-tags-page\($0).json" }
-
 let package = Package(
     name: "swiftly",
     platforms: [
@@ -100,8 +97,7 @@ let package = Package(
         .testTarget(
             name: "SwiftlyTests",
             dependencies: ["Swiftly"],
-            exclude: ghApiCacheExcludedResources,
-            resources: ghApiCacheResources + [
+            resources: [
                 .embedInCode("mock-signing-key-private.pgp"),
             ]
         ),

README.md

@@ -272,13 +272,6 @@ swift-5.7-DEVELOPMENT-SNAPSHOT-2022-08-30-a
 
 The canonical name format was chosen to reduce the keystrokes needed to refer to a snapshot toolchain, but the longer form is also useful when copy/pasting a toolchain name provided from somewhere else.
 
-### Specifying a GitHub access token
-
-swiftly currently uses the GitHub API to look up the available Swift toolchains. To avoid running up against rate limits, you can provide a [GitHub access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) via the `--token` option (the token doesn't need any permissions):
-
-```
-$ swiftly install latest --token <GitHub authentication token>
-```
 ## Contributing 
 Welcome to the Swift community!
 
@@ -298,7 +291,7 @@ Swift.org currently provides experimental [`.rpm` and `.deb`](https://forums.swi
 
 swiftenv is an existing Swift version manager which already has much of the functionality that swiftly will eventually have. It's an awesome tool, and if it's part of your workflow then we encourage you to keep using it! That said, swiftly is/will be different a few ways:
 
-- swiftly is being built as a community driven effort led by the Swift server workgroup, and through this collaboration, swiftly will eventually become an official installation tool for Swift toolchains. As first step towards that, swiftly will help inform the creation of API endpoints maintained by the Swift project that it will use to retrieve information about what toolchains are available to install and to verify their expected signatures. swiftenv currently uses a third party API layer for this. Using an official API reduces the avenues for security vulnerabilities and also reduces the risk of downtime affecting Swift installations. Note that this is planned for the future--swiftly currently uses the GitHub API for this purpose. 
+- swiftly is being built as a community driven effort led by the Swift server workgroup, and through this collaboration, swiftly will eventually become an official installation tool for Swift toolchains. As first step towards that, swiftly will help inform the creation of API endpoints maintained by the Swift project that it will use to retrieve information about what toolchains are available to install and to verify their expected signatures. swiftenv currently uses a third party API layer for this. Using an official API reduces the avenues for security vulnerabilities and also reduces the risk of downtime affecting Swift installations.
 
 - swiftly will be written in Swift, which we think is important for maintainability and encouraging community contributions. 
 

RELEASING.md

@@ -13,7 +13,7 @@ Swift has a tool for producing final product packages, suitable for distribution
 5. Create a tag on that commit with the format "x.y.z". Do not omit "z", even if its value is 0.
 
 6. Build the executables for the release by running `swift run build-swiftly-release <version>` from the root of the swiftly repository
-  * Build on a Apple silicon macOS machine to produce a universal package for x86_64 and arm64
+  * Build on an Apple macOS machine to produce a universal package for x86_64 and arm64 (add the --cert option to provide a signing certificate file for the .pkg)
   * Build on an Amazon Linux 2 image for x86_64
   * Build on an Amazon Linux 2 image for arm64
 

Sources/LinuxPlatform/Linux.swift

@@ -247,23 +247,23 @@ public struct Linux: Platform {
 
         let manager: String? = switch platformName {
         case "ubuntu1804":
-            "apt"
+            "apt-get"
         case "ubuntu2004":
-            "apt"
+            "apt-get"
         case "ubuntu2204":
-            "apt"
+            "apt-get"
         case "ubuntu2310":
-            "apt"
+            "apt-get"
         case "ubuntu2404":
-            "apt"
+            "apt-get"
         case "amazonlinux2":
             "yum"
         case "ubi9":
             "yum"
         case "fedora39":
             "yum"
         case "debian12":
-            "apt"
+            "apt-get"
         default:
             nil
         }
@@ -307,21 +307,38 @@ public struct Linux: Platform {
             return nil
         }
 
-        let missingPackages = packages.filter { !self.isSystemPackageInstalled(manager, $0) }.joined(separator: " ")
+        var missingPackages: [String] = []
+
+        for pkg in packages {
+            if case let pkgInstalled = await self.isSystemPackageInstalled(manager, pkg), !pkgInstalled {
+                missingPackages.append(pkg)
+            }
+        }
 
         guard !missingPackages.isEmpty else {
             return nil
         }
 
-        return "\(manager) -y install \(missingPackages)"
+        return "\(manager) -y install \(missingPackages.joined(separator: " "))"
     }
 
-    public func isSystemPackageInstalled(_ manager: String?, _ package: String) -> Bool {
+    public func isSystemPackageInstalled(_ manager: String?, _ package: String) async -> Bool {
         do {
             switch manager {
-            case "apt":
-                try self.runProgram("dpkg", "-l", package, quiet: true)
-                return true
+            case "apt-get":
+                if let pkgList = try await self.runProgramOutput("dpkg", "-l", package) {
+                    // The package might be listed but not in an installed non-error state.
+                    //
+                    // Look for something like this:
+                    //
+                    //   Desired=Unknown/Install/Remove/Purge/Hold
+                    //   | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
+                    //   |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
+                    //   ||/
+                    //   ii  pkgfoo         1.0.0ubuntu12        My description goes here....
+                    return pkgList.contains("\nii ")
+                }
+                return false
             case "yum":
                 try self.runProgram("yum", "list", "installed", package, quiet: true)
                 return true

Sources/Swiftly/Install.swift

@@ -46,15 +46,6 @@ struct Install: SwiftlyCommand {
     @Flag(name: .shortAndLong, help: "Mark the newly installed toolchain as in-use.")
     var use: Bool = false
 
-    @Option(help: ArgumentHelp(
-        "A GitHub authentication token to use for any GitHub API requests.",
-        discussion: """
-        This is useful to avoid GitHub's low rate limits. If an installation
-        fails with an \"unauthorized\" status code, it likely means the rate limit has been hit.
-        """
-    ))
-    var token: String?
-
     @Flag(inversion: .prefixedNo, help: "Verify the toolchain's PGP signature before proceeding with installation.")
     var verify = true
 
@@ -68,15 +59,14 @@ struct Install: SwiftlyCommand {
     var postInstallFile: String?
 
     private enum CodingKeys: String, CodingKey {
-        case version, token, use, verify, postInstallFile
+        case version, use, verify, postInstallFile
     }
 
     mutating func run() async throws {
         try validateSwiftly()
 
         let selector = try ToolchainSelector(parsing: self.version)
         var config = try Config.load()
-        SwiftlyCore.httpClient.githubToken = self.token
         let toolchainVersion = try await self.resolve(config: config, selector: selector)
         let postInstallScript = try await Self.execute(
             version: toolchainVersion,
@@ -224,8 +214,7 @@ struct Install: SwiftlyCommand {
         return postInstallScript
     }
 
-    /// Utilize the GitHub API along with the provided selector to select a toolchain for install.
-    /// TODO: update this to use an official swift.org API
+    /// Utilize the swift.org API along with the provided selector to select a toolchain for install.
     func resolve(config: Config, selector: ToolchainSelector) async throws -> ToolchainVersion {
         switch selector {
         case .latest:
@@ -266,17 +255,27 @@ struct Install: SwiftlyCommand {
             }
 
             SwiftlyCore.print("Fetching the latest \(branch) branch snapshot...")
+
             // If a date was not provided, perform a lookup to find the most recent snapshot
             // for the given branch.
-            let snapshot = try await SwiftlyCore.httpClient.getSnapshotToolchains(platform: config.platform, branch: branch, limit: 1) { snapshot in
-                snapshot.branch == branch
-            }.first
+            let snapshots: [ToolchainVersion.Snapshot]
+            do {
+                snapshots = try await SwiftlyCore.httpClient.getSnapshotToolchains(platform: config.platform, branch: branch, limit: 1) { snapshot in
+                    snapshot.branch == branch
+                }
+            } catch let branchNotFoundErr as SwiftlyHTTPClient.SnapshotBranchNotFoundError {
+                throw Error(message: "You have requested to install a snapshot toolchain from branch \(branchNotFoundErr.branch). It cannot be found on swift.org. Note that snapshots are only available from the current `main` release and the latest x.y (major.minor) release. Try again with a different branch.")
+            } catch {
+                throw error
+            }
+
+            let firstSnapshot = snapshots.first
 
-            guard let snapshot else {
+            guard let firstSnapshot else {
                 throw Error(message: "No snapshot toolchain found for branch \(branch)")
             }
 
-            return .snapshot(snapshot)
+            return .snapshot(firstSnapshot)
         }
     }
 }

Sources/Swiftly/ListAvailable.swift

@@ -28,9 +28,9 @@ struct ListAvailable: SwiftlyCommand {
         The installed snapshots for a given devlopment branch can be listed by specifying the branch as the selector:
 
             $ swiftly list-available main-snapshot
-            $ swiftly list-available 6.0-snapshot
+            $ swiftly list-available x.y-snapshot
 
-        Note that listing available snapshots before 6.0 is unsupported.
+        Note that listing available snapshots before the latest release (major and minor number) is unsupported.
         """
     ))
     var toolchainSelector: String?
@@ -51,11 +51,13 @@ struct ListAvailable: SwiftlyCommand {
 
         switch selector {
         case let .snapshot(branch, _):
-            if case let .release(major, _) = branch, major < 6 {
-                throw Error(message: "Listing available snapshots previous to 6.0 is not supported.")
+            do {
+                tc = try await SwiftlyCore.httpClient.getSnapshotToolchains(platform: config.platform, branch: branch).map { ToolchainVersion.snapshot($0) }
+            } catch let branchNotFoundError as SwiftlyHTTPClient.SnapshotBranchNotFoundError {
+                throw Error(message: "The snapshot branch \(branchNotFoundError.branch) was not found on swift.org. Note that snapshot toolchains are only available for the current `main` release and the previous x.y (major.minor) release.")
+            } catch {
+                throw error
             }
-
-            tc = try await SwiftlyCore.httpClient.getSnapshotToolchains(platform: config.platform, branch: branch).map { ToolchainVersion.snapshot($0) }
         case .stable, .latest:
             tc = try await SwiftlyCore.httpClient.getReleaseToolchains(platform: config.platform).map { ToolchainVersion.stable($0) }
         default:

Sources/Swiftly/Update.swift

@@ -180,7 +180,7 @@ struct Update: SwiftlyCommand {
     }
 
     /// Tries to find a toolchain version that meets the provided parameters, if one exists.
-    /// This does not download the toolchain, but it does query the GitHub API to find the suitable toolchain.
+    /// This does not download the toolchain, but it does query the swift.org API to find the suitable toolchain.
     private func lookupNewToolchain(_ config: Config, _ bounds: UpdateParameters) async throws -> ToolchainVersion? {
         switch bounds {
         case let .stable(old, range):
@@ -195,9 +195,18 @@ struct Update: SwiftlyCommand {
                 }
             }.first.map(ToolchainVersion.stable)
         case let .snapshot(old):
-            return try await SwiftlyCore.httpClient.getSnapshotToolchains(platform: config.platform, branch: old.branch, limit: 1) { snapshot in
-                snapshot.branch == old.branch && snapshot.date > old.date
-            }.first.map(ToolchainVersion.snapshot)
+            let newerSnapshotToolchains: [ToolchainVersion.Snapshot]
+            do {
+                newerSnapshotToolchains = try await SwiftlyCore.httpClient.getSnapshotToolchains(platform: config.platform, branch: old.branch, limit: 1) { snapshot in
+                    snapshot.branch == old.branch && snapshot.date > old.date
+                }
+            } catch let branchNotFoundErr as SwiftlyHTTPClient.SnapshotBranchNotFoundError {
+                throw Error(message: "Snapshot branch \(branchNotFoundErr.branch) cannot be updated. One possible reason for this is that there has been a new release published to swift.org and this snapshot is for an older release. Snapshots are only available for the newest release and the main branch. You can install a fresh snapshot toolchain from the either the latest release x.y (major.minor) with `swiftly install x.y-snapshot` or from the main branch with `swiftly install main-snapshot`.")
+            } catch {
+                throw error
+            }
+
+            return newerSnapshotToolchains.first.map(ToolchainVersion.snapshot)
         }
     }
 }