Skip to content
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,6 @@ Target: x86_64-unknown-linux-gnu
## Platform support

- Linux-based platforms listed on https://swift.org/download
- CentOS 7 will not be supported due to some dependencies of swiftly not supporting it, however.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth noting here that 23.10 isn't supported for 5.10 releases? technically that is listed on swift.org/download? Just thinking out loud, it doesn't matter much


Right now, swiftly is in early stages of development and is supported on Linux and macOS. For more detailed information about swiftly's intended features and implementation, check out the [design document](DESIGN.md).

Expand Down
189 changes: 102 additions & 87 deletions Sources/LinuxPlatform/Linux.swift
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,17 @@ var swiftGPGKeysRefreshed = false
/// This implementation can be reused for any supported Linux platform.
/// TODO: replace dummy implementations
public struct Linux: Platform {
let linuxPlatforms = [
PlatformDefinition.ubuntu2404,
PlatformDefinition.ubuntu2204,
PlatformDefinition.ubuntu2004,
PlatformDefinition.ubuntu1804,
PlatformDefinition.fedora39,
PlatformDefinition.rhel9,
PlatformDefinition.amazonlinux2,
PlatformDefinition.debian12,
]

public init() {}

public var appDataDirectory: URL {
Expand Down Expand Up @@ -125,6 +136,26 @@ public struct Linux: Platform {
"tzdata",
"zlib1g-dev",
]
case "ubuntu2404":
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we're missing 2310 from this list

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't find a list of packages for ubuntu 23.10 from the Dockerfiles. If someone knows where the docker file is, I can add it here.

This is where I usually look for them: https://github.com/swiftlang/swift-docker/tree/main/6.0/ubuntu

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I manually pieced together the list by looking at the differences between ubuntu 22.04, and 24.04. Tested it out in docker and it looks to be working fine. There's only the swift 5.10.1 release available for that one though, so I wonder how useful and used this version of ubuntu will be.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

23.10 only exists under https://github.com/swiftlang/swift-docker/blob/main/5.10/ubuntu/23.10/Dockerfile; but 23.10 reached end of life on July 2024.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have removed 23.10 from the list because we can't bring up infrastructure for a platform that is EOL.

[
"binutils",
"git",
"unzip",
"gnupg2",
"libc6-dev",
"libcurl4-openssl-dev",
"libedit2",
"libgcc-13-dev",
"libpython3-dev",
"libsqlite3-0",
"libstdc++-13-dev",
"libxml2-dev",
"libncurses-dev",
"libz3-dev",
"pkg-config",
"tzdata",
"zlib1g-dev",
]
case "amazonlinux2":
[
"binutils",
Expand Down Expand Up @@ -158,6 +189,36 @@ public struct Linux: Platform {
"unzip",
"zip",
]
case "fedora39":
[
"binutils",
"gcc",
"git",
"unzip",
"libcurl-devel",
"libedit-devel",
"libicu-devel",
"sqlite-devel",
"libuuid-devel",
"libxml2-devel",
"python3-devel",
]
case "debian12":
[
"binutils-gold",
"libicu-dev",
"libcurl4-openssl-dev",
"libedit-dev",
"libsqlite3-dev",
"libncurses-dev",
"libpython3-dev",
"libxml2-dev",
"pkg-config",
"uuid-dev",
"tzdata",
"git",
"gcc",
]
default:
[]
}
Expand All @@ -169,10 +230,16 @@ public struct Linux: Platform {
"apt-get"
case "ubuntu2204":
"apt-get"
case "ubuntu2404":
"apt-get"
case "amazonlinux2":
"yum"
case "ubi9":
"yum"
case "fedora39":
"yum"
case "debian12":
"apt-get"
default:
nil
}
Expand All @@ -196,7 +263,7 @@ public struct Linux: Platform {
// Import the latest swift keys, but only once per session, which will help with the performance in tests
if !swiftGPGKeysRefreshed {
let tmpFile = self.getTempFilePath()
FileManager.default.createFile(atPath: tmpFile.path, contents: nil, attributes: [.posixPermissions: 0o600])
let _ = FileManager.default.createFile(atPath: tmpFile.path, contents: nil, attributes: [.posixPermissions: 0o600])
defer {
try? FileManager.default.removeItem(at: tmpFile)
}
Expand Down Expand Up @@ -407,7 +474,7 @@ public struct Linux: Platform {
public func verifySignature(httpClient: SwiftlyHTTPClient, archiveDownloadURL: URL, archive: URL) async throws {
SwiftlyCore.print("Downloading toolchain signature...")
let sigFile = self.getTempFilePath()
FileManager.default.createFile(atPath: sigFile.path, contents: nil)
let _ = FileManager.default.createFile(atPath: sigFile.path, contents: nil)
defer {
try? FileManager.default.removeItem(at: sigFile)
}
Expand All @@ -425,59 +492,43 @@ public struct Linux: Platform {
}
}

private func manualSelectPlatform(_ platformPretty: String?) -> PlatformDefinition {
private func manualSelectPlatform(_ platformPretty: String?) async -> PlatformDefinition {
if let platformPretty = platformPretty {
print("\(platformPretty) is not an officially supported platform, but the toolchains for another platform may still work on it.")
} else {
print("This platform could not be detected, but a toolchain for one of the supported platforms may work on it.")
}

let selections = self.linuxPlatforms.enumerated().map { "\($0 + 1)) \($1.namePretty)" }.joined(separator: "\n")

print("""
Please select the platform to use for toolchain downloads:

0) Cancel
1) Ubuntu 22.04
2) Ubuntu 20.04
3) Ubuntu 18.04
4) RHEL 9
5) Amazon Linux 2
\(selections)
""")

let choice = SwiftlyCore.readLine(prompt: "> ") ?? "0"
let choice = SwiftlyCore.readLine(prompt: "Pick one of the available selections [0-\(self.linuxPlatforms.count)] ") ?? "0"

switch choice {
case "1":
return PlatformDefinition.ubuntu2204
case "2":
return PlatformDefinition.ubuntu2004
case "3":
return PlatformDefinition.ubuntu1804
case "4":
return PlatformDefinition.rhel9
case "5":
return PlatformDefinition.amazonlinux2
default:
guard let choiceNum = Int(choice) else {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think right now if you were to run this and enter 0 for cancel, it would explode, it would go on to run
return self.linuxPlatforms[choiceNum - 1] and you'd get an index exception

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that this guard will guard against non-integer input, such as "foo", or "bar". The guard below should check for 0, or something larger than the linuxPlatforms count and cancel the installation. The final array subscript at the return statement should be safely in range.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh I see, the guard clause tripped me up, Makes sense

fatalError("Installation canceled")
}

guard choiceNum > 0 && choiceNum <= self.linuxPlatforms.count else {
fatalError("Installation canceled")
}

return self.linuxPlatforms[choiceNum - 1]
}

public func detectPlatform(disableConfirmation: Bool, platform: String?) async throws -> PlatformDefinition {
// We've been given a hint to use
if let platform = platform {
switch platform {
case "ubuntu22.04":
return PlatformDefinition.ubuntu2204
case "ubuntu20.04":
return PlatformDefinition.ubuntu2004
case "ubuntu18.04":
return PlatformDefinition.ubuntu1804
case "amazonlinux2":
return PlatformDefinition.amazonlinux2
case "rhel9":
return PlatformDefinition.rhel9
default:
fatalError("Unrecognized platform \(platform)")
if let platform {
guard let pd = linuxPlatforms.first(where: { $0.nameFull == platform }) else {
fatalError("Unrecognized platform \(platform). Recognized values: \(self.linuxPlatforms.map(\.nameFull).joined(separator: ", ")).")
}

return pd
}

let osReleaseFiles = ["/etc/os-release", "/usr/lib/os-release"]
Expand All @@ -498,98 +549,62 @@ public struct Linux: Platform {
} else {
print(message)
}
return self.manualSelectPlatform(platformPretty)
}

let data = FileManager.default.contents(atPath: releaseFile)
guard let data = data else {
let message = "Unable to read OS release information from file \(releaseFile)"
if disableConfirmation {
throw Error(message: message)
} else {
print(message)
}
return self.manualSelectPlatform(platformPretty)
return await self.manualSelectPlatform(platformPretty)
}

guard let releaseInfo = String(data: data, encoding: .utf8) else {
let message = "Unable to read OS release information from file \(releaseFile)"
if disableConfirmation {
throw Error(message: message)
} else {
print(message)
}
return self.manualSelectPlatform(platformPretty)
}
let releaseInfo = try String(contentsOfFile: releaseFile, encoding: .utf8)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we still wrap this in an equivalent try/catch which calls to self.manualSelectPlatform if the info can't be read correctly?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that we want to heavily emphasize the platform auto-detection because it will make the user's life a ton easier. If there's some kind of problem reading the /etc/os-release or /usr/lib/os-release to the point where we can't even read the bytes into a String with the expected UTF-8 encoding the user should probably be made aware of this corruption.

From the linux manual it says that "[a]ll strings should be in UTF-8 encoding, and non-printable characters should not be used."

https://man7.org/linux/man-pages/man5/os-release.5.html


var id: String?
var idlike: String?
var versionID: String?
var ubuntuCodeName: String?
for info in releaseInfo.split(separator: "\n").map(String.init) {
if info.hasPrefix("ID=") {
id = String(info.dropFirst("ID=".count)).replacingOccurrences(of: "\"", with: "")
} else if info.hasPrefix("ID_LIKE=") {
idlike = String(info.dropFirst("ID_LIKE=".count)).replacingOccurrences(of: "\"", with: "")
} else if info.hasPrefix("VERSION_ID=") {
versionID = String(info.dropFirst("VERSION_ID=".count)).replacingOccurrences(of: "\"", with: "")
} else if info.hasPrefix("UBUNTU_CODENAME=") {
ubuntuCodeName = String(info.dropFirst("UBUNTU_CODENAME=".count)).replacingOccurrences(of: "\"", with: "")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we still need to account for stings which start with "UBUNTU_CODENAME" here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're reading this file line-by-line and we don't have any need for the ubuntu codename anymore. I think it's safe to just skip ubuntu codename lines.

versionID = String(info.dropFirst("VERSION_ID=".count)).replacingOccurrences(of: "\"", with: "").replacingOccurrences(of: ".", with: "")
} else if info.hasPrefix("PRETTY_NAME=") {
platformPretty = String(info.dropFirst("PRETTY_NAME=".count)).replacingOccurrences(of: "\"", with: "")
}
}

guard let id = id, let idlike = idlike else {
guard let id, let versionID else {
let message = "Unable to find release information from file \(releaseFile)"
if disableConfirmation {
throw Error(message: message)
} else {
print(message)
}
return self.manualSelectPlatform(platformPretty)
return await self.manualSelectPlatform(platformPretty)
}

if (id + idlike).contains("amzn") {
guard let versionID = versionID, versionID == "2" else {
if (id + (idlike ?? "")).contains("amzn") {
guard versionID == "2" else {
let message = "Unsupported version of Amazon Linux"
if disableConfirmation {
throw Error(message: message)
} else {
print(message)
}
return self.manualSelectPlatform(platformPretty)
return await self.manualSelectPlatform(platformPretty)
}

return PlatformDefinition(name: "amazonlinux2", nameFull: "amazonlinux2", namePretty: "Amazon Linux 2")
} else if (id + idlike).contains("ubuntu") {
if ubuntuCodeName == "jammy" {
return PlatformDefinition(name: "ubuntu2204", nameFull: "ubuntu22.04", namePretty: "Ubuntu 22.04")
} else if ubuntuCodeName == "focal" {
return PlatformDefinition(name: "ubuntu2004", nameFull: "ubuntu20.04", namePretty: "Ubuntu 20.04")
} else if ubuntuCodeName == "bionic" {
return PlatformDefinition(name: "ubuntu1804", nameFull: "ubuntu18.04", namePretty: "Ubuntu 18.04")
} else {
let message = "Unsupported version of Ubuntu Linux"
if disableConfirmation {
throw Error(message: message)
} else {
print(message)
}
return self.manualSelectPlatform(platformPretty)
}
} else if (id + idlike).contains("rhel") {
guard let versionID = versionID, versionID.hasPrefix("9") else {
return PlatformDefinition.amazonlinux2
} else if (id + (idlike ?? "")).contains("rhel") {
guard versionID.hasPrefix("9") else {
let message = "Unsupported version of RHEL"
if disableConfirmation {
throw Error(message: message)
} else {
print(message)
}
return self.manualSelectPlatform(platformPretty)
return await self.manualSelectPlatform(platformPretty)
}

return PlatformDefinition(name: "ubi9", nameFull: "ubi9", namePretty: "RHEL 9")
return PlatformDefinition.rhel9
} else if let pd = [PlatformDefinition.ubuntu1804, .ubuntu2004, .ubuntu2204, .ubuntu2404, .debian12, .fedora39].first(where: { $0.name == id + versionID }) {
return pd
}

let message = "Unsupported Linux platform"
Expand All @@ -598,7 +613,7 @@ public struct Linux: Platform {
} else {
print(message)
}
return self.manualSelectPlatform(platformPretty)
return await self.manualSelectPlatform(platformPretty)
}

public func getShell() async throws -> String {
Expand Down
2 changes: 0 additions & 2 deletions Sources/SwiftlyCore/HTTPClient.swift
Original file line number Diff line number Diff line change
Expand Up @@ -72,8 +72,6 @@ struct SwiftOrgPlatform: Codable {
PlatformDefinition.ubuntu2204
case "Red Hat Universal Base Image 9":
PlatformDefinition.rhel9
case "Ubuntu 23.10":
PlatformDefinition(name: "ubuntu2310", nameFull: "ubuntu23.10", namePretty: "Ubuntu 23.10")
case "Ubuntu 24.04":
PlatformDefinition(name: "ubuntu2404", nameFull: "ubuntu24.04", namePretty: "Ubuntu 24.04")
case "Debian 12":
Expand Down
4 changes: 4 additions & 0 deletions Sources/SwiftlyCore/Platform.swift
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,15 @@ public struct PlatformDefinition: Codable, Equatable {
}

public static let macOS = PlatformDefinition(name: "xcode", nameFull: "osx", namePretty: "macOS")

public static let ubuntu2404 = PlatformDefinition(name: "ubuntu2404", nameFull: "ubuntu24.04", namePretty: "Ubuntu 24.04")
public static let ubuntu2204 = PlatformDefinition(name: "ubuntu2204", nameFull: "ubuntu22.04", namePretty: "Ubuntu 22.04")
public static let ubuntu2004 = PlatformDefinition(name: "ubuntu2004", nameFull: "ubuntu20.04", namePretty: "Ubuntu 20.04")
public static let ubuntu1804 = PlatformDefinition(name: "ubuntu1804", nameFull: "ubuntu18.04", namePretty: "Ubuntu 18.04")
public static let rhel9 = PlatformDefinition(name: "ubi9", nameFull: "ubi9", namePretty: "RHEL 9")
public static let fedora39 = PlatformDefinition(name: "fedora39", nameFull: "fedora39", namePretty: "Fedora Linux 39")
public static let amazonlinux2 = PlatformDefinition(name: "amazonlinux2", nameFull: "amazonlinux2", namePretty: "Amazon Linux 2")
public static let debian12 = PlatformDefinition(name: "debian12", nameFull: "debian12", namePretty: "Debian GNU/Linux 12")
}

public protocol Platform {
Expand Down
19 changes: 15 additions & 4 deletions Tests/SwiftlyTests/HTTPClientTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -48,11 +48,20 @@ final class HTTPClientTests: SwiftlyTests {
func testGetMetdataFromSwiftOrg() async throws {
let supportedPlatforms = [
PlatformDefinition.macOS,
PlatformDefinition.ubuntu2404,
PlatformDefinition.ubuntu2204,
PlatformDefinition.ubuntu2004,
// PlatformDefinition.ubuntu1804, // There are no releases for Ubuntu 18.04 in the branches being tested below
PlatformDefinition.rhel9,
PlatformDefinition.fedora39,
PlatformDefinition.amazonlinux2,
PlatformDefinition.debian12,
]

let newPlatforms = [
PlatformDefinition.ubuntu2404,
PlatformDefinition.fedora39,
PlatformDefinition.debian12,
]

let branches = [
Expand All @@ -65,15 +74,17 @@ final class HTTPClientTests: SwiftlyTests {
// GIVEN: we have a swiftly http client with swift.org metadata capability
// WHEN: we ask for the first five releases of a supported platform in a supported arch
let releases = try await SwiftlyCore.httpClient.getReleaseToolchains(platform: platform, arch: arch, limit: 5)
// THEN: we get five releases
XCTAssertEqual(5, releases.count)
// THEN: we get at least 1 release
XCTAssertTrue(1 <= releases.count)

if newPlatforms.contains(platform) { continue } // Newer distros don't have main snapshots yet
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How will we eventually know to remove this for proper testing? Seems like the kind of thing I'd forget 😄

Copy link
Member Author

@cmcgee1024 cmcgee1024 Oct 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Integration tests that are in the middle of the test pyramid like this are tricky, but necessary.

The trouble with mocking the responses is that we won't know if the API might change in some way that breaks swiftly. I think that we want to catch obvious integration problems between the swift.org API and what we can potentially request from it. If that means that we don't fully cover all possibilities with all supported platforms I think that's ok. This was a good test when I started working on the patch to verify that the platform definition strings were correct from the standpoint of the swift.org API.


for branch in branches {
// GIVEN: we have a swiftly http client with swift.org metadata capability
// WHEN: we ask for the first five snapshots on a branch for a supported platform and arch
let snapshots = try await SwiftlyCore.httpClient.getSnapshotToolchains(platform: platform, arch: arch, branch: branch, limit: 5)
// THEN: we get five snapshots
XCTAssertEqual(5, snapshots.count)
// THEN: we get at least 3 releases
XCTAssertTrue(3 <= snapshots.count)
}
}
}
Expand Down
Loading