Skip to content

[windows] upgrade to Python 3.10.1 embeddable #83488

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[windows] upgrade to Python 3.10.1 embeddable #83488

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 58 additions & 2 deletions utils/build.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ param
[string] $PinnedSHA256 = "",
[string] $PinnedVersion = "",
[ValidatePattern('^\d+(\.\d+)*$')]
[string] $PythonVersion = "3.9.10",
[string] $PythonVersion = "3.10.1",
[ValidatePattern("^r(?:[1-9]|[1-9][0-9])(?:[a-z])?$")]
[string] $AndroidNDKVersion = "r27c",
[ValidatePattern("^\d+\.\d+\.\d+(?:-\w+)?")]
Expand Down Expand Up @@ -332,6 +332,24 @@ $KnownPythons = @{
URL = "https://www.nuget.org/api/v2/package/pythonarm64/3.9.10";
SHA256 = "429ada77e7f30e4bd8ff22953a1f35f98b2728e84c9b1d006712561785641f69";
};
};
"3.10.1" = @{
AMD64 = @{
URL = "https://www.nuget.org/api/v2/package/python/3.10.1";
SHA256 = "987a0e446d68900f58297bc47dc7a235ee4640a49dace58bc9f573797d3a8b33";
};
AMD64_Embedded = @{
URL = "https://www.python.org/ftp/python/3.10.1/python-3.10.1-embed-amd64.zip";
SHA256 = "502670dcdff0083847abf6a33f30be666594e7e5201cd6fccd4a523b577403de";
};
ARM64 = @{
URL = "https://www.nuget.org/api/v2/package/pythonarm64/3.10.1";
SHA256 = "16becfccedf1269ff0b8695a13c64fac2102a524d66cecf69a8f9229a43b10d3";
};
ARM64_Embedded = @{
URL = "https://www.python.org/ftp/python/3.10.1/python-3.10.1-embed-arm64.zip";
SHA256 = "1f9e215fe4e8f22a8e8fba1859efb1426437044fb3103ce85794630e3b511bc2";
};
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets leave the definitions for 3.9, allowing to optionally upgrade to 3.10 while we iterate to get this working. That will allow you to make progress and get the changes merged earlier rather than trying to keep up to date with the changes in this script.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, thanks

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a particular reason that we need both the regular and the embedded URLs? Could we not get away with just the embedded?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, embeddable python does not come with pip. We could download and run get-pip.py, but I don't think we would gain a lot of CI time in doing so. I have not tried it out.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reasoning for the version selection here is not CI times but rather to match what llvm.org distributes. I think that we don't need to inject pip unless we are building the installer for distribution (i.e. nightlies/releases).

}

Expand Down Expand Up @@ -597,6 +615,10 @@ function Get-PythonPath([Hashtable] $Platform) {
return [IO.Path]::Combine("$BinaryCache\", "Python$($Platform.Architecture.CMakeName)-$PythonVersion")
}

function Get-EmbeddedPythonPath([Hashtable] $Platform) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, is there a bigger picture around the embedded vs non-embedded python that the extraction needs special logic?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only reason is pip missing in embeddedable python. We build with the regular Python, but we bundle the embeddable python in the installer.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There-in lies the trick! We can inject pip into the packaging :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think injecting pip would work well because of this SO comment, especially because the Python documentation explicitly states:

Using pip to manage dependencies as for a regular Python installation is not supported with this distribution [embeddable Python]

Given that, and the extra steps it would take to manually install pip with the embeddable python, I think it's simpler to:

  • Build with the regular Python
  • Test with the regular Python (easier to install dependencies)
  • Package the embeddable Python.

return [IO.Path]::Combine("$BinaryCache\", "EmbeddedPython$($Platform.Architecture.CMakeName)-$PythonVersion")
}

function Get-PythonExecutable {
return [IO.Path]::Combine((Get-PythonPath $BuildPlatform), "tools", "python.exe")
}
Expand All @@ -605,6 +627,10 @@ function Get-PythonScriptsPath {
return [IO.Path]::Combine((Get-PythonPath $BuildPlatform), "tools", "Scripts")
}

function Get-EmbeddedPythonInstallDir() {
return [IO.Path]::Combine("$ImageRoot\", "Program Files", "Swift", "Python-$PythonVersion")
}

function Get-Syft {
return $KnownSyft[$SyftVersion][$BuildArchName]
}
Expand Down Expand Up @@ -1098,11 +1124,33 @@ function Get-Dependencies {
return $KnownPythons[$PythonVersion].$ArchName
}

function Get-KnownEmbeddedPython([string] $ArchName) {
if (-not $KnownPythons.ContainsKey($PythonVersion)) {
throw "Unknown python version: $PythonVersion"
}
if (-not $KnownPythons[$PythonVersion].ContainsKey("${ArchName}_Embedded")) {
return $null
}
return $KnownPythons[$PythonVersion]["${ArchName}_Embedded"]
}

function Install-Python([string] $ArchName) {
$Python = Get-KnownPython $ArchName
DownloadAndVerify $Python.URL "$BinaryCache\Python$ArchName-$PythonVersion.zip" $Python.SHA256
if (-not $ToBatch) {
Expand-ZipFile Python$ArchName-$PythonVersion.zip "$BinaryCache" Python$ArchName-$PythonVersion
Expand-ZipFile "Python$ArchName-$PythonVersion.zip" "$BinaryCache" "Python$ArchName-$PythonVersion"
}
}

function Install-EmbeddedPython([string] $ArchName) {
$Python = Get-KnownEmbeddedPython $ArchName
if ($null -eq $Python) {
Write-Output "Python $PythonVersion does not have an embeddable version."
return
}
DownloadAndVerify $Python.URL "$BinaryCache\EmbeddedPython$ArchName-$PythonVersion.zip" $Python.SHA256
if (-not $ToBatch) {
Expand-ZipFile "EmbeddedPython$ArchName-$PythonVersion.zip" "$BinaryCache" "EmbeddedPython$ArchName-$PythonVersion"
}
}

Expand Down Expand Up @@ -1153,6 +1201,7 @@ function Get-Dependencies {
}

Install-Python $HostArchName
Install-EmbeddedPython $HostArchName
if ($IsCrossCompiling) {
Install-Python $BuildArchName
}
Expand Down Expand Up @@ -3531,6 +3580,12 @@ function Install-HostToolchain() {
Copy-Item -Force `
-Path $SwiftDriver `
-Destination "$($HostPlatform.ToolchainInstallRoot)\usr\bin\swiftc.exe"

# Copy embeddable Python
New-Item -Type Directory -Path "$(Get-EmbeddedPythonInstallDir)" -ErrorAction Ignore | Out-Null
Copy-Item -Force -Recurse `
-Path "$(Get-EmbeddedPythonPath $HostPlatform)\*" `
-Destination "$(Get-EmbeddedPythonInstallDir)"
}

function Build-Inspect([Hashtable] $Platform) {
Expand Down Expand Up @@ -3601,6 +3656,7 @@ function Build-Installer([Hashtable] $Platform) {
INCLUDE_SWIFT_DOCC = $INCLUDE_SWIFT_DOCC;
SWIFT_DOCC_BUILD = "$(Get-ProjectBinaryCache $HostPlatform DocC)\release";
SWIFT_DOCC_RENDER_ARTIFACT_ROOT = "${SourceCache}\swift-docc-render-artifact";
PythonRoot = "$(Get-EmbeddedPythonInstallDir)"
}

Invoke-IsolatingEnvVars {
Expand Down