Skip to content

Conversation

dom96
Copy link
Contributor

@dom96 dom96 commented Aug 6, 2025

This PR implements dedicated memory snapshot support in Python Workers. These dedicated snapshots are intended to be returned via the validator and then stored in the WorkerBundle.

Run down of features this PR implements:

  • new compatibility flag marked experimental which enables code paths to generate and load dedicated snapshots.
  • new exceptions PythonRuntimeError and PythonUserError to help us distinguish between internal errors in our runtime code and user code errors Implements error types for Python runtime and PyodideEntrypointHelper. #4746
  • dynamic libraries are now loaded from the vendored packages directory when necessary
  • hiwire custom serialiser/deserialiser for global JS objects that our internal code references with a user-facing error message if their code attempts the same at the top-level of their worker
  • collection of the snapshot at the very end, to ensure the full top-level execution is captured
  • type for pyodide_entrypoint_helper Implements error types for Python runtime and PyodideEntrypointHelper. #4746

@dom96 dom96 force-pushed the dominik/python-dedicated-snapshot branch 2 times, most recently from 1aeb0dd to d527488 Compare August 8, 2025 10:11
Copy link

github-actions bot commented Aug 8, 2025

The generated output of @cloudflare/workers-types matches the snapshot in types/generated-snapshot 🎉

@dom96 dom96 force-pushed the dominik/python-dedicated-snapshot branch 4 times, most recently from 06ea785 to 0b7088f Compare August 8, 2025 15:19
@dom96 dom96 requested a review from hoodmane August 8, 2025 15:27
@dom96 dom96 marked this pull request as ready for review August 8, 2025 15:27
@dom96 dom96 requested review from a team as code owners August 8, 2025 15:27
@dom96 dom96 force-pushed the dominik/python-dedicated-snapshot branch 3 times, most recently from 014ba2b to 26fad91 Compare August 8, 2025 17:10
@dom96 dom96 force-pushed the dominik/python-dedicated-snapshot branch from 26fad91 to d82d47f Compare August 11, 2025 10:22
@dom96 dom96 force-pushed the dominik/python-dedicated-snapshot branch 4 times, most recently from acc51f9 to 0c15b23 Compare August 11, 2025 17:10
@hoodmane
Copy link
Contributor

The worst case I can think of for this is:

from js import Proxy, Object

p = Proxy.revocable(Object.new(), Object.new())
p.revoke()
q = p.proxy
del p

but this seems to mess up Pyodide's code too.

@hoodmane
Copy link
Contributor

Opened upstream PR to fix:
pyodide/pyodide#5832

@hoodmane
Copy link
Contributor

Suggested change to test this new flag against all current workerd tests:

diff --git a/build/python_metadata.bzl b/build/python_metadata.bzl
index 82b812a37..b8385e6b6 100644
--- a/build/python_metadata.bzl
+++ b/build/python_metadata.bzl
@@ -80,7 +80,10 @@ def _make_bundle_version_info(versions):
         if name != "development":
             entry["id"] = _bundle_id(**entry)
         entry["feature_flags"] = [entry["flag"]]
-        entry["feature_string_flags"] = [entry["enable_flag_name"]]
+        string_flags = [entry["enable_flag_name"]]
+        if entry["pyodide_version"] != "0.26.0a2":
+            string_flags.append("python_dedicated_snapshot")
+        entry["feature_string_flags"] = string_flags
         if "packages" in entry:
             entry["packages"] = entry["packages"]["info"]["tag"]
         _add_integrity(entry)

@hoodmane
Copy link
Contributor

What happens if this is turned on with 0.26.0a2? Presumably we should bail out early?

@dom96 dom96 force-pushed the dominik/python-dedicated-snapshot branch from 0c15b23 to de28394 Compare August 12, 2025 09:59
@dom96
Copy link
Contributor Author

dom96 commented Aug 12, 2025

Suggested change to test this new flag against all current workerd tests:

Even better I think I can use impliedByAfterDate to enable this compat flag if the 0.28 flag is enabled

@dom96 dom96 force-pushed the dominik/python-dedicated-snapshot branch 4 times, most recently from 30219b2 to 8c9b44f Compare August 12, 2025 11:12
@dom96 dom96 force-pushed the dominik/python-dedicated-snapshot branch from 8c9b44f to a825bb1 Compare August 12, 2025 11:38
Copy link
Contributor

@hoodmane hoodmane left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks!

@dom96 dom96 force-pushed the dominik/python-dedicated-snapshot branch from a825bb1 to 2ebc17c Compare August 12, 2025 11:39
@dom96 dom96 force-pushed the dominik/python-dedicated-snapshot branch from 2ebc17c to bcacdfd Compare August 12, 2025 15:32
@dom96 dom96 force-pushed the dominik/python-dedicated-snapshot branch from bcacdfd to b97bbab Compare August 13, 2025 09:47
@dom96 dom96 merged commit e08e889 into main Aug 13, 2025
32 of 35 checks passed
@dom96 dom96 deleted the dominik/python-dedicated-snapshot branch August 13, 2025 11:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants