gh-138489: Install `build-details.json` on Windows #138490

AA-Turner · 2025-09-03T23:36:09Z

Closes #138489. cc @FFY00 as PEP author.

I'd appreciate a review/sanity check on the .vcxproj files.

I've also set abi.flags in the JSON document to the empty list, rather than e.g. copying the new ABIFLAGS code. The PEP defines it as:

Build configuration flags, used to calculate the extension suffix.

The flags MUST be defined in the order they appear on the extension suffix.

Examples: ['t', 'd'], etc.

A

Issue: PEP 739 build-details.json is missing on some platforms #138489

PCbuild/python.vcxproj

AA-Turner · 2025-09-04T00:22:20Z

PCbuild/python.vcxproj

+    <Exec Command='setlocal
+set PYTHONPATH=$(PySourcePath)Lib


Unsure if these bits are needed, I copied them from the entry slightly further down.

* KeyError is not raised for defaultdict * Fix relative paths on different drives on Windows

Lib/test/test_build_details.py

Split to python#138510

AA-Turner · 2025-09-04T19:57:04Z

Tools/build/generate-build-details.py

    data['language']['version_info'] = version_info_to_dict(sys.version_info)

-    data['implementation'] = vars(sys.implementation)
+    data['implementation'] = vars(sys.implementation).copy()


This is needed, currently the script overwrites sys.implementation.version in the running interpreter.

zooba · 2025-09-05T15:06:34Z

I'll be honest, I haven't looked at the files these generate. For Windows, I'd expect them to be basically static - installs are entirely portable and all options are defined in our repo (though some can be overridden at build time, so picking them up later is okay). Is that the case here?

We likely also need changes to the PC/layout scripts to make sure they get picked up, and probably files in Tools/msi/exe or /core (depends on whether having the exe installed is a necessary condition, which I think it is, so putting into exe makes more sense). I don't think the deprecation of the old installer extends to not including changes to CPython, only changes that are unique to the installer (e.g. we'd definitely include a new module in it, but wouldn't add new UI options to include/exclude that module).

AA-Turner · 2025-09-05T15:10:10Z

Example build-details.json (expand)

{
  "schema_version": "1.0",
  "base_prefix": "C:\\path\\to\\cpython",
  "base_interpreter": "C:\\path\\to\\cpython\\Scripts\\python315",
  "platform": "win-amd64",
  "language": {
    "version": "3.15",
    "version_info": {
      "major": 3,
      "minor": 15,
      "micro": 0,
      "releaselevel": "alpha",
      "serial": 0
    }
  },
  "implementation": {
    "name": "cpython",
    "cache_tag": "cpython-315",
    "version": {
      "major": 3,
      "minor": 15,
      "micro": 0,
      "releaselevel": "alpha",
      "serial": 0
    },
    "hexversion": 51314848,
    "supports_isolated_interpreters": true
  },
  "abi": {
    "flags": [],
    "extension_suffix": "_d.cp315-win_amd64.pyd"
  },
  "suffixes": {
    "source": [
      ".py",
      ".pyw"
    ],
    "bytecode": [
      ".pyc"
    ],
    "extensions": [
      "_d.cp315-win_amd64.pyd",
      "_d.pyd"
    ]
  },
  "libpython": {
    "dynamic": "C:\\path\\to\\cpython\\libs\\python315_d.dll",
    "link_extensions": false
  },
  "c_api": {
    "headers": "C:\\path\\to\\cpython\\Include"
  }
}

zooba · 2025-09-05T15:21:52Z

Yeah, that's what I thought. It's okay for one you've built in place, but we shouldn't include any absolute paths in a "real" build (and nothing relative to "outside" the prefix).

AA-Turner · 2025-09-05T15:33:16Z

@zooba there is a --relative-paths option (currently broken on Windows as it doesn't account for cross-drive paths, but oh well) that would produce something like the below. Should we use it by default on the Windows builds?

The remaining issue would be the base_prefix key, as that's what all the other paths are relative to. Perhaps setting to . could work? (the spec allows setting the key as a relative path from the directory containing build_details.json to the prefix.

example with relative paths

{
  "schema_version": "1.0",
  "base_prefix": "C:\\path\\to\\cpython",
  "base_interpreter": ".\\Scripts\\python315",
  "platform": "win-amd64",
  "language": {
    "version": "3.15",
    "version_info": {
      "major": 3,
      "minor": 15,
      "micro": 0,
      "releaselevel": "alpha",
      "serial": 0
    }
  },
  "implementation": {
    "name": "cpython",
    "cache_tag": "cpython-315",
    "version": {
      "major": 3,
      "minor": 15,
      "micro": 0,
      "releaselevel": "alpha",
      "serial": 0
    },
    "hexversion": 51314848,
    "supports_isolated_interpreters": true
  },
  "abi": {
    "flags": [],
    "extension_suffix": "_d.cp315-win_amd64.pyd"
  },
  "suffixes": {
    "source": [
      ".py",
      ".pyw"
    ],
    "bytecode": [
      ".pyc"
    ],
    "extensions": [
      "_d.cp315-win_amd64.pyd",
      "_d.pyd"
    ]
  },
  "libpython": {
    "dynamic": ".\\libs\\python315_d.dll",
    "link_extensions": false
  },
  "c_api": {
    "headers": ".\\Include"
  }
}

A

zooba · 2025-09-09T20:06:02Z

currently broken on Windows as it doesn't account for cross-drive paths

This shouldn't matter for real, as none of the paths should ever point outside of the prefix. For local builds it might, but a cross-drive path in that case just becomes an absolute path.

And yeah, setting base_prefix to . is the idea (though it probably needs to be ..\\.. in a source tree build). That's why we allowed it to be relative.

# Conflicts: # Lib/test/test_build_details.py

AA-Turner · 2025-09-16T10:52:51Z

I've updated this, but I've had to make a few changes to generate_build_details.py to account for that source-tree layouts are different from installed layouts on Windows, and as far as I can tell there are no good existing sysconfig variables to account for this.

cc @zooba if you have time for a review!

A

zooba

Most of the change is good (based on the last 4 commits), but we're going to have to clone a lot more of the logic into PC/layout in order to handle cross-compiling/packaging.

zooba · 2025-09-16T13:42:27Z

PC/layout/support/build_details.py

+import sys
+from pathlib import Path
+
+PEP739_SCHEMA_VERSION = '1.0'


Suggested change

PEP739_SCHEMA_VERSION = '1.0'

__all__ = ["write_relative_build_details"]

PEP739_SCHEMA_VERSION = '1.0'

zooba · 2025-09-16T14:11:05Z

PC/layout/support/build_details.py

+ROOT_DIR = Path(
+    __file__,  # PC/layout/support/build_details.py
+    '..',  # PC/layout/support
+    '..',  # PC/layout
+    '..',  # PC
+    '..',  # <src/install dir>
+).resolve()
+TOOLS_BUILD_DIR = ROOT_DIR / 'Tools' / 'build'
+
+sys_path = sys.path[:]
+try:
+    sys.path.insert(0, str(TOOLS_BUILD_DIR))
+    import generate_build_details
+except ImportError:
+    generate_build_details = None
+finally:
+    sys.path = sys_path
+    del sys_path


I'm afraid this probably needs to replicate most of the functionality from that script and do trickier calculations. The repo containing the running PC/layout isn't necessarily the same as the commit used to build, and the running interpreter isn't necessarily the same as the one we just built.

So the details have to be calculated from the referenced source files (i.e. layout.support.constants and layout.support.arch), not from the current runtime at all.

That said, I haven't dug into the write_build_details function and maybe it does it, but it really doesn't look like enough information is being provided to make it viable (especially when you factor in that {base_path}\python.exe is not executable).

zooba · 2025-09-16T14:12:01Z

PC/layout/support/options.py

            "flat-dlls",
            "underpth",
            "precompile",
+            "build-details-json",


I suspect the contents of the file needs to be different for the embeddable package (no Lib directory or Includes/libs, for example). Maybe just omit it for now

FFY00 · 2025-10-04T14:33:30Z

PC/layout/support/build_details.py

+ROOT_DIR = Path(
+    __file__,  # PC/layout/support/build_details.py
+    '..',  # PC/layout/support
+    '..',  # PC/layout
+    '..',  # PC
+    '..',  # <src/install dir>
+).resolve()
+TOOLS_BUILD_DIR = ROOT_DIR / 'Tools' / 'build'
+
+sys_path = sys.path[:]
+try:
+    sys.path.insert(0, str(TOOLS_BUILD_DIR))
+    import generate_build_details
+except ImportError:
+    generate_build_details = None
+finally:
+    sys.path = sys_path
+    del sys_path


Do we really need to import the script here instead of invoking it?

We can't do either - we might need to generate the file without being able to launch the runtime. See my comment above on the same section of code.

Install build-details.json on Windows

1901e80

AA-Turner requested a review from zooba September 3, 2025 23:36

AA-Turner added the needs backport to 3.14 bugs and security fixes label Sep 3, 2025

AA-Turner requested a review from a team as a code owner September 3, 2025 23:36

bedevere-app bot added the awaiting core review label Sep 3, 2025

bedevere-app bot mentioned this pull request Sep 3, 2025

PEP 739 build-details.json is missing on some platforms #138489

Open

AA-Turner commented Sep 4, 2025

View reviewed changes

PCbuild/python.vcxproj Outdated Show resolved Hide resolved

Update python.vcxproj

dbe1cbd

AA-Turner commented Sep 4, 2025

View reviewed changes

AA-Turner added 4 commits September 4, 2025 19:45

Fix `--relative-paths

c80344c

* KeyError is not raised for defaultdict * Fix relative paths on different drives on Windows

Specify encoding when writing file

ee39ea7

Add test for --relative-paths

f5f0548

Lint (splitroot requires 3.12)

fdba9f0

AA-Turner commented Sep 4, 2025

View reviewed changes

Lib/test/test_build_details.py Show resolved Hide resolved

Revert relative-path changes.

6a2cc71

Split to python#138510

AA-Turner commented Sep 4, 2025

View reviewed changes

This comment was marked as outdated.

Sign in to view

AA-Turner added 4 commits September 15, 2025 15:52

Merge branch 'main' into pep739-windows

bd9bf46

# Conflicts: # Lib/test/test_build_details.py

Regenerate build-details.json via PC\layout\

8d58be4

Fix the generated data for Windows

a927601

Fail gracefully

d743693

AA-Turner requested review from emmatyping, erlend-aasland, ezio-melotti and hugovk as code owners September 16, 2025 10:40

AA-Turner added 2 commits September 16, 2025 11:46

bin the type statements

d18f70e

yet more linting banality

57f3b26

zooba reviewed Sep 16, 2025

View reviewed changes

FFY00 reviewed Oct 4, 2025

View reviewed changes

Uh oh!

gh-138489: Install build-details.json on Windows #138490

Are you sure you want to change the base?

gh-138489: Install build-details.json on Windows #138490

Uh oh!

Conversation

AA-Turner commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

AA-Turner Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AA-Turner Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

zooba commented Sep 5, 2025

Uh oh!

AA-Turner commented Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zooba commented Sep 5, 2025

Uh oh!

AA-Turner commented Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zooba commented Sep 9, 2025

Uh oh!

This comment was marked as outdated.

AA-Turner commented Sep 16, 2025

Uh oh!

zooba left a comment

Choose a reason for hiding this comment

Uh oh!

zooba Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

zooba Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

zooba Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

FFY00 Oct 4, 2025

Choose a reason for hiding this comment

Uh oh!

zooba Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gh-138489: Install `build-details.json` on Windows #138490

gh-138489: Install `build-details.json` on Windows #138490

AA-Turner commented Sep 3, 2025 •

edited

Loading

AA-Turner commented Sep 5, 2025 •

edited

Loading

AA-Turner commented Sep 5, 2025 •

edited

Loading