Skip to content

Conversation

@naveen521kk
Copy link
Member

@naveen521kk naveen521kk commented Nov 2, 2024

Simple rebase no conflicts. Squashed a few commits related to CI into the original.

Range Diff
$ git range-diff v3.12.5...mingw-v3.12.5 v3.12.7...wip-mingw-v3.12.7

  1:  c0a4ccb7cd6 =   1:  a3acfeb6dd8 sysconfig: make _sysconfigdata.py relocatable
  2:  cba5764ad34 =   2:  262284cbae0 build: add --with-nt-threads and make it default on mingw
  3:  ba3cbfb9639 =   3:  e6ae7005a6c Define `MS_WINDOWS` (and others) when compiling with MINGW
  4:  52e1ab063f3 =   4:  dd1020bb4d3 configure: add `MACHDEP` and `platform` on MINGW
  5:  5c2e0d853cd =   5:  39620f22035 Add default configuration for MINGW
  6:  e5a3b41af34 =   6:  7048825ad5a configure: enable largefile support by default for Mingw
  7:  cf489827528 =   7:  bc169c5d202 Add `PC/` to `CPPFLAGS` and to `SRCDIRS` on Mingw
  8:  758204f7c60 =   8:  cb807d5a80d posixmodule.c: Define necessary headers for compiling on MINGW
  9:  7f35cbb5de2 =   9:  77fc2b43579 Build winreg by default when compiling for MINGW
 10:  cea500dc8f5 =  10:  04937213dbb pyport.h: Add support for MINGW
 11:  6d1cb393654 =  11:  e7b1e1a84f3 configure: add options so that shared build is possible on MINGW
 12:  8aef384e2fc =  12:  a2ff85e633c Add dynamic loading support for MINGW
 13:  52ce4e2537a =  13:  6e630ffa2c0 Ignore main program for frozen scripts on MINGW
 14:  725cca02509 =  14:  2efd716d574 Detect winsock2 and setup _socket module on MINGW
 15:  28e1af00861 =  15:  f0a3ee08224 Enable some windows specific modules
 16:  46c000c4f87 =  16:  1d0026b0921 sysconfig: MINGW build extensions with GCC
 17:  94dd7ca741a =  17:  6ae60c52c77 sysconfig: treat MINGW builds as POSIX builds
 18:  a50bcff1946 =  18:  edf2c0631fd Add support for stdcall without underscore
 19:  0a5d63dc324 =  19:  016e0bf3059 Customize site for MINGW
 20:  26696a62a14 =  20:  3c9decb422f add python config sh
 21:  1609aab34f0 =  21:  7b0926df1ba mingw: prefer unix sep if MSYSTEM environment variable
 22:  f87b7e1d1f2 =  22:  4708b0c5f4d msys cygwin semi native build sysconfig
 23:  a7b5ff9e752 =  23:  0c14590e934 sysconfig: mingw sysconfig like posix
 24:  ca791303c7d =  24:  b0afcf86bf1 mingw use backslashes in compileall py
 25:  7833d58666a =  25:  ae428a409e5 mingw pdcurses_ISPAD
 26:  146660c6155 =  26:  725a22a4f82 remove_path_max.default
 27:  42bc1b5a9c0 =  27:  16d07919ec4 dont link with gettext
 28:  ea649239935 =  28:  1273d164548 ctypes python dll
 29:  6a05ec3dd13 =  29:  a9fbda35530 gdbm module includes
 30:  5058627e43a =  30:  aefff35b65f use gnu_printf in format
 31:  be6ded6b02d =  31:  7c70dce3c66 mingw fix ssl dont use enum_certificates
 32:  15a2ea53204 =  32:  8a211fa2c11 fix using dllhandle and winver mingw
 33:  93b4bbd6258 =  33:  8380abb8918 Add AMD64 to sys config so msvccompiler get_build_version works
 34:  138a4c6a446 =  34:  5d950f6f6c9 MINGW link with additional library
 35:  161f3b644ce =  35:  948778ba012 install msilib
 36:  9cc9d3d0c3c =  36:  d1b94dfe4f3 fix signal module build
 37:  474a18d07ad =  37:  d53c6dd566a build: build winconsoleio and _testconsole
 38:  d05be5c118d =  38:  5450adb3215 expose sem_unlink
 39:  debddaf4b4a =  39:  12844044107 Use cygpty while using isatty
 40:  8a3fd6109d8 =  40:  9971c7f6328 build: link win resource files and build pythonw
 41:  29cc76e8473 =  41:  63e9a2c4403 fix isselectable
 42:  ecacc24bff6 =  42:  42fbb4b7612 configure.ac: fix inet_pton check
 43:  3fa8828664f =  43:  9161633a3fb pass gen profile ldflags
 44:  8920050edd8 =  44:  bed5e43e13c pkg config windows must link ext with python lib
 45:  8a01da234af =  45:  60fa368cd5c importlib bootstrap path sep
 46:  9a372d04def =  46:  574eac719f6 warnings fixes
 47:  7cfcf8aba71 =  47:  6f08c45053a fix build testinternalcapi
 48:  df36a51a5ac =  48:  2f465790437 clang arm64
 49:  9e43b2b81cc =  49:  3df1908f6ba configure.ac: set MINGW stack reserve
 50:  c80ff73cde3 =  50:  3e98f3264aa Don't use os.pathsep to find EOF not all distributions in win32 have them as \ instead check using sys.platform
 51:  25c454ecefa !  51:  6718a05a990 Fix extension suffix for c-extensions on mingw
    @@ Makefile.pre.in: CONFINCLUDEPY=	$(CONFINCLUDEDIR)/python$(LDVERSION)
     +PYD_PLATFORM_TAG = @PYD_PLATFORM_TAG@
      LDSHARED=	@LDSHARED@ $(PY_LDFLAGS)
      BLDSHARED=	@BLDSHARED@ $(PY_CORE_LDFLAGS)
    - LDCXXSHARED=	@LDCXXSHARED@
    + LDCXXSHARED=	@LDCXXSHARED@ $(PY_LDFLAGS)
     @@ Makefile.pre.in: Python/dynload_hpux.o: $(srcdir)/Python/dynload_hpux.c Makefile
      
      Python/dynload_win.o: $(srcdir)/Python/dynload_win.c Makefile
 52:  c29d51b354d =  52:  09c807bd87a Change the `get_platform()` method in sysconfig
 53:  9ea12576fd0 =  53:  874c4cf58ba build: Fix ncursesw include lookup
 54:  507ec682547 =  54:  faa345e076d tests: fix test_bytes
 55:  d28b9d97b11 =  55:  c5e37c1ec1c time: fix strftime not raising for invalid year values
 56:  3a658507823 =  56:  6832f49d106 ctypes: find_library('c') should return None with ucrt
 57:  af9d6179fa1 =  57:  5e67480997f build: Disable checks for dlopen/dlfcn
 58:  3a43ce5faf7 =  58:  544938e0f84 Fix install location of the import library
 59:  b0e5bce4325 =  59:  3cbee8a8829 build: Integrate venvlauncher build/installation into the Makefile
 60:  0ef0faaf641 =  60:  28a15ba4aa1 configure.ac: set _WIN32_WINNT version
 61:  11883cf272a =  61:  680549f22bf configure.ac: don't check for clock_ functions
 62:  44d8cdddaa2 =  62:  ed460384afa expanduser: normpath paths coming from env vars
 63:  6ff102ff681 !  63:  542f89de426 CI: test the build and add some mingw specific tests
    @@ Commit message
           ubuntu-18.04 is gone now
         - CI: updates for mstorsjo/llvm-mingw moving to 20.04
         - CI: 3.10 -> 3.11
    +    - CI: remove `--with-system-libmpdec` for cross builds
    +    - CI: move gcc cross build to Ubuntu for newer mingw-w64
    +
    +            The build requires headers that are only available in v12 (processsnapshot.h)
    +            and Arch is still on v11. So move things to Ubuntu.
    +            Similarely to other cross builds remove the system libs configure flags, those
    +            were never there and now fail properly with the 3.12 build system.
    +
    +    Co-authored-by: Christoph Reiter <[email protected]>
     
      ## .github/workflows/mingw.yml (new) ##
     @@
    @@ .github/workflows/mingw.yml (new)
     +  cross-gcc-x86_64:
     +    runs-on: ubuntu-latest
     +    container:
    -+      image: archlinux:base-devel
    ++      image: ubuntu:24.10
     +    steps:
     +      - uses: actions/checkout@v3
     +      - name: Install deps
     +        run: |
    -+          pacman --noconfirm -Suuy
    -+          pacman --needed --noconfirm -S mingw-w64-gcc autoconf-archive autoconf automake zip
    ++          apt-get update
    ++          DEBIAN_FRONTEND=noninteractive apt-get install -y \
    ++            gcc-mingw-w64-x86-64 \
    ++            g++-mingw-w64-x86-64 \
    ++            autoconf-archive \
    ++            autoconf \
    ++            automake \
    ++            zip \
    ++            make \
    ++            pkg-config
     +
     +      - uses: actions/setup-python@v4
     +        with:
    @@ .github/workflows/mingw.yml (new)
     +            --host=x86_64-w64-mingw32 \
     +            --build=x86_64-pc-linux-gnu \
     +            --enable-shared \
    -+            --with-system-expat \
    -+            --with-system-ffi \
    -+            --with-system-libmpdec \
     +            --without-ensurepip \
     +            --enable-loadable-sqlite-extensions \
     +            --with-build-python=yes
    @@ .github/workflows/mingw.yml (new)
     +            --host=${{ matrix.arch }}-w64-mingw32 \
     +            --build=x86_64-pc-linux-gnu \
     +            --enable-shared \
    -+            --with-system-expat \
    -+            --with-system-ffi \
    -+            --with-system-libmpdec \
     +            --without-ensurepip \
     +            --without-c-locale-coercion \
     +            --enable-loadable-sqlite-extensions \
 64:  67680b59f59 =  64:  cf1a7c414ad Define PY3_DLLNAME to fix build
 65:  9277ca4f906 =  65:  e73edf0faac _testconsole.c: Fix casing & path sep
 66:  04211609bed =  66:  c42eb5114b3 Return consistent architecture markers for python on mingw/armv7
 67:  38336800b0a =  67:  453c2f9ebd7 handle ncursesw pkg-config when cross-compiling
 68:  dffc0d2ab82 =  68:  7fa078ba30f mingw_smoketests: fix _UCRT condition
 69:  672bbf2b622 =  69:  9d9709384da Modify `sys.winver` to match upstream
 70:  d6fae2190db =  70:  4f07f8d4b6c Change user site-packages path to include the environment info
 71:  d8ccb0fad07 =  71:  75adaac9684 configure: Include a header in the check for _beginthread
 72:  285cf3ea318 =  72:  a7c51e3c9c4 configure.ac: Default to --without-c-locale-coercion on Windows
 73:  0b86018d877 =  73:  c7078c9ed5a Fix failing tests
 74:  840f2a52c12 =  74:  9f29cd55077 Don't change os.sep with an empty MSYSTEM env var, not just a missing one
 75:  1d988b67d45 =  75:  155512dbee7 def VPATH when compiling `Python/sysmodule.c`
 76:  80014a3bf3a =  76:  a3ee7854351 Make `_Py_CheckPython3` extern
 77:  64771d10137 =  77:  81203b0f0f3 link with bcrypt
 78:  14393dc6a7b =  78:  a0d79febd4d correctly find native python
 79:  d24aa611f64 =  79:  787f7f34c4d Add extra flags for `_bootstrap_python`
 80:  b27d90674f1 =  80:  d57acf9678d posixmodule: undefine `HAVE_OPENDIR`
 81:  eaec91d4821 =  81:  f8b3c54197b getpath.py: add support for mingw
 82:  4e6989b780c =  82:  dcca8724224 Don't  build _posixsubprocess on Windows.
 83:  5bab8fce8b0 =  83:  f1a210bc0e0 `_ssl`: link with `ws2_32`
 84:  74332f47b99 =  84:  54a4ff72c02 Always normalize path in abspath
 85:  c93499dc035 =  85:  4f37004dbbd Include `winsock.h` when checking for netdb function
 86:  7df8eb5f8a9 =  86:  c8f3ed76985 include `_multiprocessing/semaphore.c` on win32
 87:  e279f8f0bd8 =  87:  5b7c3678bef configure: build `mmap` module on win32
 88:  4140a8c8aaa =  88:  fe13b5f2a3f venv creation fixes
 89:  4e70d6b5af6 =  89:  5f3fa39b537 move the `shutdown` function where `winsock.h` is included
 90:  a4b7fa1c447 =  90:  2b6ce332f09 configure.ac: set BUILDEXEEXT and EXEEXT
 91:  9463917a83e =  91:  b3e2010c959 configure.ac: fix building some test modules
 92:  ba16e2554e5 =  92:  c8160196252 Don't convert `sysconfig.get_config_var('VPATH')` to an absolute path
 93:  1948d4ca165 =  93:  5a66a12f6c3 Always convert `/` to `\\` before passing though pathcch functions
 94:  da9de5a0471 =  94:  9876cfde362 Build venvlauncher.exe from PC/launcher.c
 95:  cac4689f803 =  95:  b3abc553bfc getpath.py: fix `dirname`
 96:  0f82cd178b9 =  96:  2b65ae5addc getpath: use normpath on all generated paths
 97:  189df6a793e =  97:  8e3e69f73bd pathconfig: normpath sys.path[0]
 98:  2e3a8f2eca2 =  98:  1f2fc73991f smoketests: add some tests for sys/site paths
 99:  c932248adbf =  99:  5a1fa7ee5b0 Search DLLs only on paths added using `add_dll_directory()`.
100:  7c15ee5a841 = 100:  d144a65dc68 Build and install libpython3.dll
101:  b3aa1ac9595 = 101:  b14f90e795f Port GetPythonImport() to mingw
102:  3b97429a0d4 = 102:  212fb2b5a7b LoadLibraryExW: make sure to only use backslashes for paths
103:  83ae58bc179 = 103:  f555defa5ce mingw_smoketests: add a test to check if sysconfig returns correct values for `LIBDIR`
104:  07c04618380 = 104:  292f121b1fd Add missing library for multiprocessing module
105:  4de6986b458 = 105:  14c2ac8e2bd selectmodule: add support for MINGW
106:  06fec657bc1 = 106:  8bbd40b25c0 readline: disable on MINGW
107:  0f8827aabb1 = 107:  fae06841620 Add libraries to fix compile of ctypes on MINGW
108:  27e0ab2d7d9 ! 108:  9728b830157 Add `-lpython<version>` while linking stdlib
    @@ Metadata
      ## Commit message ##
         Add `-lpython<version>` while linking stdlib
     
    +    Generalize the `MINGW` case.
    +    In theory, it should be correct for other system too
    +    and would fix cross-compilation
    +
      ## Modules/makesetup ##
     @@ Modules/makesetup: NL='\
      '
    @@ Modules/makesetup: CYGWIN*) if test $libdir = .
      	 	ExtraLibDir='$(LIBPL)'
      	 fi
      	 ExtraLibs="-L$ExtraLibDir -lpython\$(LDVERSION)";;
    -+MINGW*)
    ++*)
     +	 ExtraLibs='$(BLDLIBRARY)'
     +	 ExtraLibDepends='$(LIBRARY_DEPS)';;
      esac
109:  cc5349878a8 = 109:  5f64c7c7f09 Enable `_uuid` on MINGW
110:  f1aee591d3c = 110:  6727c38a571 Enable curses library on MINGW
111:  0502c67031a = 111:  45a1b7216be mingw_smoketests: disable distutils related tests
112:  40d6d7b4fa5 = 112:  bef3bbfe383 mingw_smoketests: skip test_c_ext_build for in-tree builds
113:  7d03eb94846 = 113:  c05a521d7ca configure.ac: use autoconf 2.69
114:  0789bf9b2fe = 114:  e95b98d8ec2 CI: clean up ignored tests
115:  d97008f0b08 = 115:  18bfe4a6cef gcc: make `incompatible-pointer-types` as warning
116:  ce773f350d1 ! 116:  038178f55e9 CI: normalize path in test_makefile
    @@ Metadata
     Author: Naveen M K <[email protected]>
     
      ## Commit message ##
    -    CI: normalize path in test_makefile
    +    test_makefile: normalize path
    +
    +    so that it passes on mingw
     
      ## Lib/test/test_tools/test_makefile.py ##
     @@ Lib/test/test_tools/test_makefile.py: def list_test_dirs(self):
117:  d6355bcf174 = 117:  6f807cd16d8 `sysconfig.get_platform()`: use consistent naming
118:  b00d375f1ca = 118:  265de885d2d Add ucrt to version string
119:  00a892e3b33 <   -:  ----------- CI: remove `--with-system-libmpdec` for cross builds
120:  9245c493a3a = 119:  ae4d019ded8 socketmodule: fix captilization of headers
121:  12c06f32e2a <   -:  ----------- makesetup: generalize the `MINGW` case
122:  074f4156dd0 <   -:  ----------- CI: move gcc cross build to Ubuntu for newer mingw-w64

mingwandroid and others added 30 commits November 2, 2024 15:32
Co-authored-by: Алексей <[email protected]>
Co-authored-by: lovetox <[email protected]>
Co-authored-by: Naveen M K <[email protected]>
Co-authored-by: Алексей <[email protected]>
Co-authored-by: Christoph Reiter <[email protected]>
This is used throughout the Python code but is not defined
when compiling with Mingw-w64 compilers, so define it.
In MSVC these are defined in PC/pyconfig.h, we have to
define it manually because pyconfig.h is created by
autotools.

Co-authored-by: Алексей <[email protected]>
Co-authored-by: Christoph Reiter <[email protected]>
Co-authored-by: Naveen M K <[email protected]>
Co-authored-by: Алексей <[email protected]>
Co-authored-by: Christoph Reiter <[email protected]>
Largefile is supported on Windows.

Co-authored-by: Naveen M K <[email protected]>
Co-authored-by: Алексей <[email protected]>
Co-authored-by: Christoph Reiter <[email protected]>
Also, build `nt` module instead of `posix` when compiling
with MINGW.

Co-authored-by: Naveen M K <[email protected]>
Co-authored-by: Алексей <[email protected]>
Co-authored-by: Алексей <[email protected]>
Co-authored-by: Christoph Reiter <[email protected]>
Co-authored-by: Naveen M K <[email protected]>
Co-authored-by: Алексей <[email protected]>
Co-authored-by: Алексей <[email protected]>
Co-authored-by: Naveen M K <[email protected]>
Co-authored-by: Алексей <[email protected]>
Co-authored-by: Naveen M K <[email protected]>
This is the case used in MINGW

Co-authored-by: Алексей <[email protected]>
Co-authored-by: Алексей <[email protected]>
Co-authored-by: Christoph Reiter <[email protected]>
Co-authored-by: Алексей <[email protected]>
Co-authored-by: Christoph Reiter <[email protected]>
Co-authored-by: Алексей <[email protected]>
Co-authored-by: Christoph Reiter <[email protected]>
Co-authored-by: cat <[email protected]>
Co-authored-by: Naveen M K <[email protected]>
Co-authored-by: Алексей <[email protected]>
Co-authored-by: Алексей <[email protected]>
Co-authored-by: Алексей <[email protected]>
Co-authored-by: Алексей <[email protected]>
Co-authored-by: Christoph Reiter <[email protected]>
Co-authored-by: Алексей <[email protected]>
Co-authored-by: Алексей <[email protected]>
naveen521kk and others added 26 commits November 2, 2024 15:44
- Use function available in original msvcrt.dll
  _wdupenv_s -> _wgetenv_s
  fread_s -> fread
- Add a test for checking the new launchers
also, fix finding prefix when in a venv
Instead of just calling normpath in abspath just call normpath
on all the config results. This makes sure we don't change getpath.py
too much and still cover all outputs.

This fixes sys.exec_prefix not matching sys.prefix, see
msys2-contrib#142
this gets added as first element in sys.path.
use normpath to make sure it uses the right path separator
This is the default behavior on upstream Python. We previously
reverted this behavior because it broke some use cases.

The old behavior can be restored by setting the environment variable
PYTHONLEGACYWINDOWSDLLLOADING to 1.

Fixes msys2-contrib#48
Also fixes msys2-contrib#141

Signed-off-by: Naveen M K <[email protected]>
This provides the limited ABI as a separate DLL, forwarding to the
real one. This makes linking with "-lpython3" work.

Fixes python#147
This looks for DLL names in the import table but while with MSVC the DLL
is named python311.dll in our case it is named libpython3.11.dll.
Adjust the strings and lengths accordingly.
It seems like in case the path passed to it is absolute, but contains
forward slashes then LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR does not work
and DLLs in the same directory as the extension are not considered.

This occurs in our fork because in MSYS2-mode the extension loader will
normalize to forward slashes before.

Normalize everything to backslashes again before passing it to LoadLibraryExW.

Fixes python#151
`ws2_32` is required when building that module
Generalize the `MINGW` case.
In theory, it should be correct for other system too
and would fix cross-compilation
llvm-mingw only has autoconf 2.69
Remove tests which are passing now, or no longer exist, and
move tests which only fail with msvcrt into a separate file.
gcc 14 changed `incompatible-pointer-types` from warning to error
for now, make it as warning to avoid build failure
so that it passes on mingw
Currently, the platform names are hardcoded in many places,
and are not named consistently. This commit fixes it by
standardizing the names to be returned by `sysconfig.get_platform`.
The naming is based on msys2-contrib#167 (comment)

Similarly, `EXT_SUFFIX` is also standardized to be consistent with the platform names.

Signed-off-by: Naveen M K <[email protected]>
`Rpc` should be `rpc`. It error's out in
cross-compilation.
@naveen521kk naveen521kk marked this pull request as ready for review November 2, 2024 10:24
@naveen521kk naveen521kk merged commit d21f60d into msys2-contrib:mingw-v3.12.7 Nov 2, 2024
14 checks passed
@naveen521kk naveen521kk deleted the wip-mingw-v3.12.7 branch November 2, 2024 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants