Link error for Python 3.11 on Windows in github actions CI

[LilyAcorn]

Three days ago the CI build for Kolo started failing with the following error:

Run source <(cargo llvm-cov show-env --export-prefix --manifest-path=rust/Cargo.toml)
  source <(cargo llvm-cov show-env --export-prefix --manifest-path=rust/Cargo.toml)
  .venv/Scripts/maturin develop --profile dev --extras=test_generation
  shell: C:\Program Files\Git\bin\bash.EXE --noprofile --norc -e -o pipefail {0}
  env:
    CARGO_INCREMENTAL: 0
    CARGO_TERM_COLOR: always
    CACHE_ON_FAILURE: false
    pythonLocation: C:\hostedtoolcache\windows\Python\3.11.8\x64
info: cargo-llvm-cov currently setting cfg(coverage); you can opt-out it by passing --no-cfg-coverage
    Updating crates.io index
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings
🐍 Found CPython 3.11 at D:\a\kolo\kolo\python\.venv\Scripts\python.exe
📡 Using build options features from pyproject.toml
Ignoring importlib_metadata: markers 'python_version < "3.10"' don't match your environment
Ignoring tomli: markers 'python_version < "3.11"' don't match your environment
Requirement already satisfied: asgiref>=3.6.0 in d:\a\kolo\kolo\python\.venv\lib\site-packages (3.7.2)
Collecting cerberus>=1.3.4
  Downloading Cerberus-1.3.5-py3-none-any.whl.metadata (6.0 kB)
Requirement already satisfied: click>=7.1.2 in d:\a\kolo\kolo\python\.venv\lib\site-packages (8.1.7)
Requirement already satisfied: httpx in d:\a\kolo\kolo\python\.venv\lib\site-packages (0.26.0)
Collecting msgpack>=1.0.0
  Downloading msgpack-1.0.7-cp311-cp311-win_amd64.whl.metadata (9.4 kB)
Collecting toolz>=0.11.1
  Downloading toolz-0.12.1-py3-none-any.whl.metadata (5.1 kB)
Collecting ulid-py>=1.1.0
  Downloading ulid_py-1.1.0-py2.py3-none-any.whl (25 kB)
Collecting jinja2>=3.0.0
  Downloading Jinja2-3.1.3-py3-none-any.whl.metadata (3.3 kB)
Collecting more_itertools
  Downloading more_itertools-10.2.0-py3-none-any.whl.metadata (34 kB)
Collecting sqlglot>=16.1.1
  Downloading sqlglot-21.1.1-py3-none-any.whl.metadata (16 kB)
Requirement already satisfied: colorama in d:\a\kolo\kolo\python\.venv\lib\site-packages (from click>=7.1.2) (0.4.6)
Requirement already satisfied: anyio in d:\a\kolo\kolo\python\.venv\lib\site-packages (from httpx) (4.2.0)
Requirement already satisfied: certifi in d:\a\kolo\kolo\python\.venv\lib\site-packages (from httpx) (2024.2.2)
Requirement already satisfied: httpcore==1.* in d:\a\kolo\kolo\python\.venv\lib\site-packages (from httpx) (1.0.3)
Requirement already satisfied: idna in d:\a\kolo\kolo\python\.venv\lib\site-packages (from httpx) (3.6)
Requirement already satisfied: sniffio in d:\a\kolo\kolo\python\.venv\lib\site-packages (from httpx) (1.3.0)
Requirement already satisfied: h11<0.15,>=0.13 in d:\a\kolo\kolo\python\.venv\lib\site-packages (from httpcore==1.*->httpx) (0.14.0)
Collecting MarkupSafe>=2.0 (from jinja2>=3.0.0)
  Downloading MarkupSafe-2.1.5-cp311-cp311-win_amd64.whl.metadata (3.1 kB)
Downloading Cerberus-1.3.5-py3-none-any.whl (30 kB)
Downloading msgpack-1.0.7-cp311-cp311-win_amd64.whl (222 kB)
   --------------------------------------- 222.9/222.9 kB 14.2 MB/s eta 0:00:00
Downloading toolz-0.12.1-py3-none-any.whl (56 kB)
   ---------------------------------------- 56.1/56.1 kB 3.1 MB/s eta 0:00:00
Downloading Jinja2-3.1.3-py3-none-any.whl (133 kB)
   ---------------------------------------- 133.2/133.2 kB 7.7 MB/s eta 0:00:00
Downloading more_itertools-10.2.0-py3-none-any.whl (57 kB)
   ---------------------------------------- 57.0/57.0 kB 2.9 MB/s eta 0:00:00
Downloading sqlglot-21.1.1-py3-none-any.whl (363 kB)
   --------------------------------------- 363.6/363.6 kB 11.4 MB/s eta 0:00:00
Downloading MarkupSafe-2.1.5-cp311-cp311-win_amd64.whl (17 kB)
Installing collected packages: ulid-py, cerberus, toolz, sqlglot, msgpack, more_itertools, MarkupSafe, jinja2
Successfully installed MarkupSafe-2.1.5 cerberus-1.3.5 jinja2-3.1.3 more_itertools-10.2.0 msgpack-1.0.7 sqlglot-21.1.1 toolz-0.12.1 ulid-py-1.1.0
   Compiling _kolo v2.18.0 (D:\a\kolo\kolo\python\rust)
error: linking with `link.exe` failed: exit code: 1181
  |
  = note: "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\VC\\Tools\\MSVC\\14.38.33130\\bin\\HostX64\\x64\\link.exe" "/DEF:C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\rustc3whaSn\\lib.def" "/NOLOGO" "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\rustc3whaSn\\symbols.o" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\_kolo._kolo.698db0848e58bb22-cgu.00.rcgu.o" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\_kolo._kolo.698db0848e58bb22-cgu.01.rcgu.o" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\_kolo._kolo.698db0848e58bb22-cgu.02.rcgu.o" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\_kolo._kolo.698db0848e58bb22-cgu.03.rcgu.o" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\_kolo._kolo.698db0848e58bb22-cgu.04.rcgu.o" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\_kolo._kolo.698db0848e58bb22-cgu.05.rcgu.o" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\_kolo._kolo.698db0848e58bb22-cgu.06.rcgu.o" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\_kolo._kolo.698db0848e58bb22-cgu.07.rcgu.o" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\_kolo._kolo.698db0848e58bb22-cgu.08.rcgu.o" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\_kolo._kolo.698db0848e58bb22-cgu.09.rcgu.o" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\_kolo._kolo.698db0848e58bb22-cgu.10.rcgu.o" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\_kolo._kolo.698db0848e58bb22-cgu.11.rcgu.o" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\_kolo._kolo.698db0848e58bb22-cgu.12.rcgu.o" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\_kolo._kolo.698db0848e58bb22-cgu.13.rcgu.o" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\_kolo._kolo.698db0848e58bb22-cgu.14.rcgu.o" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\_kolo._kolo.698db0848e58bb22-cgu.15.rcgu.o" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\_kolo.4vi8bja45v3cawdc.rcgu.o" "/LIBPATH:D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps" "/LIBPATH:C:\\Users\\runneradmin\\.cargo\\registry\\src\\index.crates.io-6f17d22bba15001f\\windows_x86_64_msvc-0.48.5\\lib" "/LIBPATH:C:\\hostedtoolcache\\windows\\Python\\3.11.7\\x64\\libs" "/LIBPATH:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\librmpv-6838be4b743a372d.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\librmp-a97769deadeae4a3.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\libbyteorder-6561bce2e64523f1.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\libnum_traits-30ae5a2a76d72d1f.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\libthread_local-d2f0a5a349fbfc63.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\libulid-ba3217bf2a9531cd.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\librand-bddb23aa6ee871ed.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\librand_chacha-7d519b4453f5ebd0.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\libppv_lite86-f8cc4735c20fe03a.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\librand_core-05210783e170a5a6.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\libgetrandom-fc3264852fdbbfd2.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\libonce_cell-2ef8eb3cc292aaa0.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\libbstr-66d64b5a9cb0ae6a.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\libregex_automata-a7d0db87253168e9.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\libmemchr-270a18fa33a31d5a.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\libpyo3-4a7b40c733ffafcd.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\libmemoffset-0d03c562752b51f2.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\libparking_lot-2712a6177ac4309d.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\libparking_lot_core-c73ccf32738ae8dd.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\libwindows_targets-5ed392ba99a296da.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\libcfg_if-0ac0f7494b269cb1.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\libsmallvec-0100cae57399aa89.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\liblock_api-3e4136a9e2ab229e.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\libscopeguard-5296a71d3721966e.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\libpyo3_ffi-0a83e923e513055b.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\liblibc-eca98e9b7510f5a2.rlib" "D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\libunindent-9d5b3c53fd76065f.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libprofiler_builtins-dbd8a1dffce6e5c6.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd-e493bcbfdc66a475.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libpanic_unwind-467acea86c440d1f.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_demangle-8bcec20f8d868561.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd_detect-e75d865d889e433e.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libhashbrown-cd24a5810f58b720.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_alloc-44b4ecbbc95939b2.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libunwind-ff93d2b34eb6aecc.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcfg_if-17346c417f0e9166.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\liblibc-ff5a06ebf4571d10.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\liballoc-cb2478631e21007b.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_core-58d59322b34f2b51.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcore-c192803e6895f627.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcompiler_builtins-20106e86b5dab94f.rlib" "bcrypt.lib" "advapi32.lib" "windows.0.48.5.lib" "python311.lib" "legacy_stdio_definitions.lib" "kernel32.lib" "advapi32.lib" "bcrypt.lib" "kernel32.lib" "ntdll.lib" "userenv.lib" "ws2_32.lib" "kernel32.lib" "ws2_32.lib" "kernel32.lib" "ntdll.lib" "kernel32.lib" "msvcrt.lib" "/NXCOMPAT" "/LIBPATH:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib" "/OUT:D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\_kolo.dll" "/OPT:REF,NOICF" "/DLL" "/IMPLIB:D:\\a\\kolo\\kolo\\python\\rust\\target\\debug\\deps\\_kolo.dll.lib" "/DEBUG" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\intrinsic.natvis" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\liballoc.natvis" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libcore.natvis" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libstd.natvis"
  = note: LINK : fatal error LNK1181: cannot open input file 'python311.lib'

          

error: could not compile `_kolo` (lib) due to 1 previous error
💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit code: 101": `"cargo" "rustc" "--profile" "dev" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "D:\\a\\kolo\\kolo\\python\\rust\\Cargo.toml" "--lib"`
Error: Process completed with exit code 1.

This happens reliably to our Python 3.11 on Windows build, but not to any of the other builds. I haven't changed my github actions config and I'm 99% sure it's not related to any code changes (it even failed on pure python code changes).

Does anyone have any advice for debugging this? I'm guessing it's most likely a bug in github actions but I haven't got anywhere with tracking it down.

Here's the github actions workflow up to the failure point:

  windows:
    if:
      (github.actor != 'dependabot[bot]' && github.event_name == 'push') ||
      (github.actor == 'dependabot[bot]' && github.event_name == 'pull_request_target')
    strategy:
      matrix:
        python: ["3.8", "3.9", "3.10", "3.11", "3.12", "pypy-3.9", "pypy-3.10"]
      fail-fast: false
    runs-on: windows-latest
    defaults:
      run:
        working-directory: ./python
        shell: bash
    steps:
    - name: Checkout
      if: ${{ github.event_name != 'pull_request_target' }}
      uses: actions/checkout@v3

    - name: Checkout PR
      if: ${{ github.event_name == 'pull_request_target' }}
      uses: actions/checkout@v3
      with:
        ref: ${{ github.event.pull_request.head.sha }}

    - name: Install Rust toolchain
      uses: dtolnay/rust-toolchain@stable
      with:
        components: llvm-tools-preview

    - name: Install cargo-llvm-cov
      uses: taiki-e/install-action@cargo-llvm-cov

    - uses: Swatinem/rust-cache@v2
      with:
        workspaces: python/rust

    - uses: actions/setup-python@v2
      with:
        python-version: ${{ matrix.python }}
        cache: "pip"
        cache-dependency-path: python/dev-requirements.txt
    - run: python -m venv .venv
    - run: .venv/Scripts/pip install -r dev-requirements.txt
    - run: .venv/Scripts/pip install django -U
    - run: .venv/Scripts/pip install maturin
    - run: |
        source <(cargo llvm-cov show-env --export-prefix --manifest-path=rust/Cargo.toml)
        .venv/Scripts/maturin develop --profile dev --extras=test_generation

[davidhewitt]

Hmm. Couple ideas which we could start with:

• Maybe try clearing the GitHub Actions cache?
• You could try building with PYO3_PRINT_CONFIG=1 environment variable, to see what it's configuring against and check it looks sensible.

Strange that it's just Python 3.11. From timing of three days ago, that would be when the latest runner image got rolled out: https://github.com/actions/runner-images/commits/main/images/windows/Windows2022-Readme.md

May or may not be related...

[LilyAcorn]

Thanks for suggesting clearing cache, that appears to have sorted it!