Linkage error when cross-compiling for `i686-pc-windows-gnu`

[rami3l]

Problem:

Hi! It's me again 🙇

This time, Rustup's new release promotion PR (rust-lang/rustup#4105) has been blocked by a failed i686-pc-windows-gnu CI workflow.

We use the same setup for i686 and x86_64 Windows GNU environments:

        include:
          - target: x86_64-pc-windows-gnu
            arch: x86_64
            mingwdir: mingw64
            gcc: x86_64-w64-mingw32-gcc
          - target: i686-pc-windows-gnu
            arch: i686
            mingwdir: mingw32
            gcc: i686-w64-mingw32-gcc

      ...

      - name: Install mingw
        uses: bwoodsend/setup-winlibs-action@v1
        if: matrix.mingwdir != ''
        with:
          architecture: ${{ matrix.arch }}

https://github.com/rust-lang/rustup/blob/14470b2b14c10e1578d0d343c4a4f8445cd96a0f/.github/workflows/ci.yaml#L369-L406

... however only the i686 build failed, with the following log (the original log is over 65535 characters so cannot be uploaded in full):

Log

error: linking with `i686-w64-mingw32-gcc` failed: exit code: 1
  |
  = note: "i686-w64-mingw32-gcc" "-Wl,C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\rustcPW6OFu\\list.def" "-fno-use-linker-plugin" "-Wl,--dynamicbase" "-Wl,--disable-auto-image-base" "-Wl,--large-address-aware" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\rsbegin.o" "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\rustcPW6OFu\\symbols.o" "D:\\a\\rustup\\rustup\\target\\i686-pc-windows-gnu\\debug\\deps\\rustls_platform_verifier-cd788630caa3cb65.rustls_platform_verifier.cb336a1d810c7676-cgu.0.rcgu.o" "D:\\a\\rustup\\rustup\\target\\i686-pc-windows-gnu\\debug\\deps\\rustls_platform_verifier-cd788630caa3cb65.eaq9d82n7422zh9uikyz3u50r.rcgu.o" "-Wl,-Bstatic" "D:\\a\\rustup\\rustup\\target\\i686-pc-windows-gnu\\debug\\deps\\libwindows_sys-dce398073f37b429.rlib" "D:\\a\\rustup\\rustup\\target\\i686-pc-windows-gnu\\debug\\deps\\libwindows_targets-000f2a8795805fbd.rlib" "D:\\a\\rustup\\rustup\\target\\i686-pc-windows-gnu\\debug\\deps\\libonce_cell-5479ca988703e69c.rlib" "D:\\a\\rustup\\rustup\\target\\i686-pc-windows-gnu\\debug\\deps\\librustls-5286631754029a85.rlib" "D:\\a\\rustup\\rustup\\target\\i686-pc-windows-gnu\\debug\\deps\\libsubtle-a69e41dc448e12df.rlib" "D:\\a\\rustup\\rustup\\target\\i686-pc-windows-gnu\\debug\\deps\\libwebpki-147aa3a981f2519e.rlib" "D:\\a\\rustup\\rustup\\target\\i686-pc-windows-gnu\\debug\\deps\\libuntrusted-069618dd2bc22d7e.rlib" "D:\\a\\rustup\\rustup\\target\\i686-pc-windows-gnu\\debug\\deps\\libaws_lc_rs-0e1787fb39a68020.rlib" "D:\\a\\rustup\\rustup\\target\\i686-pc-windows-gnu\\debug\\deps\\libaws_lc_sys-f3edf43c9b29fb8a.rlib" "D:\\a\\rustup\\rustup\\target\\i686-pc-windows-gnu\\debug\\deps\\libzeroize-4ba4909b92ef0aeb.rlib" "D:\\a\\rustup\\rustup\\target\\i686-pc-windows-gnu\\debug\\deps\\librustls_pki_types-ec0a369099d5cdef.rlib" "D:\\a\\rustup\\rustup\\target\\i686-pc-windows-gnu\\debug\\deps\\liblog-e11b85a6f279c33e.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\libstd-8247db7d2014e4fe.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\libpanic_unwind-ddf6fbebc9480ac4.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\libobject-4e6069d877d8dfc9.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\libmemchr-af6b43f2653c68e0.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\libaddr2line-9160f96af7844034.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\libgimli-6f0b92784d474a40.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\libwindows_targets-3fc83359f8a29f47.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\librustc_demangle-e2e8ef4e7b6864ff.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\libstd_detect-dc78e5469202334e.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\libhashbrown-4e3e443c3a790fad.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\librustc_std_workspace_alloc-c8d170f98288b2e9.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\libminiz_oxide-fda25cc20b99350c.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\libadler-de8fcea2d7411db1.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\libunwind-8d607a9b0a35faa2.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\libcfg_if-34201b55dbf435d3.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\liblibc-a79b341223122ed9.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\liballoc-4737cdd58b92f89b.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\librustc_std_workspace_core-f5db7684bfb9a0e4.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\libcore-c6dcb1820ad384c8.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\libcompiler_builtins-b21cc6440b3af0f6.rlib" "-Wl,-Bdynamic" "-lwindows.0.52.0" "-lkernel32" "-lkernel32" "-ladvapi32" "-lntdll" "-luserenv" "-lws2_32" "-ldbghelp" "-lgcc_eh" "-l:libpthread.a" "-lmsvcrt" "-lmingwex" "-lmingw32" "-lgcc" "-lmsvcrt" "-lmingwex" "-luser32" "-lkernel32" "-Wl,--nxcompat" "-L" "D:\\a\\rustup\\rustup\\target\\i686-pc-windows-gnu\\debug\\build\\aws-lc-sys-a61a0f688c326d80\\out\\build\\artifacts\\" "-L" "C:\\Users\\runneradmin\\.cargo\\registry\\src\\index.crates.io-6f17d22bba15001f\\windows_i686_gnu-0.52.6\\lib" "-o" "D:\\a\\rustup\\rustup\\target\\i686-pc-windows-gnu\\debug\\deps\\rustls_platform_verifier-cd788630caa3cb65.dll" "-Wl,--gc-sections" "-shared" "-Wl,--out-implib=D:\\a\\rustup\\rustup\\target\\i686-pc-windows-gnu\\debug\\deps\\librustls_platform_verifier-cd788630caa3cb65.dll.a" "-nodefaultlibs" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\i686-pc-windows-gnu\\lib\\rsend.o"
  = note: C:/Users/runneradmin/AppData/Local/mingw32/bin/../lib/gcc/i686-w64-mingw32/14.2.0/../../../../i686-w64-mingw32/bin/ld.exe: D:\a\rustup\rustup\target\i686-pc-windows-gnu\debug\deps\librustls-5286631754029a85.rlib(rustls-5286631754029a85.rustls.e7195d9f7e4d8336-cgu.02.rcgu.o): in function `ZN9aws_lc_rs4aead11unbound_key10UnboundKey13open_combined17ha9600c49838a732bE':␍
          C:\Users\runneradmin\.cargo\registry\src\index.crates.io-6f17d22bba15001f\aws-lc-rs-1.11.1\src\aead/unbound_key.rs:226:(.text+0x7f50): undefined reference to `aws_lc_0_23_1_EVP_AEAD_CTX_open'␍
          C:/Users/runneradmin/AppData/Local/mingw32/bin/../lib/gcc/i686-w64-mingw32/14.2.0/../../../../i686-w64-mingw32/bin/ld.exe: D:\a\rustup\rustup\target\i686-pc-windows-gnu\debug\deps\librustls-5286631754029a85.rlib(rustls-5286631754029a85.rustls.e7195d9f7e4d8336-cgu.02.rcgu.o): in function `ZN9aws_lc_rs4aead11unbound_key10UnboundKey13seal_combined17h2615079914f53816E':␍
          C:\Users\runneradmin\.cargo\registry\src\index.crates.io-6f17d22bba15001f\aws-lc-rs-1.11.1\src\aead/unbound_key.rs:320:(.text+0x8368): undefined reference to `aws_lc_0_23_1_EVP_AEAD_CTX_seal'␍
          C:/Users/runneradmin/AppData/Local/mingw32/bin/../lib/gcc/i686-w64-mingw32/14.2.0/../../../../i686-w64-mingw32/bin/ld.exe: D:\a\rustup\rustup\target\i686-pc-windows-gnu\debug\deps\librustls-5286631754029a85.rlib(rustls-5286631754029a85.rustls.e7195d9f7e4d8336-cgu.02.rcgu.o): in function `ZN9aws_lc_rs4aead11unbound_key10UnboundKey13seal_combined17h511ec80902c27457E':␍
          C:\Users\runneradmin\.cargo\registry\src\index.crates.io-6f17d22bba15001f\aws-lc-rs-1.11.1\src\aead/unbound_key.rs:320:(.text+0x8798): undefined reference to `aws_lc_0_23_1_EVP_AEAD_CTX_seal'␍
          ...
          C:/Users/runneradmin/AppData/Local/mingw32/bin/../lib/gcc/i686-w64-mingw32/14.2.0/../../../../i686-w64-mingw32/bin/ld.exe: D:\a\rustup\rustup\target\i686-pc-windows-gnu\debug\deps\libaws_lc_rs-0e1787fb39a68020.rlib(aws_lc_rs-0e1787fb39a68020.aws_lc_rs.f856f15e4b433937-cgu.7.rcgu.o): in function `ZN9aws_lc_rs3rsa10encryption5pkcs132configure_pkcs1_crypto_operation17h862bc4c0851f5455E':␍
          C:\Users\runneradmin\.cargo\registry\src\index.crates.io-6f17d22bba15001f\aws-lc-rs-1.11.1/src\rsa\encryption/pkcs1.rs:184:(.text+0x6b9d): undefined reference to `aws_lc_0_23_1_EVP_PKEY_CTX_set_rsa_padding'␍
          collect2.exe: error: ld returned 1 exit status
          
  = note: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#rustc-link-lib)

https://github.com/rust-lang/rustup/actions/runs/12162420263/job/33919348975?pr=4105

Many thanks in advance!

Relevant details

AWS-LC for Rust versions or commit:

[[package]]
name = "aws-lc-rs"
version = "1.11.1"

[[package]]
name = "aws-lc-sys"
version = "0.23.1"

System information: GitHub Actions Runner (Windows Server)

Current runner version: '2.321.0'
Operating System
  Microsoft Windows Server 2022
  10.0.20348
  Datacenter
Runner Image
  Image: windows-2022
  Version: 20241125.1.0
  Included Software: https://github.com/actions/runner-images/blob/win22/20241125.1/images/windows/Windows2022-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/win22%2F20241125.1
Runner Image Provisioner
  2.0.385.1

[justsmth]

Hello again!

So it appears that either the bindings generated incorrectly or the build didn't properly prefix the symbols (or both). The i686-pc-windows-gnu platform is one that I've had trouble with before in our CI. (I left a TODO related to it here.)

I seem to remember that symbols for 32-bit Windows platforms (i686) were always prefixed with an additional underscore, i.e., like _aws_lc_ instead of aws_lc_. For example, you can see that in the i686-pc-windows-msvc bindings here. Is that also true for i686-pc-windows-gnu?

Could it be that external bindgen (i.e., bindgen-cli) was used? I'm looking at the logic here with suspicion.

I'll try to reproduce this build error locally on my Windows box to diagnose. Hopefully, I'll have a diagnosis soon! 🤞

[rami3l]

@justsmth Thanks for the prompt reply!

I seem to remember that symbols for 32-bit Windows platforms (i686) were always prefixed with an additional underscore, i.e., like _aws_lc_ instead of aws_lc_. For example, you can see that in the i686-pc-windows-msvc bindings here. Is that also true for i686-pc-windows-gnu?

After some investigation I do believe you should adjust the prefix for mingw:

In a case that I believe can only be hit for Clang i686-*-windows-gnu (AKA MinGW), symbols in asm need to be prefixed with _.
JuliaMath/openlibm#238

[justsmth]

Ok, so I think the problem only affects "external" bindgen, not the "internal' bindgen. (Assuming this is correct) a workaround would be to enable the bindgen feature so build uses internal instead. (The build on my local Windows box for i686-pc-windows-gnu succeeds when I enable the bindgen feature, but fails otherwise.)

I put up a draft PR here:

• Fix i686-pc-windows-gnu external bindgen #621

I plan to add some testing for this, then mark the PR ready for review. I apologize for this gap in our test coverage! Hopefully, I can quickly figure our why the CI runner on GitHub was having trouble with this target so we can prevent a future regression.

Thanks for reporting and for helping with the diagnosis! ☺️

[rami3l]

@justsmth It looks like that PR has been merged. Thanks again! BTW do you have a specific timeline for the next release?

[justsmth]

Yeah, the plan is to have a release within this next week. Possibly, Tuesday or Wednesday. It should have this and some other important changes. 😀

[rami3l]

Yeah, the plan is to have a release within this next week. Possibly, Tuesday or Wednesday. It should have this and some other important changes. 😀

@justsmth Just one question: #627 seems to be the PR that I should be waiting for, but the release is still not available on crates.io yet after it being merged, is this expected?

[justsmth]

... the release is still not available on crates.io yet after it being merged, is this expected?

Yeah, we're still working on completing changes for aws-lc-rs v1.12.0. It will depend on updated versions of both aws-lc-sys and aws-lc-fips-sys, which will contain several important build improvements. (I'm still hoping for a release this week 🤞, but it could potentially be early next week.)

I'll be sure to update here when the new release is posted. Thanks! 😃

[justsmth]

Hello again! We just released aws-lc-rs v1.12.0. 🎉 We believe our latest release resolves the issue reported here. This issue will now be closed.

If you continue to see this or other problems after upgrading, feel free to re-open this issue or open a new one. Thanks for your help in improving our library!