build_native testing on the linux/s390x

[wojiushixiaobai]

Building with cibuildwheel failed.

Maybe need to recompile libjpeg libraries.

cd ~
git clone --depth=1 https://github.com/pypdfium2-team/pypdfium2
cd ~/pypdfium2

CIBW_ARCHS=s390x \
CIBW_BUILD="*-manylinux_s390x" \
CIBW_BEFORE_ALL_LINUX="dnf -y install ninja-build freetype-devel lcms2-devel libjpeg-devel openjpeg2-devel libpng-devel zlib-devel libicu-devel libtiff-devel glib2-devel && curl -L https://github.com/loong64/gn/releases/latest/download/gn-linux-s390x.tar.gz | tar xz -C /usr/local/bin" \
CIBW_BEFORE_BUILD_LINUX="python setupsrc/pypdfium2_setup/build_native.py" \
CIBW_ENVIRONMENT_LINUX="PDFIUM_PLATFORM=sourcebuild" \
cibuildwheel  --output-dir wheelhouse .

[400/553] CXX obj/core/fxcodec/fxcodec/cjpx_decoder.o
FAILED: obj/core/fxcodec/fxcodec/cjpx_decoder.o 
g++ -MD -MF obj/core/fxcodec/fxcodec/cjpx_decoder.o.d -DUSE_UDEV -DUSE_AURA=1 -DUSE_OZONE=1 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_NONE -D_GLIBCXX_ASSERTIONS=1 -DCOMPONENT_BUILD -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DABSL_CONSUME_DLL -DABSL_FLAGS_STRIP_NAMES=0 -DUSING_SYSTEM_ICU=1 -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_STATIC -DU_IMPORT=U_EXPORT -DUSE_SYSTEM_LCMS2 -DUSE_SYSTEM_LIBOPENJPEG2 -DUSE_SYSTEM_ZLIB -DUSE_SYSTEM_LIBJPEG -I../.. -Igen -Igen/shim_headers/icuuc_shim -I../../third_party/abseil-cpp -Wall -Wno-unused-local-typedefs -Wno-maybe-uninitialized -Wno-deprecated-declarations -Wno-comments -Wno-packed-not-aligned -Wno-missing-field-initializers -Wno-unused-parameter -Wno-psabi -fno-strict-overflow -fno-ident -fno-strict-aliasing -fstack-protector -funwind-tables -fPIC -pipe -pthread -ffp-contract=off -m64 -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -O2 -fdata-sections -ffunction-sections -fno-math-errno -fno-omit-frame-pointer -g0 -fvisibility=hidden -Wnarrowing -Wno-return-type -isystem/usr/include/freetype2 -isystem/usr/include/libpng16 -isystem/usr/include/openjpeg-2.4 -Wno-narrowing -Wno-class-memaccess -Wno-invalid-offsetof -std=gnu++2a -fno-exceptions -fno-rtti -fvisibility-inlines-hidden  -c ../../core/fxcodec/jpx/cjpx_decoder.cpp -o obj/core/fxcodec/fxcodec/cjpx_decoder.o
In file included from ../../core/fxcrt/raw_span.h:8,
                 from ../../core/fxcodec/jpx/cjpx_decoder.h:14,
                 from ../../core/fxcodec/jpx/cjpx_decoder.cpp:7:
../../core/fxcodec/jpx/cjpx_decoder.cpp: In member function ‘bool fxcodec::CJPX_Decoder::Init(pdfium::span<const unsigned char>, uint8_t, bool)’:
../../core/fxcodec/jpx/cjpx_decoder.cpp:497:11: error: ‘opj_decoder_set_strict_mode’ was not declared in this scope
  497 |     CHECK(opj_decoder_set_strict_mode(codec_.get(), false));
      |           ^~~~~~~~~~~~~~~~~~~~~~~~~~~
../../core/fxcrt/compiler_specific.h:39:41: note: in definition of macro ‘UNLIKELY’
   39 | #define UNLIKELY(x) __builtin_expect(!!(x), 0)
      |                                         ^
../../core/fxcodec/jpx/cjpx_decoder.cpp:497:5: note: in expansion of macro ‘CHECK’
  497 |     CHECK(opj_decoder_set_strict_mode(codec_.get(), false));
      |     ^~~~~
[401/553] CXX obj/core/fxcodec/fxcodec/scanlinedecoder.o
[402/553] CXX obj/core/fxcodec/fxcodec/jpegmodule.o
[403/553] CXX obj/core/fxcrt/fxcrt/cfx_datetime.o
[404/553] CXX obj/core/fxcrt/fxcrt/cfx_bitstream.o
[405/553] CXX obj/core/fxcrt/fxcrt/bytestring.o
ninja: build stopped: subcommand failed.
linux s390x ('glibc', '2.28')
Attempting to fetch chromium refs. If this causes setup to halt, set e.g. IGNORE_FULLVER=1
['git', 'ls-remote', '--sort', '-version:refname', '--tags', 'https://chromium.googlesource.com/chromium/src.git', '*.*.*.0'] (cwd=None)
Resolved 7191 -> 138.0.7191.0

[mara004]

I believe this is openjpeg 2.5 API.
Maybe you can just patch it out similar as shown here (slightly outdated):
https://github.com/tiran/libpdfium/blob/main/0002-openjpeg-2.4.patch

[wojiushixiaobai]

It seems no problem.

• https://pypi.org/project/wandb/#files

[mara004]

Sorry, what does wandb have to do with that?

Interesting project though, I just skimmed their hatch_build.py. Maybe we should consider switching away from setuptools...

[wojiushixiaobai]

wandb-0.22.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
pypdfium2-5.0.0b2-py3-none-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl

These packages are constrained by glibc limitations.

Explanation:
Even though the package uses the stable Python ABI (abi3), it still relies on the system's glibc version. The manylinux specification enforces glibc compatibility requirements to ensure the binary can run across different Linux distributions with consistent C library support.

[mara004]

I'm aware of that, but all the same both 2_27 and 2_28 does not make sense, as explained in #368 (reply in thread).

The wandb wheel, on the other hand, uses 2_17 and the older manylinux2014 scheme, which is OK.

I think this might be a bug in the cibuildwheel version we're using. I'll try and see if downgrading might fix the issue.

[wojiushixiaobai]

• https://github.com/wandb/wandb/blob/main/.github/workflows/release-sdk.yml#L144
• https://cibuildwheel.pypa.io/en/stable/options/#linux-image

      - name: Build wheels
        uses: pypa/cibuildwheel@v2.19.2

or

      - name: Build wheels
        uses: pypa/cibuildwheel@v3.2.0
        env:
          CIBW_MANYLINUX_*_IMAGE: manylinux2014

[wojiushixiaobai]

• https://github.com/wojiushixiaobai/pypdfium2/actions/runs/16837457314/job/47700332928
• --> Add RedHat patches for big-endian CPUs <--

 FAILED: obj/core/fxcrt/fxcrt/cfx_seekablestreamproxy.o 
  g++ -MD -MF obj/core/fxcrt/fxcrt/cfx_seekablestreamproxy.o.d -DUSE_UDEV -DUSE_AURA=1 -DUSE_OZONE=1 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_NONE -D_GLIBCXX_ASSERTIONS=1 -DCOMPONENT_BUILD -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DABSL_CONSUME_DLL -DABSL_FLAGS_STRIP_NAMES=0 -DU_USING_ICU_NAMESPACE=0 -DU_ENABLE_DYLOAD=0 -DUSE_CHROMIUM_ICU=1 -DU_ENABLE_TRACING=1 -DU_ENABLE_RESOURCE_TRACING=0 -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_FILE -I../.. -Igen -I../../third_party/abseil-cpp -I../../third_party/icu/source/common -I../../third_party/icu/source/i18n -I../../third_party/fast_float/src/include -Wall -Wno-unused-local-typedefs -Wno-maybe-uninitialized -Wno-deprecated-declarations -Wno-comments -Wno-packed-not-aligned -Wno-missing-field-initializers -Wno-unused-parameter -Wno-psabi -fno-strict-overflow -fno-ident -fno-strict-aliasing -fstack-protector -funwind-tables -fPIC -pipe -pthread -ffp-contract=off -m64 -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -O2 -fdata-sections -ffunction-sections -fno-math-errno -fno-omit-frame-pointer -g0 -fvisibility=hidden -Wnarrowing -Wno-return-type -isystem/usr/include/freetype2 -isystem/usr/include/libpng16 -Wno-narrowing -Wno-class-memaccess -Wno-invalid-offsetof -std=gnu++2a -fno-exceptions -fno-rtti -fvisibility-inlines-hidden  -c ../../core/fxcrt/cfx_seekablestreamproxy.cpp -o obj/core/fxcrt/fxcrt/cfx_seekablestreamproxy.o
  ../../core/fxcrt/cfx_seekablestreamproxy.cpp: In member function ‘size_t CFX_SeekableStreamProxy::ReadBlock(pdfium::span<wchar_t, 18446744073709551615, wchar_t*>)’:
  ../../core/fxcrt/cfx_seekablestreamproxy.cpp:204:9: error: ‘m_wCodePage’ was not declared in this scope; did you mean ‘GetCodePage’?
    204 |     if (m_wCodePage == FX_CodePage::kUTF16LE) {
        |         ^~~~~~~~~~~
        |         GetCodePage

[mara004]

Oh yes, that's a copy paste error in the big-endian clause, thank you for noticing!
m_wCodePage has been renamed to code_page_. I did it in the little-endian clause and compiled/tested it, but the other part was unreached.
Will fix this soon, sorry for the inconvenience.

[mara004]

Should be fixed now I hope, cf 5d9471f

[mara004]

Hi @wojiushixiaobai,

may I ask for you help again? I wanted to do some emulated testing locally and did as instructed on [1]:

docker run --privileged --rm tonistiigi/binfmt --install all
docker run --rm --platform linux/loong64 -it ghcr.io/loong64/debian:trixie-slim bash

but the latter command immediately terminates without output, and fails to enter the container.
With -D, I get

DEBU[0000] [hijack] End of stdout                       
 EBU[0000] Error resize: Error response from daemon: container 12f646fb0b76ea3db11b6a87d004c3b6ac244078c5a41b9da51a40287cbe788b is not running

Do you know this issue? What can I do about it?
(Fun fact: I tried consulting an LLM but it only spits out garbage, as usual...)

[1]: https://github.com/loong64#quick-start

[mara004]

Thanks for the suggestion. I ran your commands and rebooted, unfortunately it didn't change the situation.
The only difference is that I got this error message (after reboot):

$ docker run --rm --platform linux/loong64 -it ghcr.io/loong64/debian:trixie-slim bash
exec /usr/bin/bash: exec format error

which however disappears after running docker run --privileged --rm tonistiigi/binfmt --install all again.

[mara004]

I think, after all, the best approach might be to try this on a different device for the time being, as the binfmt state on my system just seems to be fouled up, and I'm tired of troubleshooting this any further. I'll eventually have to install a newer distro in any case.

[wojiushixiaobai]

After restarting the operating system, you need to execute the command.

docker run --privileged --rm tonistiigi/binfmt --install all

reference ...

root@VM-0-11-debian:~# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
NAME="Debian GNU/Linux"
VERSION_ID="13"
VERSION="13 (trixie)"
VERSION_CODENAME=trixie
DEBIAN_VERSION_FULL=13.1
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

root@VM-0-11-debian:~# docker version
Client: Docker Engine - Community
 Version:           28.4.0
 API version:       1.51
 Go version:        go1.24.7
 Git commit:        d8eb465
 Built:             Wed Sep  3 20:57:38 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.4.0
  API version:      1.51 (minimum version 1.24)
  Go version:       go1.24.7
  Git commit:       249d679
  Built:            Wed Sep  3 20:57:38 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.28
  GitCommit:        b98a3aace656320842a23f4a392a33f46af97866
 runc:
  Version:          1.3.0
  GitCommit:        v1.3.0-0-g4ca628d1
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

root@VM-0-11-debian:~# docker buildx ls
NAME/NODE     DRIVER/ENDPOINT   STATUS    BUILDKIT   PLATFORMS
default*      docker                                 
 \_ default    \_ default       running   v0.24.0    linux/amd64 (+4), linux/386

root@VM-0-11-debian:~# docker run --rm --platform linux/loong64 -it ghcr.io/loong64/debian:trixie-slim bash
exec /usr/bin/bash: exec format error

root@VM-0-11-debian:~# docker run --privileged --rm tonistiigi/binfmt --install all
installing: riscv64 OK
installing: loong64 OK
installing: mips64le OK
installing: mips64 OK
installing: arm64 OK
installing: arm OK
installing: s390x OK
installing: ppc64le OK
{
  "supported": [
    "linux/amd64",
    "linux/amd64/v2",
    "linux/amd64/v3",
    "linux/amd64/v4",
    "linux/arm64",
    "linux/riscv64",
    "linux/ppc64le",
    "linux/s390x",
    "linux/386",
    "linux/mips64le",
    "linux/mips64",
    "linux/loong64",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "python3.11",
    "python3.13",
    "qemu-aarch64",
    "qemu-arm",
    "qemu-loongarch64",
    "qemu-mips64",
    "qemu-mips64el",
    "qemu-ppc64le",
    "qemu-riscv64",
    "qemu-s390x"
  ]
}

root@VM-0-11-debian:~# ls -al /proc/sys/fs/binfmt_misc/
total 0
drwxr-xr-x 2 root root 0 Oct 30 21:47 .
dr-xr-xr-x 1 root root 0 Aug 17 19:26 ..
-rw-r--r-- 1 root root 0 Aug 17 19:26 python3.11
-rw-r--r-- 1 root root 0 Aug 17 19:26 python3.13
-rw-r--r-- 1 root root 0 Oct 30 21:50 qemu-aarch64
-rw-r--r-- 1 root root 0 Oct 30 21:50 qemu-arm
-rw-r--r-- 1 root root 0 Oct 30 21:50 qemu-loongarch64
-rw-r--r-- 1 root root 0 Oct 30 21:50 qemu-mips64
-rw-r--r-- 1 root root 0 Oct 30 21:50 qemu-mips64el
-rw-r--r-- 1 root root 0 Oct 30 21:50 qemu-ppc64le
-rw-r--r-- 1 root root 0 Oct 30 21:50 qemu-riscv64
-rw-r--r-- 1 root root 0 Oct 30 21:50 qemu-s390x
--w------- 1 root root 0 Aug 17 19:26 register
-rw-r--r-- 1 root root 0 Aug 17 19:26 status

root@VM-0-11-debian:~# docker buildx ls
NAME/NODE     DRIVER/ENDPOINT   STATUS    BUILDKIT   PLATFORMS
default*      docker                                 
 \_ default    \_ default       running   v0.24.0    linux/amd64 (+4), linux/arm64, linux/arm (+2), linux/ppc64le, (6 more)

root@VM-0-11-debian:~# docker run --rm --platform linux/loong64 -it ghcr.io/loong64/debian:trixie-slim uname -a
Linux 906936b3063b 6.12.38+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.38-1 (2025-07-16) loongarch64 GNU/Linux

[mara004]

Yes, I did run the command, but unfortunately same issue as before.
FWIW the docker version I'm using doesn't even have the docker buildx subcommand...

[mara004]

Oh well, I just noticed podman works just fine and didn't need any further setup to run an emulated container. So I'll use that for the portability investigation I want to do ;)
Anyway, sorry for having bothered you with this.

[mara004]

I just came across scikit-build/ninja-python-distributions#320 and https://github.com/mayeut/static-clang-images which sounds very promising. And I believe you're already aware of this, too, given the PR for loongarch mayeut/static-clang-images#2.
Do you think this is something we could use here, to speed up the builds? I'm not much a fan of building with an emulated compiler.

[mara004]

Ok, I pushed some fixes. Should work now.
https://github.com/pypdfium2-team/pypdfium2/actions/runs/19799436140/job/56724526117
016ebbf...a91a189

[mara004]

Huh, manylinux_s390x once crashed during the ctypesgen phase (1), but it passed on another occasion (2).

  [1033/1033] SOLINK ./libpdfium.so
  Packing data files for sourcebuild...
  INFO: Preprocessing /tmp/tmpw3rhp1z3.h
  INFO: gcc -E /tmp/tmpw3rhp1z3.h -dD -D __extension__= -D '__asm__(x)=' -D '__asm(x)=' -D __const=const -D CTYPESGEN=1 -U __GNUC__
  INFO: Parsing /tmp/tmpw3rhp1z3.h
  INFO: Processing description list.
  INFO: Looking for conflicting names...
  qemu: uncaught target signal 11 (Segmentation fault) - core dumped
  
  ERROR Backend subprocess exited when trying to invoke get_requires_for_build_wheel

1: https://github.com/pypdfium2-team/pypdfium2/actions/runs/19799436140/job/56724526043
2: https://github.com/pypdfium2-team/pypdfium2/actions/runs/19799177971/job/56723902120

[mara004]

The next step should have been INFO: Freeing library handle. Possibly, there's some issue with that.

[mara004]

Well, and musllinux_s390x froze on the test step. Duh.

[mara004]

Made another run, with same observation. s390x glibc crashes during ctypesgen, and s390x musl freezes during test phase.
https://github.com/pypdfium2-team/pypdfium2/actions/runs/19799762752/job/56725297780
I'll probably disable s390x for the time being.

[mara004]

Now included in releases:
https://github.com/pypdfium2-team/pypdfium2/releases/tag/5.2.0

loongarch64 can only be uploaded to GH (not PyPI) at this time.