build_native testing on the linux/s390x

[wojiushixiaobai]

Building with cibuildwheel failed.

Maybe need to recompile libjpeg libraries.

cd ~
git clone --depth=1 https://github.com/pypdfium2-team/pypdfium2
cd ~/pypdfium2

CIBW_ARCHS=s390x \
CIBW_BUILD="*-manylinux_s390x" \
CIBW_BEFORE_ALL_LINUX="dnf -y install ninja-build freetype-devel lcms2-devel libjpeg-devel openjpeg2-devel libpng-devel zlib-devel libicu-devel libtiff-devel glib2-devel && curl -L https://github.com/loong64/gn/releases/latest/download/gn-linux-s390x.tar.gz | tar xz -C /usr/local/bin" \
CIBW_BEFORE_BUILD_LINUX="python setupsrc/pypdfium2_setup/build_native.py" \
CIBW_ENVIRONMENT_LINUX="PDFIUM_PLATFORM=sourcebuild" \
cibuildwheel  --output-dir wheelhouse .

[400/553] CXX obj/core/fxcodec/fxcodec/cjpx_decoder.o
FAILED: obj/core/fxcodec/fxcodec/cjpx_decoder.o 
g++ -MD -MF obj/core/fxcodec/fxcodec/cjpx_decoder.o.d -DUSE_UDEV -DUSE_AURA=1 -DUSE_OZONE=1 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_NONE -D_GLIBCXX_ASSERTIONS=1 -DCOMPONENT_BUILD -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DABSL_CONSUME_DLL -DABSL_FLAGS_STRIP_NAMES=0 -DUSING_SYSTEM_ICU=1 -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_STATIC -DU_IMPORT=U_EXPORT -DUSE_SYSTEM_LCMS2 -DUSE_SYSTEM_LIBOPENJPEG2 -DUSE_SYSTEM_ZLIB -DUSE_SYSTEM_LIBJPEG -I../.. -Igen -Igen/shim_headers/icuuc_shim -I../../third_party/abseil-cpp -Wall -Wno-unused-local-typedefs -Wno-maybe-uninitialized -Wno-deprecated-declarations -Wno-comments -Wno-packed-not-aligned -Wno-missing-field-initializers -Wno-unused-parameter -Wno-psabi -fno-strict-overflow -fno-ident -fno-strict-aliasing -fstack-protector -funwind-tables -fPIC -pipe -pthread -ffp-contract=off -m64 -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -O2 -fdata-sections -ffunction-sections -fno-math-errno -fno-omit-frame-pointer -g0 -fvisibility=hidden -Wnarrowing -Wno-return-type -isystem/usr/include/freetype2 -isystem/usr/include/libpng16 -isystem/usr/include/openjpeg-2.4 -Wno-narrowing -Wno-class-memaccess -Wno-invalid-offsetof -std=gnu++2a -fno-exceptions -fno-rtti -fvisibility-inlines-hidden  -c ../../core/fxcodec/jpx/cjpx_decoder.cpp -o obj/core/fxcodec/fxcodec/cjpx_decoder.o
In file included from ../../core/fxcrt/raw_span.h:8,
                 from ../../core/fxcodec/jpx/cjpx_decoder.h:14,
                 from ../../core/fxcodec/jpx/cjpx_decoder.cpp:7:
../../core/fxcodec/jpx/cjpx_decoder.cpp: In member function ‘bool fxcodec::CJPX_Decoder::Init(pdfium::span<const unsigned char>, uint8_t, bool)’:
../../core/fxcodec/jpx/cjpx_decoder.cpp:497:11: error: ‘opj_decoder_set_strict_mode’ was not declared in this scope
  497 |     CHECK(opj_decoder_set_strict_mode(codec_.get(), false));
      |           ^~~~~~~~~~~~~~~~~~~~~~~~~~~
../../core/fxcrt/compiler_specific.h:39:41: note: in definition of macro ‘UNLIKELY’
   39 | #define UNLIKELY(x) __builtin_expect(!!(x), 0)
      |                                         ^
../../core/fxcodec/jpx/cjpx_decoder.cpp:497:5: note: in expansion of macro ‘CHECK’
  497 |     CHECK(opj_decoder_set_strict_mode(codec_.get(), false));
      |     ^~~~~
[401/553] CXX obj/core/fxcodec/fxcodec/scanlinedecoder.o
[402/553] CXX obj/core/fxcodec/fxcodec/jpegmodule.o
[403/553] CXX obj/core/fxcrt/fxcrt/cfx_datetime.o
[404/553] CXX obj/core/fxcrt/fxcrt/cfx_bitstream.o
[405/553] CXX obj/core/fxcrt/fxcrt/bytestring.o
ninja: build stopped: subcommand failed.
linux s390x ('glibc', '2.28')
Attempting to fetch chromium refs. If this causes setup to halt, set e.g. IGNORE_FULLVER=1
['git', 'ls-remote', '--sort', '-version:refname', '--tags', 'https://chromium.googlesource.com/chromium/src.git', '*.*.*.0'] (cwd=None)
Resolved 7191 -> 138.0.7191.0

[mara004]

I believe this is openjpeg 2.5 API.
Maybe you can just patch it out similar as shown here (slightly outdated):
https://github.com/tiran/libpdfium/blob/main/0002-openjpeg-2.4.patch

[mara004]

Another question: have you been able to verify whether these wheels are truly compatible with systems other than the specific container it was built in, e.g. a newer version of the same distro, or an entirely different distro?

[wojiushixiaobai]

PS: @wojiushixiaobai Would you mind if I upstreamed your cibuildwheel workflow, or something derived from your instructions, under the project's license?

Sure.

Another question: have you been able to verify whether these wheels are truly compatible with systems other than the specific container it was built in, e.g. a newer version of the same distro, or an entirely different distro?

No, it is not fully tested, only simple tested on docker.io/library/python.

[mara004]

Thank you!

Another question: have you been able to verify whether these wheels are truly compatible with systems other than the specific container it was built in, e.g. a newer version of the same distro, or an entirely different distro?

No, it is not fully tested, only simple tested on docker.io/library/python.

The main thing I'm concerned about here is the libc++. It doesn't seem to be among the libraries bundled by auditwheel. Will it work with a different libc++ than the one it was built with? I'm no expert, but I doubt it.

So maybe we need to vendor that as well, the options to build with system libcxx are deprecated upstream anyway. But: the more dependencies we vendor, the longer the emulated build will take.

[wojiushixiaobai]

I've tested the repaired .whl built with auditwheel on multiple systems (with different glibc versions and setups), and everything works fine so far. The wheel runs without runtime errors, and I haven't encountered any libc++-related issues.

Although it's true that libc++ isn't bundled by auditwheel and there's a theoretical risk of ABI mismatches, in practice it seems to work reliably — at least for now.

Happy to help with further testing if needed!

[wojiushixiaobai]

• wojiushixiaobai@143f73d
• https://github.com/wojiushixiaobai/pypdfium2/actions/runs/16806192275

Wheel	Size	Build identifier	Time
pypdfium2-5.0.0b2-py3-none-manylinux_2_28_x86_64.whl	4.8 MB	cp38-manylinux_x86_64	4m
pypdfium2-5.0.0b2-py3-none-manylinux_2_28_ppc64le.whl	5.5 MB	cp38-manylinux_ppc64le	1h 25m
pypdfium2-5.0.0b2-py3-none-manylinux_2_28_s390x.whl	5.1 MB	cp38-manylinux_s390x	1h 27m
pypdfium2-5.0.0b2-py3-none-manylinux_2_28_aarch64.whl	4.8 MB	cp38-manylinux_aarch64	6m

[wojiushixiaobai]

• https://github.com/wojiushixiaobai/pypdfium2/actions/runs/16837457314/job/47700332928
• --> Add RedHat patches for big-endian CPUs <--

 FAILED: obj/core/fxcrt/fxcrt/cfx_seekablestreamproxy.o 
  g++ -MD -MF obj/core/fxcrt/fxcrt/cfx_seekablestreamproxy.o.d -DUSE_UDEV -DUSE_AURA=1 -DUSE_OZONE=1 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_NONE -D_GLIBCXX_ASSERTIONS=1 -DCOMPONENT_BUILD -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DABSL_CONSUME_DLL -DABSL_FLAGS_STRIP_NAMES=0 -DU_USING_ICU_NAMESPACE=0 -DU_ENABLE_DYLOAD=0 -DUSE_CHROMIUM_ICU=1 -DU_ENABLE_TRACING=1 -DU_ENABLE_RESOURCE_TRACING=0 -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_FILE -I../.. -Igen -I../../third_party/abseil-cpp -I../../third_party/icu/source/common -I../../third_party/icu/source/i18n -I../../third_party/fast_float/src/include -Wall -Wno-unused-local-typedefs -Wno-maybe-uninitialized -Wno-deprecated-declarations -Wno-comments -Wno-packed-not-aligned -Wno-missing-field-initializers -Wno-unused-parameter -Wno-psabi -fno-strict-overflow -fno-ident -fno-strict-aliasing -fstack-protector -funwind-tables -fPIC -pipe -pthread -ffp-contract=off -m64 -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -O2 -fdata-sections -ffunction-sections -fno-math-errno -fno-omit-frame-pointer -g0 -fvisibility=hidden -Wnarrowing -Wno-return-type -isystem/usr/include/freetype2 -isystem/usr/include/libpng16 -Wno-narrowing -Wno-class-memaccess -Wno-invalid-offsetof -std=gnu++2a -fno-exceptions -fno-rtti -fvisibility-inlines-hidden  -c ../../core/fxcrt/cfx_seekablestreamproxy.cpp -o obj/core/fxcrt/fxcrt/cfx_seekablestreamproxy.o
  ../../core/fxcrt/cfx_seekablestreamproxy.cpp: In member function ‘size_t CFX_SeekableStreamProxy::ReadBlock(pdfium::span<wchar_t, 18446744073709551615, wchar_t*>)’:
  ../../core/fxcrt/cfx_seekablestreamproxy.cpp:204:9: error: ‘m_wCodePage’ was not declared in this scope; did you mean ‘GetCodePage’?
    204 |     if (m_wCodePage == FX_CodePage::kUTF16LE) {
        |         ^~~~~~~~~~~
        |         GetCodePage

[mara004]

Oh yes, that's a copy paste error in the big-endian clause, thank you for noticing!
m_wCodePage has been renamed to code_page_. I did it in the little-endian clause and compiled/tested it, but the other part was unreached.
Will fix this soon, sorry for the inconvenience.

[mara004]

Should be fixed now I hope, cf 5d9471f