Skip to content

Fix MD5 hash mismatch on big-endian systems (s390x)#177

Open
Anemptyship wants to merge 3 commits intoAcademySoftwareFoundation:mainfrom
Anemptyship:main
Open

Fix MD5 hash mismatch on big-endian systems (s390x)#177
Anemptyship wants to merge 3 commits intoAcademySoftwareFoundation:mainfrom
Anemptyship:main

Conversation

@Anemptyship
Copy link
Contributor

Fixes #149

Summary

Fix test failures on s390x (big-endian) architecture where MD5 hash verification failed with hash:mismatch errors.

Root Cause

The existing MD5 implementation in src/oapv_util.c was not endian-safe. It cast byte arrays directly to u32* pointers, causing data to be interpreted in host byte order instead of the little-endian order required by the MD5 specification (RFC 1321).

Changes

  • md5_trans(): Accept const u8 * and decode bytes to u32 words explicitly in little-endian order.
  • Optimization: Use #ifdef __BYTE_ORDER__ to preserve zero-copy performance on little-endian systems (x86/ARM).
  • md5_finish(): Write bit length and digest in little-endian byte order.
  • md5_update_16(): Safe chunked processing for 16-bit pixel data.

Testing

Platform Arch Result
IBM LinuxONE s390x ✅ 100% passed (16/16)

@Anemptyship Anemptyship marked this pull request as draft January 9, 2026 05:25
@Anemptyship Anemptyship marked this pull request as ready for review January 9, 2026 05:25
- Make md5_trans() endian-safe by explicitly decoding bytes to u32 words
- Use compile-time check to preserve zero-copy performance on little-endian
- Fix md5_finish() to write length and digest in little-endian byte order
- Add md5_update_16() for safe 16-bit pixel data processing

Signed-off-by: Anemptyship <hanbin1931@gmail.com>
@sharkcz
Copy link

sharkcz commented Jan 9, 2026

successfully tested on my s390x system, thanks

@rathann
Copy link
Contributor

rathann commented Jan 13, 2026

Tested on Fedora rawhide (44) on our s390x builders, works fine!

- oapv_port.h: Introduce cross-platform endianness swap and conversion macros (OAPV_SWAP*, OAPV_LE32_TO_CPU).

- oapv_util.c: Replace manual byte swapping with new macros in MD5 transform to improve readability and maintainability.

- Standardize endianness handling across the codebase.

Signed-off-by: Anemptyship <hanbin1931@gmail.com>
- Add cross-compilation job for s390x using QEMU

- Increase test timeouts in CMakeLists.txt for emulated environment

Signed-off-by: Anemptyship <hanbin1931@gmail.com>
@Anemptyship
Copy link
Contributor Author

Refactor Endianness Handling

  • Added cross-platform endian swap macros (OAPV_SWAP*, OAPV_LE32_TO_CPU, etc.) in oapv_port.h.

CI Update

  • Added a GitHub Actions job (build-linux-s390x) to cross-compile and test on s390x using QEMU.
  • Increased test timeouts in "CMakeLists.txt" to accommodate the slower emulation environment.

@Anemptyship
Copy link
Contributor Author

@kpchoi @dariusz-f
Hi, Could you please review this PR? Let me know if any changes are needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tests fail on s390x (big-endian) since 0.2.0.2

3 participants