Skip to content

s390x binaries included in librdkafka.redist NuGet package and static builds package #5365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pranav shah (prashah-confluent) merged 15 commits into from
Apr 9, 2026
Merged

s390x binaries included in librdkafka.redist NuGet package and static builds package #5365

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
8d12fc8
Add support for passing platform flag
prashah-confluent Mar 18, 2026
870fa87
Add linux s390x block to .semaphore.yml
prashah-confluent Mar 18, 2026
1cc170d
Add s390x linux nuget mappings
prashah-confluent Mar 18, 2026
5acff35
Add s390 linux static mapping
prashah-confluent Mar 18, 2026
f4e58aa
Comment out alpine flavour
prashah-confluent Mar 19, 2026
2bbfc93
Set GIT_TAG_NAME to workflow id for local testing
prashah-confluent Mar 19, 2026
3c044f2
Fix version string format for PR build testing
prashah-confluent Mar 20, 2026
f228eaf
Comment out tag checks in semaphore
prashah-confluent Mar 21, 2026
664b316
Add perl-bignum dependency for s390x
prashah-confluent Mar 18, 2026
8d80e87
Fix setting git tag name
prashah-confluent Mar 21, 2026
7b62aab
Nugetpackage tag conversion temporary fix
prashah-confluent Mar 21, 2026
74794d5
Fix Git tag name for documentation
prashah-confluent Mar 22, 2026
b96777c
Restore condition to run the packaging only on tags
emasab Mar 26, 2026
15165b8
librdkafka version 2.14.1
emasab Mar 26, 2026
66085fb
Remove commented blocks for alpine
prashah-confluent Apr 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions .semaphore/semaphore.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,36 @@ blocks:
commands:
- packaging/tools/build-release-artifacts.sh --disable-gssapi alpine:3.16.9 artifacts/librdkafka.tgz

- name: 'Linux s390x: release artifact docker builds'
dependencies: []
run:
when: "tag =~ '^v[0-9]\\.'"
task:
agent:
machine:
type: s1-prod-ubuntu24-04-amd64-1
prologue:
commands:
- '[[ -z $DOCKERHUB_APIKEY ]] || docker login --username $DOCKERHUB_USER --password $DOCKERHUB_APIKEY'
- docker run --privileged --rm tonistiigi/binfmt --install s390x
Copy link

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The workflow installs QEMU binfmt using tonistiigi/binfmt without a tag or digest, which makes the build non-reproducible and risks sudden CI breakage if the upstream image changes. Pin this image to a specific version tag or digest (similar to how the manylinux images are pinned).

Suggested change
- docker run --privileged --rm tonistiigi/binfmt --install s390x
- docker run --privileged --rm tonistiigi/binfmt:qemu-v8.1.5 --install s390x

Copilot uses AI. Check for mistakes.
epilogue:
commands:
- '[[ -z $SEMAPHORE_GIT_TAG_NAME ]] || artifact push workflow artifacts/ --destination artifacts/${ARTIFACT_KEY}/'
jobs:
- name: 'Build: centos8 glibc +gssapi'
env_vars:
- name: ARTIFACT_KEY
value: p-librdkafka__plat-linux__dist-centos8__arch-s390x__lnk-std__extra-gssapi
commands:
- packaging/tools/build-release-artifacts.sh --platform linux/s390x quay.io/pypa/manylinux_2_28_s390x:2024.07.01-1 artifacts/librdkafka.tgz

- name: 'Build: centos8 glibc'
env_vars:
- name: ARTIFACT_KEY
value: p-librdkafka__plat-linux__dist-centos8__arch-s390x__lnk-all
commands:
- packaging/tools/build-release-artifacts.sh --disable-gssapi --platform linux/s390x quay.io/pypa/manylinux_2_28_s390x:2024.07.01-1 artifacts/librdkafka.tgz


- name: 'Windows x64: MinGW-w64'
dependencies: []
Expand Down Expand Up @@ -313,6 +343,7 @@ blocks:
- 'OSX x64'
- 'Linux x64: release artifact docker builds'
- 'Linux arm64: release artifact docker builds'
- 'Linux s390x: release artifact docker builds'
- 'Windows x64: MinGW-w64'
- 'Windows x64: Windows SDK 10.0 / MSVC v142 / VS 2019'
run:
Expand Down
8 changes: 8 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,11 @@
# librdkafka v2.14.1

librdkafka v2.14.1 is a maintenance release:

* Bundle prebuilt binaries for linux-s390x (#5365).



# librdkafka v2.14.0

librdkafka v2.14.0 is a feature release:
Expand Down
8 changes: 8 additions & 0 deletions packaging/nuget/nugetpackage.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,14 @@ class NugetPackage (Package):
'librdkafka.tgz',
'./usr/local/lib/librdkafka.so.1',
'runtimes/linux-arm64/native/librdkafka.so'),
# Linux glibc centos8 s390x without GSSAPI (no external deps)
Mapping({'arch': 's390x',
'plat': 'linux',
'dist': 'centos8',
'lnk': 'all'},
'librdkafka.tgz',
'./usr/local/lib/librdkafka.so.1',
'runtimes/linux-s390x/native/librdkafka.so'),
Comment on lines +94 to +101
Copy link

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This adds a linux/s390x runtime mapping, but packaging/nuget/packaging.py's magic_patterns currently has ELF checks for linux x64 and arm64 only. That means the s390x .so won't be validated for architecture during packaging, reducing defense-in-depth against accidentally bundling the wrong binary. Consider adding a corresponding magic_patterns entry for (linux, s390x, .so).

Copilot uses AI. Check for mistakes.

# Linux musl alpine x64 without GSSAPI (no external deps)
Mapping({'arch': 'x64',
Expand Down
18 changes: 18 additions & 0 deletions packaging/nuget/staticpackage.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,24 @@ class StaticPackage (Package):
'./usr/local/lib/pkgconfig/rdkafka-static.pc',
'librdkafka_glibc_linux_arm64.pc'),

# glibc linux s390x static lib and pkg-config file
Mapping({'arch': 's390x',
'plat': 'linux',
'dist': 'centos8',
'lnk': 'all',
'!extra': 'gssapi'},
'librdkafka.tgz',
'./usr/local/lib/librdkafka-static.a',
'librdkafka_glibc_linux_s390x.a'),
Mapping({'arch': 's390x',
'plat': 'linux',
'dist': 'centos8',
'lnk': 'all',
'!extra': 'gssapi'},
'librdkafka.tgz',
'./usr/local/lib/pkgconfig/rdkafka-static.pc',
'librdkafka_glibc_linux_s390x.pc'),

# musl linux static lib and pkg-config file
Mapping({'arch': 'x64',
'plat': 'linux',
Expand Down
15 changes: 11 additions & 4 deletions packaging/tools/build-release-artifacts.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,10 @@
# alpine:3.16
# quay.io/pypa/manylinux_2_28_aarch64 (centos8)
# quay.io/pypa/manylinux_2_28_x86_64 (centos8)
# quay.io/pypa/manylinux_2_28_s390x (centos8, requires QEMU binfmt)
#
# Usage:
# packaging/tools/build-release-artifacts.sh [--disable-gssapi] <docker-image> <relative-output-tarball-path.tgz>
# packaging/tools/build-release-artifacts.sh [--disable-gssapi] [--platform <docker-platform>] <docker-image> <relative-output-tarball-path.tgz>
#
# The output path must be a relative path and inside the librdkafka directory
# structure.
Expand Down Expand Up @@ -42,20 +43,26 @@ else
disable_gssapi=""
fi

docker_platform=""
if [ "$1" = "--platform" ]; then
Copy link

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--platform is parsed as --platform=$2 and then shift 2 without validating that a value was actually provided. If someone calls the script with --platform but no argument, this will fail with a generic shift error rather than a clear usage message. Add an explicit check (e.g., ensure $# -ge 2 and $2 is non-empty) and print usage/exit when the argument is missing.

Suggested change
if [ "$1" = "--platform" ]; then
if [ "$1" = "--platform" ]; then
if [ $# -lt 2 ] || [ -z "$2" ]; then
echo "Usage: $0 [--disable-gssapi] [--platform <docker-platform>] <manylinux-docker-image> <output-path.tgz>"
exit 1
fi

Copilot uses AI. Check for mistakes.
docker_platform="--platform=$2"
shift 2
fi

# Check if we're running on the host or the (docker) build target.
if [ "$1" = "--in-docker" -a $# -eq 2 ]; then
output="$2"
elif [ $# -eq 2 ]; then
docker_image="$1"
output="$2"
else
echo "Usage: $0 [--disable-gssapi] <manylinux-docker-image> <output-path.tgz>"
echo "Usage: $0 [--disable-gssapi] [--platform <docker-platform>] <manylinux-docker-image> <output-path.tgz>"
exit 1
fi

if [ -n "$docker_image" ]; then
# Running on the host, spin up the docker builder.
exec docker run -v "$PWD:/v" $docker_image /v/packaging/tools/build-release-artifacts.sh $disable_gssapi --in-docker "/v/$output"
exec docker run $docker_platform -v "$PWD:/v" $docker_image /v/packaging/tools/build-release-artifacts.sh $disable_gssapi --in-docker "/v/$output"
# Only reached on exec error
exit $?
fi
Expand All @@ -76,7 +83,7 @@ if grep -q alpine /etc/os-release 2>/dev/null ; then

else
# CentOS
yum install -y libstdc++-devel gcc gcc-c++ python3 git perl-IPC-Cmd perl-Pod-Html $extra_pkgs_rpm
yum install -y libstdc++-devel gcc gcc-c++ python3 git perl-IPC-Cmd perl-Pod-Html perl-bignum $extra_pkgs_rpm
fi


Expand Down
2 changes: 1 addition & 1 deletion src-cpp/rdkafkacpp.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ namespace RdKafka {
* @remark This value should only be used during compile time,
* for runtime checks of version use RdKafka::version()
*/
#define RD_KAFKA_VERSION 0x020e00ff
#define RD_KAFKA_VERSION 0x020e01ff

/**
* @brief Returns the librdkafka version as integer.
Expand Down
2 changes: 1 addition & 1 deletion src/rdkafka.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ typedef SSIZE_T ssize_t;
* @remark This value should only be used during compile time,
* for runtime checks of version use rd_kafka_version()
*/
#define RD_KAFKA_VERSION 0x020e00ff
#define RD_KAFKA_VERSION 0x020e01ff

/**
* @brief Returns the librdkafka version as integer.
Expand Down
2 changes: 1 addition & 1 deletion vcpkg.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "librdkafka",
"version": "2.14.0",
"version": "2.14.1",
"dependencies": [
{
"name": "zstd",
Expand Down