trino container failing early due to "ERROR: already running as 43"

[maxgruber19]

I'm currently upgrading from 24.11.0 (451) to 25.3.0 (470) and are facing issues with the trino coordinators and workers not booting up because of an early error. Do you know that already? Maybe i misconfigured something, didnt change my cr configs during upgrade because I didnt find something breaking in the breaking changes of the release note

log so far:

+ echo copying /stackable/config to /stackable/rwconfig
copying /stackable/config to /stackable/rwconfig
+ cp -RL /stackable/config/access-control.properties /stackable/config/catalog /stackable/config/config.properties /stackable/config/jvm.config /stackable/config/keytab /stackable/config/krb5 /stackable/config/log.properties /stackable/config/mesh-platform-core-trino-password-ldap-auth.properties /stackable/config/node.properties /stackable/config/security.properties /stackable/rwconfig
+ set +x
+ set +x
+ rm -f /stackable/log/_vector/shutdown
+ prepare_signal_handlers
+ unset term_child_pid
+ unset term_kill_needed
+ trap handle_term_signal TERM
+ containerdebug --output=/stackable/log/containerdebug-state.json --loop
+ wait_for_termination 44
+ bin/launcher run --etc-dir=/stackable/rwconfig --data-dir=/stackable/data
+ set +e
+ term_child_pid=44
+ [[ -v term_kill_needed ]]
+ wait 44
2025-08-22T07:26:51.601292Z  INFO stackable_operator::logging: file logging enabled directory=/stackable/log/containerdebug
2025-08-22T07:26:51.601354Z  INFO containerdebug: stackable_operator::utils::logging: Starting
2025-08-22T07:26:51.601372Z  INFO containerdebug: stackable_operator::utils::logging: This is version 0.1.1 (Git information: 0.1.1), built for x86_64-unknown-linux-gnu by rustc 1.84.1 (e71f9a9a9 2025-01-27) at Tue, 25 Mar 2025 09:25:11 +0000
2025-08-22T07:26:51.601382Z  INFO containerdebug:SystemInformation::init: containerdebug::system_information: initializing
2025-08-22T07:26:51.602952Z  INFO containerdebug:SystemInformation::init: containerdebug::system_information: init finished
2025-08-22T07:26:51.602998Z  INFO containerdebug:SystemInformation::collect: containerdebug::system_information: Starting data collection
2025-08-22T07:26:51.603394Z  INFO containerdebug:SystemInformation::collect:Resources::collect: containerdebug::system_information::resources: cpus cpus.physical=8 cpus.cores.physical=8
2025-08-22T07:26:51.603417Z  INFO containerdebug:SystemInformation::collect:Resources::collect: containerdebug::system_information::resources: memory memory.total=100781862912 memory.free=70234050560 memory.available=86375415808 memory.used=14406447104
2025-08-22T07:26:51.603440Z  INFO containerdebug:SystemInformation::collect:Resources::collect: containerdebug::system_information::resources: swap swap.total=0 swap.free=0 swap.used=0
2025-08-22T07:26:51.603785Z  INFO containerdebug:SystemInformation::collect:Resources::collect: containerdebug::system_information::resources: cgroup memory cgroup.memory.total=10737418240 cgroup.memory.free=10733142016 cgroup.swap.free=0
2025-08-22T07:26:51.603976Z  INFO containerdebug:SystemInformation::collect:OperatingSystem::collect: containerdebug::system_information::os: operating system os.name="Red Hat Enterprise Linux" os.kernel.version="4.18.0-553.66.1.el8_10.x86_64" os.version="Linux 9.5 Red Hat Enterprise Linux" os.host_name="trino-primary-coordinator-default-0" os.cpu_arch="x86_64"
2025-08-22T07:26:51.604041Z  INFO containerdebug:SystemInformation::collect:User::collect_current: containerdebug::system_information::user: current user user.name="stackable" user.uid="Uid(1000)" user.gid="Uid(1000)"
2025-08-22T07:26:51.606374Z  INFO containerdebug:SystemInformation::collect:Disk::collect_all: containerdebug::system_information::disk: found disk disk.mount_point="/" disk.name="overlay" disk.space.total=214643507200 disk.space.available=171333398528
2025-08-22T07:26:51.606417Z  INFO containerdebug:SystemInformation::collect:Disk::collect_all: containerdebug::system_information::disk: found disk disk.mount_point="/stackable/server_tls" disk.name="/dev/mapper/vg_root-lv_var" disk.space.total=107363696640 disk.space.available=105067081728
2025-08-22T07:26:51.606434Z  INFO containerdebug:SystemInformation::collect:Disk::collect_all: containerdebug::system_information::disk: found disk disk.mount_point="/stackable/client_tls" disk.name="/dev/mapper/vg_root-lv_var" disk.space.total=107363696640 disk.space.available=105067081728
2025-08-22T07:26:51.606447Z  INFO containerdebug:SystemInformation::collect:Disk::collect_all: containerdebug::system_information::disk: found disk disk.mount_point="/stackable/internal_tls" disk.name="/dev/mapper/vg_root-lv_var" disk.space.total=107363696640 disk.space.available=105067081728
2025-08-22T07:26:51.606458Z  INFO containerdebug:SystemInformation::collect:Disk::collect_all: containerdebug::system_information::disk: found disk disk.mount_point="/stackable/data" disk.name="rmcestest-k8sdev01" disk.space.total=1073741824 disk.space.available=1073741824
2025-08-22T07:26:51.606468Z  INFO containerdebug:SystemInformation::collect:Disk::collect_all: containerdebug::system_information::disk: found disk disk.mount_point="/stackable/config" disk.name="/dev/mapper/vg_root-lv_var" disk.space.total=107363696640 disk.space.available=105067081728
2025-08-22T07:26:51.606482Z  INFO containerdebug:SystemInformation::collect:Disk::collect_all: containerdebug::system_information::disk: found disk disk.mount_point="/stackable/rwconfig" disk.name="/dev/mapper/vg_root-lv_var" disk.space.total=107363696640 disk.space.available=105067081728
2025-08-22T07:26:51.606492Z  INFO containerdebug:SystemInformation::collect:Disk::collect_all: containerdebug::system_information::disk: found disk disk.mount_point="/stackable/log" disk.name="/dev/mapper/vg_root-lv_var" disk.space.total=107363696640 disk.space.available=105067081728
2025-08-22T07:26:51.606502Z  INFO containerdebug:SystemInformation::collect:Disk::collect_all: containerdebug::system_information::disk: found disk disk.mount_point="/etc/hosts" disk.name="/dev/mapper/vg_root-lv_var" disk.space.total=107363696640 disk.space.available=105067081728
2025-08-22T07:26:51.606515Z  INFO containerdebug:SystemInformation::collect:Disk::collect_all: containerdebug::system_information::disk: found disk disk.mount_point="/dev/termination-log" disk.name="/dev/mapper/vg_root-lv_var" disk.space.total=107363696640 disk.space.available=105067081728
2025-08-22T07:26:51.606525Z  INFO containerdebug:SystemInformation::collect:Disk::collect_all: containerdebug::system_information::disk: found disk disk.mount_point="/etc/hostname" disk.name="/dev/mapper/vg_k8s-lv_k8s_data" disk.space.total=214643507200 disk.space.available=171333398528
2025-08-22T07:26:51.606537Z  INFO containerdebug:SystemInformation::collect:Disk::collect_all: containerdebug::system_information::disk: found disk disk.mount_point="/etc/resolv.conf" disk.name="/dev/mapper/vg_k8s-lv_k8s_data" disk.space.total=214643507200 disk.space.available=171333398528
2025-08-22T07:26:51.606547Z  INFO containerdebug:SystemInformation::collect:Disk::collect_all: containerdebug::system_information::disk: found disk disk.mount_point="/stackable/config/catalog" disk.name="/dev/mapper/vg_root-lv_var" disk.space.total=107363696640 disk.space.available=105067081728
2025-08-22T07:26:51.606557Z  INFO containerdebug:SystemInformation::collect:Disk::collect_all: containerdebug::system_information::disk: found disk disk.mount_point="/stackable/config/krb5" disk.name="/dev/mapper/vg_root-lv_var" disk.space.total=107363696640 disk.space.available=105067081728
2025-08-22T07:26:51.606567Z  INFO containerdebug:SystemInformation::collect:Disk::collect_all: containerdebug::system_information::disk: found disk disk.mount_point="/stackable/rwconfig/resource-groups.json" disk.name="/dev/mapper/vg_root-lv_var" disk.space.total=107363696640 disk.space.available=105067081728
2025-08-22T07:26:51.606577Z  INFO containerdebug:SystemInformation::collect:Disk::collect_all: containerdebug::system_information::disk: found disk disk.mount_point="/stackable/rwconfig/resource-groups.properties" disk.name="/dev/mapper/vg_root-lv_var" disk.space.total=107363696640 disk.space.available=105067081728
2025-08-22T07:26:51.607065Z  INFO containerdebug:SystemInformation::collect:SystemNetworkInfo::collect: containerdebug::system_information::network: found network interface network.interface.name="lo" network.interface.address=127.0.0.1
2025-08-22T07:26:51.607107Z  INFO containerdebug:SystemInformation::collect:SystemNetworkInfo::collect: containerdebug::system_information::network: found network interface network.interface.name="eth0" network.interface.address=192.168.6.154
2025-08-22T07:26:51.607119Z  INFO containerdebug:SystemInformation::collect:SystemNetworkInfo::collect: containerdebug::system_information::network: found network interface network.interface.name="lo" network.interface.address=::1
2025-08-22T07:26:51.607130Z  INFO containerdebug:SystemInformation::collect:SystemNetworkInfo::collect: containerdebug::system_information::network: found network interface network.interface.name="eth0" network.interface.address=fe80::f811:3bff:fe50:37dd
2025-08-22T07:26:51.607146Z  INFO containerdebug:SystemInformation::collect:SystemNetworkInfo::collect: containerdebug::system_information::network: ip addresses network.addresses.ip={127.0.0.1, 192.168.6.154, ::1, fe80::f811:3bff:fe50:37dd}
2025-08-22T07:26:51.607198Z  INFO containerdebug:SystemInformation::collect:SystemNetworkInfo::collect: containerdebug::system_information::network: performed reverse DNS lookup for IP ip=127.0.0.1 hostnames=["localhost."]
2025-08-22T07:26:51.608300Z  INFO containerdebug:SystemInformation::collect:SystemNetworkInfo::collect: containerdebug::system_information::network: performed reverse DNS lookup for IP ip=192.168.6.154 hostnames=["trino-primary-coordinator-default-0.trino-primary-coordinator-default.mesh-platform-core.svc.cluster.local."]
2025-08-22T07:26:51.608355Z  INFO containerdebug:SystemInformation::collect:SystemNetworkInfo::collect: containerdebug::system_information::network: performed reverse DNS lookup for IP ip=::1 hostnames=["localhost."]
2025-08-22T07:26:51.610558Z  WARN containerdebug:SystemInformation::collect:SystemNetworkInfo::collect: containerdebug::system_information::network: reverse DNS lookup failed ip=fe80::f811:3bff:fe50:37dd error=no record found for Query { name: Name("d.d.7.3.0.5.e.f.f.f.b.3.1.1.8.f.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa."), query_type: PTR, query_class: IN }
2025-08-22T07:26:51.610610Z  INFO containerdebug:SystemInformation::collect:SystemNetworkInfo::collect: containerdebug::system_information::network: hostnames network.addresses.hostname={"localhost.", "trino-primary-coordinator-default-0.trino-primary-coordinator-default.mesh-platform-core.svc.cluster.local."}
2025-08-22T07:26:51.610645Z  INFO containerdebug:SystemInformation::collect:SystemNetworkInfo::collect: containerdebug::system_information::network: performed forward DNS lookup for hostname hostname="localhost." ips=[127.0.0.1]
2025-08-22T07:26:51.611555Z  INFO containerdebug:SystemInformation::collect:SystemNetworkInfo::collect: containerdebug::system_information::network: performed forward DNS lookup for hostname hostname="trino-primary-coordinator-default-0.trino-primary-coordinator-default.mesh-platform-core.svc.cluster.local." ips=[192.168.6.154]
2025-08-22T07:26:51.611630Z  INFO containerdebug:SystemInformation::collect: containerdebug::system_information: Data collection finished
2025-08-22T07:26:51.611912Z  INFO containerdebug: containerdebug: scheduling next run... next_run=Instant { tv_sec: 2883, tv_nsec: 10075639 }
ERROR: already running as 43
+ trap - TERM
+ wait 44
+ set -e
+ mkdir -p /stackable/log/_vector
+ touch /stackable/log/_vector/shutdown

[sbernauer]

Hi @maxgruber19!
This bothered me for years and I finally took the time to fix this in #768. It's fixed in SDP 25.7 and there is a documented workaround in the linked issue.
Hope that helps!

[maxgruber19]

That answer was fast as hell and even better than hell. Cluster up and running again. Thank you @sbernauer as usual

I'm very looking forward to get rid of the pvcs, unfortunately it seems like upgrading to 25.7.0 is not safe yet according to your release notes 😄 I'll probably wait until that keytool thing is fixed

[sbernauer]

We are working on it, but the thing is that the keytool problem exists since the early days of the platform...
So SDP 25.7 isn't any worse than 25.3, feel free to update anyway.

[Techassi]

We noticed that our 25.7.0 release notes don't mention the removal/fix. It is added in stackabletech/documentation#776.

[maxgruber19]

Thats a good piece of information @sbernauer, just kicked off a request to connect to your new oci registry. Will check on that soon. 470 is working perfectly now. And because we talked in the past: I'm excited to see how the new s3 client handles production workloads. Thank you once again, I'll close that issue

[maxgruber19]

There is one more breaking thing when upgrading to 25.7.0, it seems like the trino coordinator service name changed to trino-coordinator from trino-coordinator-default, I think that should be part of the docs somewhere. maybe i just didnt notice :-D

[sbernauer]

Yeah, that is part of the listener-operator rollout we did across many products: https://docs.stackable.tech/home/stable/release-notes/#_listener_integration. You might also need to delete StatefulSets

[maxgruber19]

@sbernauer oh yeah I might have skipped reading that when upgrading.Thanks for the hint!