Constantrly getting messages about stalled reactor.

[hoxnox]

I'm constantly getting messages "Reactor stalled" on one of the cluster nodes. Hot to troubleshoot the problem? Resources amount is the same as in other hosts, but errors are only on this one.

Dec 08 11:00:34 host5 rpk[52221]: Reactor stalled for 52 ms, backtrace omitted (uncaught exception in progress)
Dec 08 11:00:35 host5 rpk[52221]: INFO  2025-12-08 11:00:35,512 [shard 0:main] rpc - transport.cc:215 - RPC timeout (100 ms) to {host: m2kfkne.srv, port: 33145}, method: node_status_rpc::node_status, correlation id: 6470, 1 in flight, time since: {init: 100 ms, enqueue: 100 ms, memory_reserved: 100 ms dispatch: 100 ms, written: 100 ms}, flushed: true
Dec 08 11:00:35 host5 rpk[52221]: INFO  2025-12-08 11:00:35,512 [shard 0:main] rpc - transport.cc:215 - RPC timeout (100 ms) to {host: m1kfkne.srv, port: 33145}, method: node_status_rpc::node_status, correlation id: 6462, 1 in flight, time since: {init: 100 ms, enqueue: 100 ms, memory_reserved: 100 ms dispatch: 100 ms, written: 100 ms}, flushed: true
Dec 08 11:01:25 host5 rpk[52221]: INFO  2025-12-08 11:01:25,294 [shard 0:rr  ] storage - segment.cc:847 - Creating new segment /var/lib/redpanda/data/redpanda/controller/0_0/795-81-v1.log
Dec 08 11:02:25 host5 rpk[52221]: INFO  2025-12-08 11:02:25,503 [shard 0:clus] cluster - controller_stm.cc:141 - created snapshot at offset 795 in 0 ms
Dec 08 11:02:25 host5 rpk[52221]: INFO  2025-12-08 11:02:25,838 [shard 0:clus] storage - disk_log_impl.cc:3237 - Removing "/var/lib/redpanda/data/redpanda/controller/0_0/795-81-v1.log" (remove_prefix_full_segments, {offset_tracker:{term:81, base_offset:795, committed_offset:795, stable_offset:795, dirty_offset:795}, compacted_segment=0, finished_self_compaction=0, finished_windowed_compaction=0, generation=2, reader={/var/lib/redpanda/data/redpanda/controller/0_0/795-81-v1.log, (68 bytes)}, writer={closed:false, fallocation_offset:33554432, stable_offset:68, flushed_offset:68, committed_offset:68, inflight: 0, bytes_flush_pending:0, stats: { merged_writes: 0, bytes_requested : 68, bytes_written : 4096, appends : 2, flushes: 1, fsyncs: 1, truncates: 0, fallocations: 1, last_page_hydrations: 0}}, cache={cache_size=1, dirty tracker: {min: -9223372036854775808, max: -9223372036854775808}}, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/controller/0_0/795-81-v1.base_index, offsets:795, index:{header_bitflags:0, base_offset:795, max_offset:795, base_timestamp:{timestamp: 1765191685290}, max_timestamp:{timestamp: 1765191685290}, batch_timestamps_are_monotonic:1, with_offset:true, non_data_timestamps:0, broker_timestamp:{{timestamp: 1765191685296}}, num_compactible_records_appended:{1}, clean_compact_timestamp:{nullopt}, may_have_tombstone_records:1, self_compact_timestamp:{nullopt}, may_have_transaction_batches:1, index(1, 1, 1)}, step:32768, needs_persistence:1}})
Dec 08 11:05:40 host5 rpk[52221]: Reactor stalled for 50 ms on shard 3, in scheduling group main. Backtrace: 0xb16328c 0xb164395 0x4251f 0x1d11e43 0xa8b8de9 0xaa3f3fb 0xa83671d 0xa837ef2 0xb18812e 0xb18e238 0xb1e5c04 0xb117d22 0x94ac2 0x12684f
Dec 08 11:06:11 host5 rpk[52221]: Reactor stalled for 56 ms on shard 2, in scheduling group main. Backtrace: 0xb16328c 0xb164395 0x4251f 0x1148be 0xb1937b5 0xb16fa2c 0xb170361 0xaa3f437 0xa83671d 0xa837ef2 0xb18812e 0xb18e238 0xb1e5c04 0xb117d22 0x94ac2 0x12684f

[dotnwat]

What version are you running? x86 or arm?

[noxsyc]

x86-64

[dotnwat]

What version of Redpanda are you running? I need both the version and architecture in order to decode the backtraces. I realize my first question sounded like I was just asking for the architecture.

[hoxnox]

What version of Redpanda are you running? I need both the version and architecture in order to decode the backtraces. I realize my first question sounded like I was just asking for the architecture.

rpk version: v25.3.1
Git ref: 9cd8d8c
Build date: 2025 Nov 19 16 38 18 Wed
OS/Arch: linux/amd64
Go version: go1.24.3

[hoxnox]

And another question.
Is it normal to periodically getting...

storage - segment.cc:847 - Creating new segment /var/lib/redpanda/data/redpanda/controller/0_0/795-81-v1.log

and right after that

storage - disk_log_impl.cc:3237 - Removing "/var/lib/redpanda/data/redpanda/controller/0_0/795-81-v1.log"

[dotnwat]

Yes, those message are normal.

[dotnwat]

decoded reactor stalls

/home/nwatkins/.cache/redpanda-syms/amd64/25.3.1-1-amd64/opt/redpanda/libexec/redpanda: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /opt/redpanda/lib/ld.so, for GNU/Linux 3.2.0, BuildID[md5/uuid]=e82b4b9520c0b5dc7f670e2f80ff6915, stripped

Dec 08 11:05:40 host5 rpk[52221]: Reactor stalled for 50 ms on shard 3, in scheduling group main. Backtrace:
[Backtrace #0]
seastar::guarded_backtrace(void**, int) at ./external/+non_module_dependencies+seastar/src/util/backtrace.cc:102
 (inlined by) void seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::'lambda'(seastar::frame)>(seastar::backtrace_buffer::append_backtrace_oneline()::'lambda'(seastar::frame)&&, bool) at ./external/+non_module_dependencies+seastar/include/seastar/util/backtrace.hh:101
 (inlined by) seastar::backtrace_buffer::append_backtrace_oneline() at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:793
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:815
seastar::internal::cpu_stall_detector::generate_trace() at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:1521
 (inlined by) seastar::internal::cpu_stall_detector::maybe_report() at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:1357
 (inlined by) seastar::internal::cpu_stall_detector::on_signal() at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:1377
 (inlined by) seastar::reactor::block_notifier(int) at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:1585
?? at ??:0
char const* fmt::v9::detail::parse_replacement_field<char, void fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::__1::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::__1::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type>>>::type, fmt::v9::type_identity<char>::type>>, fmt::v9::detail::locale_ref)::format_handler&>(char const*, char const*, void fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::__1::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::__1::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type>>>::type, fmt::v9::type_identity<char>::type>>, fmt::v9::detail::locale_ref)::format_handler&) at ./external/fmt+/include/fmt/core.h:2662
void fmt::v9::detail::parse_format_string<false, char, void fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::__1::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::__1::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type>>>::type, fmt::v9::type_identity<char>::type>>, fmt::v9::detail::locale_ref)::format_handler>(fmt::v9::basic_string_view<char>, void fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::__1::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::__1::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type>>>::type, fmt::v9::type_identity<char>::type>>, fmt::v9::detail::locale_ref)::format_handler&&) at ./external/fmt+/include/fmt/core.h:2688
 (inlined by) void fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::__1::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::__1::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type>>>::type, fmt::v9::type_identity<char>::type>>, fmt::v9::detail::locale_ref) at ./external/fmt+/include/fmt/format.h:4136
 (inlined by) fmt::v9::appender fmt::v9::vformat_to<fmt::v9::appender, 0>(fmt::v9::appender, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<fmt::v9::appender, char>>) at ./external/fmt+/include/fmt/core.h:3215
 (inlined by) fmt::v9::appender fmt::v9::format_to<fmt::v9::appender, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, detail::base_named_type<long, model::revision_id_model_type, std::__1::integral_constant<bool, true>> const&, long, long, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::basic_string_view<char, std::__1::char_traits<char>> const&, 0>(fmt::v9::appender, fmt::v9::basic_format_string<char, fmt::v9::type_identity<seastar::basic_sstring<char, unsigned int, 15u, true>>::type, fmt::v9::type_identity<seastar::basic_sstring<char, unsigned int, 15u, true>>::type, fmt::v9::type_identity<detail::base_named_type<long, model::revision_id_model_type, std::__1::integral_constant<bool, true>> const&>::type, fmt::v9::type_identity<long>::type, fmt::v9::type_identity<long>::type, fmt::v9::type_identity<seastar::basic_sstring<char, unsigned int, 15u, true>>::type, fmt::v9::type_identity<std::__1::basic_string_view<char, std::__1::char_traits<char>> const&>::type>, seastar::basic_sstring<char, unsigned int, 15u, true>&&, seastar::basic_sstring<char, unsigned int, 15u, true>&&, detail::base_named_type<long, model::revision_id_model_type, std::__1::integral_constant<bool, true>> const&, long&&, long&&, seastar::basic_sstring<char, unsigned int, 15u, true>&&, std::__1::basic_string_view<char, std::__1::char_traits<char>> const&) at ./external/fmt+/include/fmt/core.h:3235
 (inlined by) seastar::basic_sstring<char, unsigned int, 15u, true> seastar::format<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, detail::base_named_type<long, model::revision_id_model_type, std::__1::integral_constant<bool, true>> const&, long, long, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::basic_string_view<char, std::__1::char_traits<char>> const&>(fmt::v9::basic_format_string<char, fmt::v9::type_identity<seastar::basic_sstring<char, unsigned int, 15u, true>>::type, fmt::v9::type_identity<seastar::basic_sstring<char, unsigned int, 15u, true>>::type, fmt::v9::type_identity<detail::base_named_type<long, model::revision_id_model_type, std::__1::integral_constant<bool, true>> const&>::type, fmt::v9::type_identity<long>::type, fmt::v9::type_identity<long>::type, fmt::v9::type_identity<seastar::basic_sstring<char, unsigned int, 15u, true>>::type, fmt::v9::type_identity<std::__1::basic_string_view<char, std::__1::char_traits<char>> const&>::type>, seastar::basic_sstring<char, unsigned int, 15u, true>&&, seastar::basic_sstring<char, unsigned int, 15u, true>&&, detail::base_named_type<long, model::revision_id_model_type, std::__1::integral_constant<bool, true>> const&, long&&, long&&, seastar::basic_sstring<char, unsigned int, 15u, true>&&, std::__1::basic_string_view<char, std::__1::char_traits<char>> const&) at ./external/+non_module_dependencies+seastar/include/seastar/core/format.hh:44
 (inlined by) storage::segment_full_path::string() const at ./src/v/storage/fs_utils.cc:24
storage::segment::persistent_size() at ./src/v/storage/segment.cc:153
storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()::operator()() const at ./src/v/storage/disk_log_impl.cc:4094
 (inlined by) seastar::future<storage::usage> seastar::futurize<seastar::future<storage::usage>>::invoke<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>(storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&) at ./external/+non_module_dependencies+seastar/include/seastar/core/future.hh:1959
 (inlined by) auto seastar::futurize_invoke<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>(storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&) at ./external/+non_module_dependencies+seastar/include/seastar/core/future.hh:1990
 (inlined by) auto seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)::operator()<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>(seastar::semaphore_default_exception_factory) at ./external/+non_module_dependencies+seastar/include/seastar/core/semaphore.hh:778
decltype(std::declval<seastar::semaphore_default_exception_factory>()(std::declval<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>())) std::__1::__invoke[abi:ne200100]<seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)&, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>(seastar::semaphore_default_exception_factory&&, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&&) at ./external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__type_traits/invoke.h:179
 (inlined by) std::__1::invoke_result<seastar::semaphore_default_exception_factory, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>::type std::__1::invoke[abi:ne200100]<seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)&, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>(seastar::semaphore_default_exception_factory&&, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&&) at ./external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__functional/invoke.h:29
 (inlined by) auto seastar::internal::future_invoke<seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)&, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>(seastar::semaphore_default_exception_factory&&, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&) at ./external/+non_module_dependencies+seastar/include/seastar/core/future.hh:1133
 (inlined by) storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'() seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>::then_impl_nrvo<seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory), seastar::future<storage::usage>>(seastar::semaphore_default_exception_factory&&)::'lambda'(seastar::internal::promise_base_with_type<storage::usage>&&, seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)&, seastar::future_state<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>&&)::operator()(seastar::internal::promise_base_with_type<storage::usage>&&, seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)&, seastar::future_state<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>&&) const::'lambda'()::operator()() const at ./external/+non_module_dependencies+seastar/include/seastar/core/future.hh:1420
 (inlined by) void seastar::futurize<seastar::future<storage::usage>>::satisfy_with_result_of<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'() seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>::then_impl_nrvo<seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory), seastar::future<storage::usage>>(seastar::semaphore_default_exception_factory&&)::'lambda'(seastar::internal::promise_base_with_type<storage::usage>&&, seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)&, seastar::future_state<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>&&)::operator()(seastar::internal::promise_base_with_type<storage::usage>&&, seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)&, seastar::future_state<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>&&) const::'lambda'()>(seastar::internal::promise_base_with_type<storage::usage>&&, seastar::semaphore_default_exception_factory&&) at ./external/+non_module_dependencies+seastar/include/seastar/core/future.hh:1944
 (inlined by) storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'() seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>::then_impl_nrvo<seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory), seastar::future<storage::usage>>(seastar::semaphore_default_exception_factory&&)::'lambda'(seastar::internal::promise_base_with_type<storage::usage>&&, seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)&, seastar::future_state<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>&&)::operator()(seastar::internal::promise_base_with_type<storage::usage>&&, seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)&, seastar::future_state<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>&&) const at ./external/+non_module_dependencies+seastar/include/seastar/core/future.hh:1416
 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<storage::usage>, seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory), storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'() seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>::then_impl_nrvo<seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory), seastar::future<storage::usage>>(seastar::semaphore_default_exception_factory&&)::'lambda'(seastar::internal::promise_base_with_type<storage::usage>&&, seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)&, seastar::future_state<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>&&), seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>::run_and_dispose() at ./external/+non_module_dependencies+seastar/include/seastar/core/future.hh:730
seastar::reactor::task_queue::run_tasks() at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:2721
seastar::reactor::task_queue_group::run_tasks() at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:3222
 (inlined by) seastar::reactor::task_queue_group::run_some_tasks() at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:3206
 (inlined by) seastar::reactor::do_run() at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:3394
seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0::operator()() const at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:4685
 (inlined by) decltype(std::declval<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&>()()) std::__1::__invoke[abi:ne200100]<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&) at ./external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__type_traits/invoke.h:179
 (inlined by) void std::__1::__invoke_void_return_wrapper<void, true>::__call[abi:ne200100]<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&) at ./external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__type_traits/invoke.h:251
 (inlined by) void std::__1::__invoke_r[abi:ne200100]<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&) at ./external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__type_traits/invoke.h:273
 (inlined by) std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0>, void ()>::operator()[abi:ne200100]() at ./external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__functional/function.h:167
 (inlined by) std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0>, void ()>::operator()() at ./external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__functional/function.h:319
std::__1::__function::__value_func<void ()>::operator()[abi:ne200100]() const at ./external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__functional/function.h:436
 (inlined by) std::__1::function<void ()>::operator()() const at ./external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__functional/function.h:995
 (inlined by) seastar::posix_thread::start_routine(void*) at ./external/+non_module_dependencies+seastar/src/core/posix.cc:90
?? at ??:0
?? at ??:0

Dec 08 11:06:11 host5 rpk[52221]: Reactor stalled for 56 ms on shard 2, in scheduling group main. Backtrace:
[Backtrace #1]
seastar::guarded_backtrace(void**, int) at ./external/+non_module_dependencies+seastar/src/util/backtrace.cc:102
 (inlined by) void seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::'lambda'(seastar::frame)>(seastar::backtrace_buffer::append_backtrace_oneline()::'lambda'(seastar::frame)&&, bool) at ./external/+non_module_dependencies+seastar/include/seastar/util/backtrace.hh:101
 (inlined by) seastar::backtrace_buffer::append_backtrace_oneline() at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:793
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:815
seastar::internal::cpu_stall_detector::generate_trace() at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:1521
 (inlined by) seastar::internal::cpu_stall_detector::maybe_report() at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:1357
 (inlined by) seastar::internal::cpu_stall_detector::on_signal() at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:1377
 (inlined by) seastar::reactor::block_notifier(int) at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:1585
?? at ??:0
?? at ??:0
seastar::file_desc::write(void const*, unsigned long) at ./external/+non_module_dependencies+seastar/include/seastar/core/posix.hh:311
 (inlined by) seastar::writeable_eventfd::signal(unsigned long) at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:3863
 (inlined by) seastar::syscall_work_queue::submit_item(std::__1::unique_ptr<seastar::syscall_work_queue::work_item, std::__1::default_delete<seastar::syscall_work_queue::work_item>>)::$_0::operator()(seastar::future<void>) at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:3649
 (inlined by) seastar::future<void> seastar::futurize<void>::invoke<seastar::syscall_work_queue::submit_item(std::__1::unique_ptr<seastar::syscall_work_queue::work_item, std::__1::default_delete<seastar::syscall_work_queue::work_item>>)::$_0, seastar::future<void>>(seastar::syscall_work_queue::submit_item(std::__1::unique_ptr<seastar::syscall_work_queue::work_item, std::__1::default_delete<seastar::syscall_work_queue::work_item>>)::$_0&&, seastar::future<void>&&) at ./external/+non_module_dependencies+seastar/include/seastar/core/future.hh:1956
 (inlined by) seastar::futurize<void>::type seastar::future<void>::then_wrapped_common<true, void, seastar::syscall_work_queue::submit_item(std::__1::unique_ptr<seastar::syscall_work_queue::work_item, std::__1::default_delete<seastar::syscall_work_queue::work_item>>)::$_0>(seastar::syscall_work_queue::submit_item(std::__1::unique_ptr<seastar::syscall_work_queue::work_item, std::__1::default_delete<seastar::syscall_work_queue::work_item>>)::$_0&&) at ./external/+non_module_dependencies+seastar/include/seastar/core/future.hh:1516
 (inlined by) seastar::futurize<void>::type seastar::future<void>::then_wrapped_maybe_erase<true, void, seastar::syscall_work_queue::submit_item(std::__1::unique_ptr<seastar::syscall_work_queue::work_item, std::__1::default_delete<seastar::syscall_work_queue::work_item>>)::$_0>(seastar::syscall_work_queue::submit_item(std::__1::unique_ptr<seastar::syscall_work_queue::work_item, std::__1::default_delete<seastar::syscall_work_queue::work_item>>)::$_0&&) at ./external/+non_module_dependencies+seastar/include/seastar/core/future.hh:1475
 (inlined by) _ZNO7seastar6futureIvE12then_wrappedITkNSt3__19invocableIONS0_IT_EEEEZNS_18syscall_work_queue11submit_itemENS3_10unique_ptrINS9_9work_itemENS3_14default_deleteISB_EEEEE3$_0vEENS_8futurizeIT0_E4typeEOS5_ at ./external/+non_module_dependencies+seastar/include/seastar/core/future.hh:1466
 (inlined by) seastar::syscall_work_queue::submit_item(std::__1::unique_ptr<seastar::syscall_work_queue::work_item, std::__1::default_delete<seastar::syscall_work_queue::work_item>>) at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:3642
seastar::future<seastar::syscall_result_extra<stat>> seastar::syscall_work_queue::submit<seastar::syscall_result_extra<stat>>(seastar::noncopyable_function<seastar::syscall_result_extra<stat> ()>) at ./external/+non_module_dependencies+seastar/src/core/syscall_work_queue.hh:69
 (inlined by) seastar::future<seastar::syscall_result_extra<stat>> seastar::thread_pool::submit<seastar::syscall_result_extra<stat>, seastar::reactor::file_stat(std::__1::basic_string_view<char, std::__1::char_traits<char>>, seastar::bool_class<seastar::follow_symlink_tag>)::$_0::operator()() const::'lambda'()>(seastar::internal::thread_pool_submit_reason, seastar::reactor::file_stat(std::__1::basic_string_view<char, std::__1::char_traits<char>>, seastar::bool_class<seastar::follow_symlink_tag>)::$_0::operator()() const::'lambda'()) at ./external/+non_module_dependencies+seastar/src/core/thread_pool.hh:68
 (inlined by) seastar::reactor::file_stat(std::__1::basic_string_view<char, std::__1::char_traits<char>>, seastar::bool_class<seastar::follow_symlink_tag>)::$_0::operator()() const at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:2242
 (inlined by) seastar::future<seastar::stat_data> seastar::futurize<seastar::future<seastar::stat_data>>::invoke<seastar::reactor::file_stat(std::__1::basic_string_view<char, std::__1::char_traits<char>>, seastar::bool_class<seastar::follow_symlink_tag>)::$_0>(seastar::reactor::file_stat(std::__1::basic_string_view<char, std::__1::char_traits<char>>, seastar::bool_class<seastar::follow_symlink_tag>)::$_0&&) at ./external/+non_module_dependencies+seastar/include/seastar/core/future.hh:1959
 (inlined by) auto seastar::futurize_invoke<seastar::reactor::file_stat(std::__1::basic_string_view<char, std::__1::char_traits<char>>, seastar::bool_class<seastar::follow_symlink_tag>)::$_0>(seastar::reactor::file_stat(std::__1::basic_string_view<char, std::__1::char_traits<char>>, seastar::bool_class<seastar::follow_symlink_tag>)::$_0&&) at ./external/+non_module_dependencies+seastar/include/seastar/core/future.hh:1990
 (inlined by) seastar::reactor::file_stat(std::__1::basic_string_view<char, std::__1::char_traits<char>>, seastar::bool_class<seastar::follow_symlink_tag>) at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:2241
seastar::reactor::file_size(std::__1::basic_string_view<char, std::__1::char_traits<char>>) at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:2273
seastar::file_size(std::__1::basic_string_view<char, std::__1::char_traits<char>>) at ./external/+non_module_dependencies+seastar/src/util/file.cc:149
 (inlined by) storage::segment::persistent_size() at ./src/v/storage/segment.cc:153
storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()::operator()() const at ./src/v/storage/disk_log_impl.cc:4094
 (inlined by) seastar::future<storage::usage> seastar::futurize<seastar::future<storage::usage>>::invoke<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>(storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&) at ./external/+non_module_dependencies+seastar/include/seastar/core/future.hh:1959
 (inlined by) auto seastar::futurize_invoke<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>(storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&) at ./external/+non_module_dependencies+seastar/include/seastar/core/future.hh:1990
 (inlined by) auto seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)::operator()<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>(seastar::semaphore_default_exception_factory) at ./external/+non_module_dependencies+seastar/include/seastar/core/semaphore.hh:778
decltype(std::declval<seastar::semaphore_default_exception_factory>()(std::declval<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>())) std::__1::__invoke[abi:ne200100]<seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)&, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>(seastar::semaphore_default_exception_factory&&, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&&) at ./external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__type_traits/invoke.h:179
 (inlined by) std::__1::invoke_result<seastar::semaphore_default_exception_factory, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>::type std::__1::invoke[abi:ne200100]<seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)&, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>(seastar::semaphore_default_exception_factory&&, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&&) at ./external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__functional/invoke.h:29
 (inlined by) auto seastar::internal::future_invoke<seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)&, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>(seastar::semaphore_default_exception_factory&&, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&) at ./external/+non_module_dependencies+seastar/include/seastar/core/future.hh:1133
 (inlined by) storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'() seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>::then_impl_nrvo<seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory), seastar::future<storage::usage>>(seastar::semaphore_default_exception_factory&&)::'lambda'(seastar::internal::promise_base_with_type<storage::usage>&&, seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)&, seastar::future_state<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>&&)::operator()(seastar::internal::promise_base_with_type<storage::usage>&&, seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)&, seastar::future_state<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>&&) const::'lambda'()::operator()() const at ./external/+non_module_dependencies+seastar/include/seastar/core/future.hh:1420
 (inlined by) void seastar::futurize<seastar::future<storage::usage>>::satisfy_with_result_of<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'() seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>::then_impl_nrvo<seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory), seastar::future<storage::usage>>(seastar::semaphore_default_exception_factory&&)::'lambda'(seastar::internal::promise_base_with_type<storage::usage>&&, seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)&, seastar::future_state<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>&&)::operator()(seastar::internal::promise_base_with_type<storage::usage>&&, seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)&, seastar::future_state<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>&&) const::'lambda'()>(seastar::internal::promise_base_with_type<storage::usage>&&, seastar::semaphore_default_exception_factory&&) at ./external/+non_module_dependencies+seastar/include/seastar/core/future.hh:1944
 (inlined by) storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'() seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>::then_impl_nrvo<seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory), seastar::future<storage::usage>>(seastar::semaphore_default_exception_factory&&)::'lambda'(seastar::internal::promise_base_with_type<storage::usage>&&, seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)&, seastar::future_state<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>&&)::operator()(seastar::internal::promise_base_with_type<storage::usage>&&, seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)&, seastar::future_state<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>&&) const at ./external/+non_module_dependencies+seastar/include/seastar/core/future.hh:1416
 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<storage::usage>, seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory), storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'() seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>::then_impl_nrvo<seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory), seastar::future<storage::usage>>(seastar::semaphore_default_exception_factory&&)::'lambda'(seastar::internal::promise_base_with_type<storage::usage>&&, seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_target_time_retention(storage::gc_config)::$_1::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&)::'lambda'(seastar::semaphore_default_exception_factory)&, seastar::future_state<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>&&), seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>>::run_and_dispose() at ./external/+non_module_dependencies+seastar/include/seastar/core/future.hh:730
seastar::reactor::task_queue::run_tasks() at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:2721
seastar::reactor::task_queue_group::run_tasks() at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:3222
 (inlined by) seastar::reactor::task_queue_group::run_some_tasks() at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:3206
 (inlined by) seastar::reactor::do_run() at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:3394
seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0::operator()() const at ./external/+non_module_dependencies+seastar/src/core/reactor.cc:4685
 (inlined by) decltype(std::declval<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&>()()) std::__1::__invoke[abi:ne200100]<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&) at ./external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__type_traits/invoke.h:179
 (inlined by) void std::__1::__invoke_void_return_wrapper<void, true>::__call[abi:ne200100]<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&) at ./external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__type_traits/invoke.h:251
 (inlined by) void std::__1::__invoke_r[abi:ne200100]<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&) at ./external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__type_traits/invoke.h:273
 (inlined by) std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0>, void ()>::operator()[abi:ne200100]() at ./external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__functional/function.h:167
 (inlined by) std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0>, void ()>::operator()() at ./external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__functional/function.h:319
std::__1::__function::__value_func<void ()>::operator()[abi:ne200100]() const at ./external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__functional/function.h:436
 (inlined by) std::__1::function<void ()>::operator()() const at ./external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__functional/function.h:995
 (inlined by) seastar::posix_thread::start_routine(void*) at ./external/+non_module_dependencies+seastar/src/core/posix.cc:90
?? at ??:0
?? at ??:0

[dotnwat]

cc @WillemKauf

[WillemKauf]

The first backtrace is around a simple string formatting here:

redpanda/src/v/storage/fs_utils.cc

Lines 24 to 32 in f8004ef

	return ss::format(
	"{}/{}_{}/{}-{}-{}{}",
	dir_part.base_dir.value_or(config::node().data_directory().as_sstring()),
	dir_part.ntp.path(),
	dir_part.revision_id,
	file_part.base_offset(),
	file_part.term(),
	to_string(file_part.version),
	extension);

which is surprising to me.

The second backtrace is deeper in the seastar reactor when trying to submit a file_stat request to the syscall_work_queue which is... also surprising.

[dotnwat]

@WillemKauf i think the one with the formatting is probably because of the map_reduce loop in disk_usage_target_time_retention calling persistent_size?

[hoxnox]

I've got a lot of...

stall.log

[dotnwat]

@hoxnox what kind of hardware are you running on? and anything interesting about your workload?

[WillemKauf]

(storage filesystem & OS would also be nice to know)

[hoxnox]

@hoxnox what kind of hardware are you running on? and anything interesting about your workload?

It's virtualized hardware. 4xCPU, 8GB DDR, write speed (measured by dd ~100MB/s)

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 79
model name      : Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
stepping        : 1
microcode       : 0xb00001d
cpu MHz         : 2299.996
cache size      : 51200 KB
physical id     : 3
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 3
initial apicid  : 3
fpu             : yes
fpu_exception   : yes
cpuid level     : 20
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault pti tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat vnmi umip arch_capabilities
vmx flags       : vnmi preemption_timer posted_intr invvpid ept_x_only ept_ad ept_1gb flexpriority apicv tsc_offset vtpr mtf vapic ept vpid unrestricted_guest vapic_reg vid shadow_vmcs pml
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa mmio_stale_data bhi its
bogomips        : 4599.99
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

[hoxnox]

@hoxnox what kind of hardware are you running on? and anything interesting about your workload?

I've got stall messages even if there is no load at all.

[hoxnox]

uname -a
Linux 6.12.48+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.48-1 (2025-09-20) x86_64 GNU/Linux

[hoxnox]

There are no problems with another two machines in the cluster.
The differ a little on CPU. But the amount of resources and OS is the same.

[hoxnox]

CPU usage is much higher on the third node.

[dotnwat]

Thanks @hoxnox. One of the reactor stalls is because there is a lot of work processing files on disk. Given that it's not happening on other nodes, this is most likely an imbalance. Are you running with the partition and leadership balancers turned on?

What do your workload look like? E.g. how many topics/partitions? Do you have any single partitions that are very large (which can't be balanced).

[hoxnox]

Thanks @hoxnox. One of the reactor stalls is because there is a lot of work processing files on disk. Given that it's not happening on other nodes, this is most likely an imbalance. Are you running with the partition and leadership balancers turned on?

What do your workload look like? E.g. how many topics/partitions? Do you have any single partitions that are very large (which can't be balanced).

It's pretty ballanced

0          1       1010   [1 2 3]   0                 16534578
1          3       1060   [1 2 3]   0                 16567435
2          3       954    [1 2 3]   0                 16540531
3          3       1140   [1 2 3]   0                 16545697
4          2       946    [1 2 3]   0                 16564011
5          1       1003   [1 2 3]   0                 16527679

[hoxnox]

Thanks @hoxnox. One of the reactor stalls is because there is a lot of work processing files on disk. Given that it's not happening on other nodes, this is most likely an imbalance. Are you running with the partition and leadership balancers turned on?
What do your workload look like? E.g. how many topics/partitions? Do you have any single partitions that are very large (which can't be balanced).

It's pretty ballanced

0          1       1010   [1 2 3]   0                 16534578
1          3       1060   [1 2 3]   0                 16567435
2          3       954    [1 2 3]   0                 16540531
3          3       1140   [1 2 3]   0                 16545697
4          2       946    [1 2 3]   0                 16564011
5          1       1003   [1 2 3]   0                 16527679

Oh. I see. it's not...
I need to make more partitions?

[dotnwat]

0          1       1010   [1 2 3]   0                 16534578

what are these columns? or, what command are you using and i can look myself

[WillemKauf]

Can we see an rpk topic describe-storage [topic] -print-all?

[hoxnox]

This is the most heavy topic.

SUMMARY
=======
NAME                XXXXXX
PARTITIONS          6
REPLICAS            3
CLOUD-STORAGE-MODE  disabled
LAST-UPLOAD         0

OFFSETS
=======
PARTITION  CLOUD-START  CLOUD-LAST  LOCAL-START  LOCAL-LAST
0          0            0           0            16573723
1          0            0           0            16605825
2          0            0           0            16580844
3          0            0           0            16583711
4          0            0           0            16603714
5          0            0           0            16567902

SIZE
====
PARTITION  CLOUD-BYTES  LOCAL-BYTES  TOTAL-BYTES  CLOUD-SEGMENTS  LOCAL-SEGMENTS
0          0            3028211570   3028211570   0               979
1          0            3032810210   3032810210   0               1043
2          0            3032743477   3032743477   0               932
3          0            3035774399   3035774399   0               1086
4          0            3041041730   3041041730   0               922
5          0            3025112028   3025112028   0               977

SYNC
====
PARTITION  LAST-SEGMENT-UPLOAD  LAST-MANIFEST-UPLOAD  METADATA-UPDATE-PENDING
0          0                    0                     false
1          0                    0                     false
2          0                    0                     false
3          0                    0                     false
4          0                    0                     false
5          0                    0                     false

[hoxnox]

But why the problem persists even if there is no load?

[dotnwat]

there may be background work going on, and if there is an imbalance then one or more cores may be overloaded. it's also possible that the virtualized environment is causing some issues--for example if the VM is competing for CPU resources.

FWIW the reactor stall messages don't mean there is necessarily a problem and if the system is working well then they can be ignored. They are intended to help reveal where sources of tail latency may be originating from. But if they are printed frequently yes they can become annoying and concerning.

[hoxnox]

there may be background work going on, and if there is an imbalance then one or more cores may be overloaded. it's also possible that the virtualized environment is causing some issues--for example if the VM is competing for CPU resources.

FWIW the reactor stall messages don't mean there is necessarily a problem and if the system is working well then they can be ignored. They are intended to help reveal where sources of tail latency may be originating from. But if they are printed frequently yes they can become annoying and concerning.

Unfortunately other hosts complains on the stalled one.

Dec 08 18:28:28 host1 rpk[1284856]: INFO  2025-12-08 18:28:28,082 [shard 0:main] rpc - transport.cc:215 - RPC timeout (100 ms) to {host: host5, port: 33145}, method: node_status_rpc::node_status, correlation id: 267681, 1 in flight, time since: {init: 100 ms, enqueue: 100 ms, memory_reserved: 100 ms dispatch: 100 ms, written: 100 ms}, flushed: true
Dec 08 18:28:51 host1 rpk[1284856]: INFO  2025-12-08 18:28:51,198 [shard 0:main] rpc - transport.cc:215 - RPC timeout (100 ms) to {host: host5, port: 33145}, method: node_status_rpc::node_status, correlation id: 267905, 1 in flight, time since: {init: 100 ms, enqueue: 100 ms, memory_reserved: 100 ms dispatch: 100 ms, written: 100 ms}, flushed: true
Dec 08 18:29:30 host1 rpk[1284856]: INFO  2025-12-08 18:29:30,108 [shard 0:main] rpc - transport.cc:215 - RPC timeout (100 ms) to {host: host5, port: 33145}, method: node_status_rpc::node_status, correlation id: 268281, 1 in flight, time since: {init: 100 ms, enqueue: 100 ms, memory_reserved: 100 ms dispatch: 100 ms, written: 100 ms}, flushed: true
Dec 08 18:29:30 host1 rpk[1284856]: INFO  2025-12-08 18:29:30,310 [shard 0:main] rpc - transport.cc:215 - RPC timeout (100 ms) to {host: host5, port: 33145}, method: node_status_rpc::node_status, correlation id: 268282, 1 in flight, time since: {init: 100 ms, enqueue: 100 ms, memory_reserved: 100 ms dispatch: 100 ms, written: 100 ms}, flushed: true

[dotnwat]

Happy to keep helping out here, but you may also find additional help in the Community Slack channel.

[WillemKauf]

rpk cluster self-test start --only-disk-test and rpk cluster self-test status may also help us determine if there is heavily degraded performance on one of your nodes versus the others.

[dotnwat]

Yes, this is a good idea. I have a feeling that perhaps Redpanda is a bit under powered

[hoxnox]

rpk cluster self-test start --only-disk-test

It's under workload now. When it will be finished - i'll test.
Thank you a lot for your help.

[hoxnox]

rpk cluster self-test start --only-disk-test and rpk cluster self-test status may also help us determine if there is heavily degraded performance on one of your nodes versus the others.

According to the data the third node has worst filesystem?

selftest.log.gz

[dotnwat]

the io performance seems to have very high latency (e.g. p50 of 311ms below)

LATENCY       P50       P90       P99       P999      MAX
              311295us  491519us  524287us  524287us  524287us

normally we are working with single to low double digit ms. are you running on spinning disks by chance? even then it seems quite high. perhaps there is other load on your system?

[hoxnox]

the io performance seems to have very high latency (e.g. p50 of 311ms below)

LATENCY       P50       P90       P99       P999      MAX
              311295us  491519us  524287us  524287us  524287us

normally we are working with single to low double digit ms. are you running on spinning disks by chance? even then it seems quite high. perhaps there is other load on your system?

Thank you for the guidelines.
It's cloud storage. I've requested the other block storage.