setting allow_ec_optimizations on an in use pool makes OSD crash in a loop #1009
Description
Ceph version
20.2.0 tentacle (stable - RelWithDebInfo) using cephadm
after setting allow_ec_optimizations on a pool, as soon as there are write/recovery any IO operations affected OSDs crash in a loop making the cluster unusable.
Also there are hundreds of thousands of scrub errors reported:
ec_overwrites was enabled on the pool already.
[ERR] OSD_SCRUB_ERRORS: 4508840 scrub errors
7/2/26 10:53 PM
[ERR]
scrub 9.1fs0 9:f8ffd014:::10000e90b53.0000002f:head : on disk size (1399392) does not match object info size (4194304) adjusted for ondisk to (1400832)
OSD logs give the following errors:
Feb 07 18:46:24 hn-dl380g8-ceph01 ceph-osd[115780]: osd.11 pg_epoch: 146484 pg[9.b8s0( v 145720'1157428 (144861'1147767,145720'1157428] local-lis/les=146483/146484 n=64601 ec=145630/1481 lis/c=146476/146473 les/c/f=146477/146474/18577 sis=146483) [11,9,4,2]p11(0) r=0 lpr=146483 pi=[146473,146483)/1 crt=145720'1157428 lcod 0'0 mlcod 0'0 active mbc={} ps=62] state<Started/Primary/Active>: react AllReplicasActivated Activating complete
Feb 07 18:46:24 hn-dl380g8-ceph01 ceph-osd[115780]: osd.11 pg_epoch: 146484 pg[9.2bs3( v 145720'1155174 (145627'1152983,145720'1155174] local-lis/les=146483/146484 n=64748 ec=1478/1478 lis/c=146473/146407 les/c/f=146474/146408/18577 sis=146483) [NONE,12,9,11]p11(3) r=3 lpr=146483 pi=[146407,146483)/1 crt=145720'1155174 lcod 0'0 mlcod 0'0 active+undersized+degraded mbc={} ps=62] state<Started/Primary/Active>: react AllReplicasActivated Activating complete
Feb 07 18:46:24 hn-dl380g8-ceph01 ceph-osd[115780]: osd.11 pg_epoch: 146484 pg[9.abs3( v 145720'1155094 (145627'1152828,145720'1155094] local-lis/les=146483/146484 n=64729 ec=145630/1478 lis/c=146473/146407 les/c/f=146474/146408/18577 sis=146483) [NONE,12,9,11]p11(3) r=3 lpr=146483 pi=[146407,146483)/1 crt=145720'1155094 lcod 0'0 mlcod 0'0 active+undersized+degraded mbc={} ps=62] state<Started/Primary/Active>: react AllReplicasActivated Activating complete
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: /ceph/rpmbuild/BUILD/ceph-20.2.0/src/include/interval_set.h: In function 'void interval_set<T, C, strict>::intersection_size_asym(const interval_set<T, C, strict>&, const interval_set<T, C, strict>&) [with T = long unsigned int; C = boost::container::flat_map; bool strict = false]' thread 7f258bc88640 time 2026-02-07T18:46:25.026864+0000
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: /ceph/rpmbuild/BUILD/ceph-20.2.0/src/include/interval_set.h: 365: FAILED ceph_assert(en > start)
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11f) [0x5643921da236]
Feb 07 18:46:25 hn-dl380g8-ceph01 ceph-osd[115780]: /ceph/rpmbuild/BUILD/ceph-20.2.0/src/include/interval_set.h: In function 'void interval_set<T, C, strict>::intersection_size_asym(const interval_set<T, C, strict>&, const interval_set<T, C, strict>&) [with T = long unsigned int; C = boost::container::flat_map; bool strict = false]' thread 7f258bc88640 time 2026-02-07T18:46:25.026864+0000
/ceph/rpmbuild/BUILD/ceph-20.2.0/src/include/interval_set.h: 365: FAILED ceph_assert(en > start)
ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11f) [0x5643921da236]
2: /usr/bin/ceph-osd(+0x41bdf0) [0x564392185df0]
3: (ECCommon::ReadPipeline::get_min_avail_to_read_shards(hobject_t const&, bool, bool, ECCommon::read_request_t&, std::optional<std::set<pg_shard_t, std::less<pg_shard_t>, std::allocator<pg_shard_t> > > const&)+0x1222) [0x5643926f0c72]
4: (ECCommon::ReadPipeline::objects_read_and_reconstruct(std::map<hobject_t, std::__cxx11::list<ec_align_t, std::allocator<ec_align_t> >, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, std::__cxx11::list<ec_align_t, std::allocator<ec_align_t> > > > > const&, bool, unsigned long, std::unique_ptr<GenContext<std::map<hobject_t, ECCommon::ec_extent_t, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, ECCommon::ec_extent_t> > >&&>, std::default_delete<GenContext<std::map<hobject_t, ECCommon::ec_extent_t, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, ECCommon::ec_extent_t> > >&&> > >&&)+0x5ec) [0x5643926f595c]
5: (ECBackend::objects_read_async(hobject_t const&, unsigned long, std::__cxx11::list<std::pair<ec_align_t, std::pair<ceph::buffer::v15_2_0::list*, Context*> >, std::allocator<std::pair<ec_align_t, std::pair<ceph::buffer::v15_2_0::list*, Context*> > > > const&, Context*, bool)+0x37a) [0x56439271d00a]
6: (PrimaryLogPG::OpContext::start_async_reads(PrimaryLogPG*)+0x205) [0x56439247e3c5]
7: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x65f) [0x5643924a0d2f]
8: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x2d5f) [0x564392491f5f]
9: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x19f) [0x5643923c105f]
10: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x69) [0x564392621f89]
11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x8bc) [0x5643923ea11c]
12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x23a) [0x56439296ce1a]
13: /usr/bin/ceph-osd(+0xc033d4) [0x56439296d3d4]
14: /lib64/libc.so.6(+0x8b2fa) [0x7f25a94012fa]
15: /lib64/libc.so.6(+0x110400) [0x7f25a9486400]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 2: /usr/bin/ceph-osd(+0x41bdf0) [0x564392185df0]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 3: (ECCommon::ReadPipeline::get_min_avail_to_read_shards(hobject_t const&, bool, bool, ECCommon::read_request_t&, std::optional<std::set<pg_shard_t, std::less<pg_shard_t>, std::allocator<pg_shard_t> > > const&)+0x1222) [0x5643926f0c72]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 4: (ECCommon::ReadPipeline::objects_read_and_reconstruct(std::map<hobject_t, std::__cxx11::list<ec_align_t, std::allocator<ec_align_t> >, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, std::__cxx11::list<ec_align_t, std::allocator<ec_align_t> > > > > const&, bool, unsigned long, std::unique_ptr<GenContext<std::map<hobject_t, ECCommon::ec_extent_t, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, ECCommon::ec_extent_t> > >&&>, std::default_delete<GenContext<std::map<hobject_t, ECCommon::ec_extent_t, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, ECCommon::ec_extent_t> > >&&> > >&&)+0x5ec) [0x5643926f595c]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 5: (ECBackend::objects_read_async(hobject_t const&, unsigned long, std::__cxx11::list<std::pair<ec_align_t, std::pair<ceph::buffer::v15_2_0::list*, Context*> >, std::allocator<std::pair<ec_align_t, std::pair<ceph::buffer::v15_2_0::list*, Context*> > > > const&, Context*, bool)+0x37a) [0x56439271d00a]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 6: (PrimaryLogPG::OpContext::start_async_reads(PrimaryLogPG*)+0x205) [0x56439247e3c5]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 7: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x65f) [0x5643924a0d2f]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 8: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x2d5f) [0x564392491f5f]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 9: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x19f) [0x5643923c105f]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 10: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x69) [0x564392621f89]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x8bc) [0x5643923ea11c]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x23a) [0x56439296ce1a]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 13: /usr/bin/ceph-osd(+0xc033d4) [0x56439296d3d4]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 14: /lib64/libc.so.6(+0x8b2fa) [0x7f25a94012fa]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 15: /lib64/libc.so.6(+0x110400) [0x7f25a9486400]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: *** Caught signal (Aborted) **
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: in thread 7f258bc88640 thread_name:tp_osd_tp
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 2026-02-07T18:46:25.031+0000 7f258bc88640 -1 /ceph/rpmbuild/BUILD/ceph-20.2.0/src/include/interval_set.h: In function 'void interval_set<T, C, strict>::intersection_size_asym(const interval_set<T, C, strict>&, const interval_set<T, C, strict>&) [with T = long unsigned int; C = boost::container::flat_map; bool strict = false]' thread 7f258bc88640 time 2026-02-07T18:46:25.026864+0000
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: /ceph/rpmbuild/BUILD/ceph-20.2.0/src/include/interval_set.h: 365: FAILED ceph_assert(en > start)
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]:
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11f) [0x5643921da236]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 2: /usr/bin/ceph-osd(+0x41bdf0) [0x564392185df0]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 3: (ECCommon::ReadPipeline::get_min_avail_to_read_shards(hobject_t const&, bool, bool, ECCommon::read_request_t&, std::optional<std::set<pg_shard_t, std::less<pg_shard_t>, std::allocator<pg_shard_t> > > const&)+0x1222) [0x5643926f0c72]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 4: (ECCommon::ReadPipeline::objects_read_and_reconstruct(std::map<hobject_t, std::__cxx11::list<ec_align_t, std::allocator<ec_align_t> >, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, std::__cxx11::list<ec_align_t, std::allocator<ec_align_t> > > > > const&, bool, unsigned long, std::unique_ptr<GenContext<std::map<hobject_t, ECCommon::ec_extent_t, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, ECCommon::ec_extent_t> > >&&>, std::default_delete<GenContext<std::map<hobject_t, ECCommon::ec_extent_t, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, ECCommon::ec_extent_t> > >&&> > >&&)+0x5ec) [0x5643926f595c]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 5: (ECBackend::objects_read_async(hobject_t const&, unsigned long, std::__cxx11::list<std::pair<ec_align_t, std::pair<ceph::buffer::v15_2_0::list*, Context*> >, std::allocator<std::pair<ec_align_t, std::pair<ceph::buffer::v15_2_0::list*, Context*> > > > const&, Context*, bool)+0x37a) [0x56439271d00a]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 6: (PrimaryLogPG::OpContext::start_async_reads(PrimaryLogPG*)+0x205) [0x56439247e3c5]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 7: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x65f) [0x5643924a0d2f]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 8: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x2d5f) [0x564392491f5f]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 9: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x19f) [0x5643923c105f]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 10: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x69) [0x564392621f89]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x8bc) [0x5643923ea11c]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x23a) [0x56439296ce1a]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 13: /usr/bin/ceph-osd(+0xc033d4) [0x56439296d3d4]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 14: /lib64/libc.so.6(+0x8b2fa) [0x7f25a94012fa]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 15: /lib64/libc.so.6(+0x110400) [0x7f25a9486400]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]:
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 1: /lib64/libc.so.6(+0x3fc30) [0x7f25a93b5c30]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 2: /lib64/libc.so.6(+0x8d03c) [0x7f25a940303c]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 3: raise()
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 4: abort()
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17a) [0x5643921da291]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 6: /usr/bin/ceph-osd(+0x41bdf0) [0x564392185df0]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 7: (ECCommon::ReadPipeline::get_min_avail_to_read_shards(hobject_t const&, bool, bool, ECCommon::read_request_t&, std::optional<std::set<pg_shard_t, std::less<pg_shard_t>, std::allocator<pg_shard_t> > > const&)+0x1222) [0x5643926f0c72]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 8: (ECCommon::ReadPipeline::objects_read_and_reconstruct(std::map<hobject_t, std::__cxx11::list<ec_align_t, std::allocator<ec_align_t> >, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, std::__cxx11::list<ec_align_t, std::allocator<ec_align_t> > > > > const&, bool, unsigned long, std::unique_ptr<GenContext<std::map<hobject_t, ECCommon::ec_extent_t, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, ECCommon::ec_extent_t> > >&&>, std::default_delete<GenContext<std::map<hobject_t, ECCommon::ec_extent_t, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, ECCommon::ec_extent_t> > >&&> > >&&)+0x5ec) [0x5643926f595c]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 9: (ECBackend::objects_read_async(hobject_t const&, unsigned long, std::__cxx11::list<std::pair<ec_align_t, std::pair<ceph::buffer::v15_2_0::list*, Context*> >, std::allocator<std::pair<ec_align_t, std::pair<ceph::buffer::v15_2_0::list*, Context*> > > > const&, Context*, bool)+0x37a) [0x56439271d00a]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 10: (PrimaryLogPG::OpContext::start_async_reads(PrimaryLogPG*)+0x205) [0x56439247e3c5]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 11: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x65f) [0x5643924a0d2f]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 12: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x2d5f) [0x564392491f5f]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 13: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x19f) [0x5643923c105f]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 14: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x69) [0x564392621f89]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x8bc) [0x5643923ea11c]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x23a) [0x56439296ce1a]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 17: /usr/bin/ceph-osd(+0xc033d4) [0x56439296d3d4]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 18: /lib64/libc.so.6(+0x8b2fa) [0x7f25a94012fa]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 19: /lib64/libc.so.6(+0x110400) [0x7f25a9486400]
Feb 07 18:46:25 hn-dl380g8-ceph01 ceph-osd[115780]: *** Caught signal (Aborted) **
in thread 7f258bc88640 thread_name:tp_osd_tp
ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)
1: /lib64/libc.so.6(+0x3fc30) [0x7f25a93b5c30]
2: /lib64/libc.so.6(+0x8d03c) [0x7f25a940303c]
3: raise()
4: abort()
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17a) [0x5643921da291]
6: /usr/bin/ceph-osd(+0x41bdf0) [0x564392185df0]
7: (ECCommon::ReadPipeline::get_min_avail_to_read_shards(hobject_t const&, bool, bool, ECCommon::read_request_t&, std::optional<std::set<pg_shard_t, std::less<pg_shard_t>, std::allocator<pg_shard_t> > > const&)+0x1222) [0x5643926f0c72]
8: (ECCommon::ReadPipeline::objects_read_and_reconstruct(std::map<hobject_t, std::__cxx11::list<ec_align_t, std::allocator<ec_align_t> >, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, std::__cxx11::list<ec_align_t, std::allocator<ec_align_t> > > > > const&, bool, unsigned long, std::unique_ptr<GenContext<std::map<hobject_t, ECCommon::ec_extent_t, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, ECCommon::ec_extent_t> > >&&>, std::default_delete<GenContext<std::map<hobject_t, ECCommon::ec_extent_t, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, ECCommon::ec_extent_t> > >&&> > >&&)+0x5ec) [0x5643926f595c]
9: (ECBackend::objects_read_async(hobject_t const&, unsigned long, std::__cxx11::list<std::pair<ec_align_t, std::pair<ceph::buffer::v15_2_0::list*, Context*> >, std::allocator<std::pair<ec_align_t, std::pair<ceph::buffer::v15_2_0::list*, Context*> > > > const&, Context*, bool)+0x37a) [0x56439271d00a]
10: (PrimaryLogPG::OpContext::start_async_reads(PrimaryLogPG*)+0x205) [0x56439247e3c5]
11: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x65f) [0x5643924a0d2f]
12: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x2d5f) [0x564392491f5f]
13: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x19f) [0x5643923c105f]
14: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x69) [0x564392621f89]
15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x8bc) [0x5643923ea11c]
16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x23a) [0x56439296ce1a]
17: /usr/bin/ceph-osd(+0xc033d4) [0x56439296d3d4]
18: /lib64/libc.so.6(+0x8b2fa) [0x7f25a94012fa]
19: /lib64/libc.so.6(+0x110400) [0x7f25a9486400]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 2026-02-07T18:46:25.039+0000 7f258bc88640 -1 *** Caught signal (Aborted) **
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: in thread 7f258bc88640 thread_name:tp_osd_tp
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]:
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 1: /lib64/libc.so.6(+0x3fc30) [0x7f25a93b5c30]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 2: /lib64/libc.so.6(+0x8d03c) [0x7f25a940303c]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 3: raise()
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 4: abort()
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17a) [0x5643921da291]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 6: /usr/bin/ceph-osd(+0x41bdf0) [0x564392185df0]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 7: (ECCommon::ReadPipeline::get_min_avail_to_read_shards(hobject_t const&, bool, bool, ECCommon::read_request_t&, std::optional<std::set<pg_shard_t, std::less<pg_shard_t>, std::allocator<pg_shard_t> > > const&)+0x1222) [0x5643926f0c72]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 8: (ECCommon::ReadPipeline::objects_read_and_reconstruct(std::map<hobject_t, std::__cxx11::list<ec_align_t, std::allocator<ec_align_t> >, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, std::__cxx11::list<ec_align_t, std::allocator<ec_align_t> > > > > const&, bool, unsigned long, std::unique_ptr<GenContext<std::map<hobject_t, ECCommon::ec_extent_t, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, ECCommon::ec_extent_t> > >&&>, std::default_delete<GenContext<std::map<hobject_t, ECCommon::ec_extent_t, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, ECCommon::ec_extent_t> > >&&> > >&&)+0x5ec) [0x5643926f595c]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 9: (ECBackend::objects_read_async(hobject_t const&, unsigned long, std::__cxx11::list<std::pair<ec_align_t, std::pair<ceph::buffer::v15_2_0::list*, Context*> >, std::allocator<std::pair<ec_align_t, std::pair<ceph::buffer::v15_2_0::list*, Context*> > > > const&, Context*, bool)+0x37a) [0x56439271d00a]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 10: (PrimaryLogPG::OpContext::start_async_reads(PrimaryLogPG*)+0x205) [0x56439247e3c5]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 11: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x65f) [0x5643924a0d2f]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 12: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x2d5f) [0x564392491f5f]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 13: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x19f) [0x5643923c105f]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 14: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x69) [0x564392621f89]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x8bc) [0x5643923ea11c]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x23a) [0x56439296ce1a]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 17: /usr/bin/ceph-osd(+0xc033d4) [0x56439296d3d4]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 18: /lib64/libc.so.6(+0x8b2fa) [0x7f25a94012fa]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: 19: /lib64/libc.so.6(+0x110400) [0x7f25a9486400]
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Feb 07 18:46:25 hn-dl380g8-ceph01 conmon[115775]:
Feb 07 18:46:25 hn-dl380g8-ceph01 ceph-osd[115780]: asok(0x56439a510000) register_command assert hook 0x56439a422cd0
Feb 07 18:46:25 hn-dl380g8-ceph01 ceph-osd[115780]: asok(0x56439a510000) register_command abort hook 0x56439a422cd0
Feb 07 18:46:25 hn-dl380g8-ceph01 ceph-osd[115780]: asok(0x56439a510000) register_command leak_some_memory hook 0x56439a422cd0
Feb 07 18:46:25 hn-dl380g8-ceph01 ceph-osd[115780]: asok(0x56439a510000) register_command perfcounters_dump hook 0x56439a422cd0
Feb 07 18:46:25 hn-dl380g8-ceph01 ceph-osd[115780]: asok(0x56439a510000) register_command 1 hook 0x56439a422cd0