Index worker stopped importing block, which is seemingly related to `Failed to fetch seqno ... ltdb: block not found`

[qnduncan]

Version: 22a3755 (ton-indexer v1.1.4)

The index worker stopped importing blocks at 43728121. Below are the logs around that block.

logs from index-worker

[ 3][t 5][2025-01-07 21:31:10.355308276][IndexScheduler.cpp:235][!indexscheduler]        43728119 / 43728130        0.358533/s (30s)        Q[0M, 0b,0t, 0m, 0T]
[ 1][t 7][2025-01-07 21:31:10.484729078][IndexScheduler.cpp:131][!indexquery]        Failed to fetch seqno 43728127: [Error : 651 : (0,2000000000000000,48749537): ltdb: block not found]
[ 1][t 1][2025-01-07 21:31:10.484736492][IndexScheduler.cpp:131][!indexquery]        Failed to fetch seqno 43728126: [Error : 651 : (0,2000000000000000,48749536): ltdb: block not found]
[ 2][t 7][2025-01-07 21:31:10.484767550][IndexScheduler.cpp:142][!indexscheduler]        Rescheduling seqno 43728127
[ 2][t 7][2025-01-07 21:31:10.484773081][IndexScheduler.cpp:142][!indexscheduler]        Rescheduling seqno 43728126
[ 1][t 1][2025-01-07 21:31:10.484847460][IndexScheduler.cpp:131][!indexquery]        Failed to fetch seqno 43728121: [Error : 651 : (0,2000000000000000,48749531): ltdb: block not found]
[ 2][t 1][2025-01-07 21:31:10.484869361][IndexScheduler.cpp:142][!indexscheduler]        Rescheduling seqno 43728121
[ 1][t 7][2025-01-07 21:31:10.484873569][IndexScheduler.cpp:131][!indexquery]        Failed to fetch seqno 43728122: [Error : 651 : (0,2000000000000000,48749532): ltdb: block not found]
[ 1][t 1][2025-01-07 21:31:10.484875503][IndexScheduler.cpp:131][!indexquery]        Failed to fetch seqno 43728130: [Error : 651 : (0,2000000000000000,48749540): ltdb: block not found]
[ 2][t 7][2025-01-07 21:31:10.484889199][IndexScheduler.cpp:142][!indexscheduler]        Rescheduling seqno 43728122
[ 2][t 7][2025-01-07 21:31:10.484892044][IndexScheduler.cpp:142][!indexscheduler]        Rescheduling seqno 43728130
[ 1][t 4][2025-01-07 21:31:10.484905650][IndexScheduler.cpp:131][!indexquery]        Failed to fetch seqno 43728125: [Error : 651 : (0,2000000000000000,48749535): ltdb: block not found]
[ 2][t 4][2025-01-07 21:31:10.484931699][IndexScheduler.cpp:142][!indexscheduler]        Rescheduling seqno 43728125
[ 1][t 6][2025-01-07 21:31:10.504767532][IndexScheduler.cpp:131][!indexquery]        Failed to fetch seqno 43728126: [Error : 651 : (0,2000000000000000,48749536): ltdb: block not found]
[ 2][t 6][2025-01-07 21:31:10.504788100][IndexScheduler.cpp:142][!indexscheduler]        Rescheduling seqno 43728126
[ 1][t 7][2025-01-07 21:31:10.504799512][IndexScheduler.cpp:131][!indexquery]        Failed to fetch seqno 43728127: [Error : 651 : (0,2000000000000000,48749537): ltdb: block not found]
[ 2][t 7][2025-01-07 21:31:10.504816133][IndexScheduler.cpp:142][!indexscheduler]        Rescheduling seqno 43728127
[ 1][t 5][2025-01-07 21:31:10.504821122][IndexScheduler.cpp:131][!indexquery]        Failed to fetch seqno 43728128: [Error : 651 : (0,2000000000000000,48749538): ltdb: block not found]
[ 2][t 5][2025-01-07 21:31:10.504832574][IndexScheduler.cpp:142][!indexscheduler]        Rescheduling seqno 43728128
[ 1][t 5][2025-01-07 21:31:10.504836341][IndexScheduler.cpp:131][!indexquery]        Failed to fetch seqno 43728123: [Error : 651 : (0,2000000000000000,48749533): ltdb: block not found]
[ 2][t 5][2025-01-07 21:31:10.504843805][IndexScheduler.cpp:142][!indexscheduler]        Rescheduling seqno 43728123
[ 1][t 7][2025-01-07 21:31:10.504877619][IndexScheduler.cpp:131][!indexquery]        Failed to fetch seqno 43728124: [Error : 651 : (0,2000000000000000,48749534): ltdb: block not found]
[ 2][t 7][2025-01-07 21:31:10.504891505][IndexScheduler.cpp:142][!indexscheduler]        Rescheduling seqno 43728124
[ 1][t 3][2025-01-07 21:31:10.504891234][IndexScheduler.cpp:131][!indexquery]        Failed to fetch seqno 43728129: [Error : 651 : (0,2000000000000000,48749539): ltdb: block not found]
[ 2][t 3][2025-01-07 21:31:10.504904690][IndexScheduler.cpp:142][!indexscheduler]        Rescheduling seqno 43728129
[ 1][t 7][2025-01-07 21:31:10.504919808][IndexScheduler.cpp:131][!indexquery]        Failed to fetch seqno 43728131: [Error : 651 : (0,2000000000000000,48749541): ltdb: block not found]
[ 2][t 7][2025-01-07 21:31:10.504930769][IndexScheduler.cpp:142][!indexscheduler]        Rescheduling seqno 43728131
[ 3][t 7][2025-01-07 21:31:10.512375152][InsertManagerBase.cpp:81][!insertmanager]        Inserting batch[mb=1, b=5, txs=69, msgs=119, traces=43]
[ 3][t 7][2025-01-07 21:31:10.646914064][InsertManagerPostgres.cpp:1243][!insert_batch_postgres]        Indexed mintless jetton: 0:09F2E59DEC406AB26A5259A45D7FF23EF11F3E5C7C21DE0B0D2A1CBE52B76B3D
[ 1][t 2][2025-01-07 21:31:11.379093934][IndexScheduler.cpp:131][!indexquery]        Failed to fetch seqno 43728128: [Error : 651 : (0,2000000000000000,48749538): ltdb: block not found]
[ 2][t 2][2025-01-07 21:31:11.379123159][IndexScheduler.cpp:142][!indexscheduler]        Rescheduling seqno 43728128
[ 1][t 1][2025-01-07 21:31:11.379129732][IndexScheduler.cpp:131][!indexquery]        Failed to fetch seqno 43728121: [Error : 651 : (0,2000000000000000,48749531): ltdb: block not found]
[ 2][t 1][2025-01-07 21:31:11.379144169][IndexScheduler.cpp:142][!indexscheduler]        Rescheduling seqno 43728121
[ 1][t 1][2025-01-07 21:31:11.379148757][IndexScheduler.cpp:131][!indexquery]        Failed to fetch seqno 43728125: [Error : 651 : (0,2000000000000000,48749535): ltdb: block not found]
[ 2][t 1][2025-01-07 21:31:11.379155791][IndexScheduler.cpp:142][!indexscheduler]        Rescheduling seqno 43728125
[ 1][t 2][2025-01-07 21:31:11.379182090][IndexScheduler.cpp:131][!indexquery]        Failed to fetch seqno 43728126: [Error : 651 : (0,2000000000000000,48749536): ltdb: block not found]
[ 2][t 2][2025-01-07 21:31:11.379198270][IndexScheduler.cpp:142][!indexscheduler]        Rescheduling seqno 43728126
[ 1][t 2][2025-01-07 21:31:11.379210103][IndexScheduler.cpp:131][!indexquery]        Failed to fetch seqno 43728127: [Error : 651 : (0,2000000000000000,48749537): ltdb: block not found]
[ 2][t 2][2025-01-07 21:31:11.379220502][IndexScheduler.cpp:142][!indexscheduler]        Rescheduling seqno 43728127
[ 1][t 1][2025-01-07 21:31:11.379235390][IndexScheduler.cpp:131][!indexquery]        Failed to fetch seqno 43728130: [Error : 651 : (0,2000000000000000,48749540): ltdb: block not found]
[ 2][t 1][2025-01-07 21:31:11.379246441][IndexScheduler.cpp:142][!indexscheduler]        Rescheduling seqno 43728130
[ 1][t 1][2025-01-07 21:31:11.379257221][IndexScheduler.cpp:131][!indexquery]        Failed to fetch seqno 43728122: [Error : 651 : (0,2000000000000000,48749532): ltdb: block not found]
[ 2][t 1][2025-01-07 21:31:11.379265447][IndexScheduler.cpp:142][!indexscheduler]        Rescheduling seqno 43728122
[ 1][t 1][2025-01-07 21:31:12.483179167][IndexScheduler.cpp:131][!indexquery]        Failed to fetch seqno 43728131: [Error : 651 : (0,2000000000000000,48749541): ltdb: block not found]
[ 2][t 1][2025-01-07 21:31:12.483701048][IndexScheduler.cpp:142][!indexscheduler]        Rescheduling seqno 43728131
[ 3][t 2][2025-01-07 21:31:12.513831050][InsertManagerBase.cpp:81][!insertmanager]        Inserting batch[mb=1, b=5, txs=68, msgs=108, traces=52]
[ 3][t 4][2025-01-07 21:31:15.947600837][TraceAssembler.cpp:109][!trace_assembler]         Pending traces: 1982518 Pending edges: 1119 Broken traces: 208
[ 3][t 6][2025-01-07 21:31:20.365768900][IndexScheduler.cpp:235][!indexscheduler]        43728121 / 43728134        0.342706/s (37s)        Q[0M, 0b, 0t, 0m, 0T]
[ 3][t 7][2025-01-07 21:31:25.948640834][TraceAssembler.cpp:109][!trace_assembler]         Pending traces: 1982518 Pending edges: 1119 Broken traces: 208
[ 3][t 7][2025-01-07 21:31:30.375861692][IndexScheduler.cpp:235][!indexscheduler]        43728121 / 43728138        0.309937/s (54s)        Q[0M, 0b, 0t, 0m, 0T]
[ 3][t 3][2025-01-07 21:31:35.949723140][TraceAssembler.cpp:109][!trace_assembler]         Pending traces: 1982518 Pending edges: 1119 Broken traces: 208
[ 3][t 3][2025-01-07 21:31:40.387099056][IndexScheduler.cpp:235][!indexscheduler]        43728121 / 43728142        0.280302/s (1m 14s)        Q[0M, 0b, 0t, 0m, 0T]
[ 3][t 5][2025-01-07 21:31:45.950286060][TraceAssembler.cpp:109][!trace_assembler]         Pending traces: 1982518 Pending edges: 1119 Broken traces: 208
[ 3][t 7][2025-01-07 21:31:50.397797517][IndexScheduler.cpp:235][!indexscheduler]        43728121 / 43728144        0.253500/s (1m 30s)        Q[0M, 0b, 0t, 0m, 0T]
[ 3][t 4][2025-01-07 21:31:55.951017106][TraceAssembler.cpp:109][!trace_assembler]         Pending traces: 1982518 Pending edges: 1119 Broken traces: 208
[ 3][t 4][2025-01-07 21:32:00.409799198][IndexScheduler.cpp:235][!indexscheduler]        43728121 / 43728148        0.229261/s (1m 57s)        Q[0M, 0b, 0t, 0m, 0T]
[ 3][t 5][2025-01-07 21:32:05.951496609][TraceAssembler.cpp:109][!trace_assembler]         Pending traces: 1982518 Pending edges: 1119 Broken traces: 208
[ 3][t 2][2025-01-07 21:32:10.420505593][IndexScheduler.cpp:235][!indexscheduler]        43728121 / 43728152        0.207339/s (2m 29s)        Q[0M, 0b, 0t, 0m, 0T]
[ 3][t 3][2025-01-07 21:32:15.952265367][TraceAssembler.cpp:109][!trace_assembler]         Pending traces: 1982518 Pending edges: 1119 Broken traces: 208
[ 3][t 6][2025-01-07 21:32:20.430982408][IndexScheduler.cpp:235][!indexscheduler]        43728121 / 43728156        0.187514/s (3m 6s)        Q[0M, 0b, 0t, 0m, 0T]
[ 3][t 5][2025-01-07 21:32:25.953566625][TraceAssembler.cpp:109][!trace_assembler]         Pending traces: 1982518 Pending edges: 1119 Broken traces: 208
[ 3][t 4][2025-01-07 21:32:30.442025346][IndexScheduler.cpp:235][!indexscheduler]        43728121 / 43728160        0.169584/s (3m 49s)        Q[0M, 0b, 0t, 0m, 0T]
[ 3][t 4][2025-01-07 21:32:35.954163778][TraceAssembler.cpp:109][!trace_assembler]         Pending traces: 1982518 Pending edges: 1119 Broken traces: 208

logs from event-classifier

Jan 07 21:32:49 36bc7c913e8b[1135]: ERROR:indexer.events.blocks.basic_matchers:Error while building block !call_contract:=0xf8a7ea5 with matcher JettonTransferBlockMatcher: bitarray index out of range. Trace id: 1FmJfVECZKWhpE6MnmK2YZI2XCknLUzvoAC+ywv8jPg=
Jan 07 21:32:49 36bc7c913e8b[1135]: ERROR:indexer.events.blocks.basic_matchers:Error while building block !call_contract:=0xf8a7ea5 with matcher JettonTransferBlockMatcher: bitarray index out of range. Trace id: zCFh9ik1R2ceprJ3JYDgj2L07KYM0UgjZaTfdYGuN+4=

[qnduncan]

The index worker can resume importing blocks after a restart, but this issue happens from time to time.

[dungeon-master-666]

Thank you for reporting this @qnduncan. We haven't reproduced this issue yet, but recent updates may have fixed it. Could you please update to latest main (commit 4afeda1) and let us know if the problem persists? Don't forget to update submodules with git submodule update --init --recursive

[qnduncan]

Hello @dungeon-master-666,
Sorry for missing your message and late reply.
We didn't get this issue after upgrading ton-indexer stack to v1.1.5 and ton-liteserver to TON v2025.03.

Thanks for the help.