A large number of traces are in a pending state?

[wangxiaojia-wxj]

A large number of traces are in a pending state, but they have indeed been completed

Pending traces: 329966 Pending edges: 1543 Broken traces: 429

For example, one of them:trace_id='ycqOqX6papQ/CXrFY2Mu7ukytoFhExbz/PxhnNAIb9c='

[wangxiaojia-wxj]

For example, one of them:trace_id='ycqOqX6papQ/CXrFY2Mu7ukytoFhExbz/PxhnNAIb9c='
{
"trace_id": "ycqOqX6papQ/CXrFY2Mu7ukytoFhExbz/PxhnNAIb9c=",
"external_hash": "0ioaUAVoZqChAO0aHZMhfmWK96OoWICtlVIh/GJwXIU=",
"mc_seqno_start": "42339034",
"mc_seqno_end": "42339037",
"start_lt": "51239066000003",
"start_utime": 1732531647,
"end_lt": "51239069000002",
"end_utime": 1732531655,
"trace_info": {
"trace_state": "pending",
"messages": 13,
"transactions": 13,
"pending_messages": 2,
"classification_state": "unclassified"
},

Block 42339038 does indeed exist in the database，but mc_deqno_end=42339037
Can you give me some advice? What might be the problem？

[wangxiaojia-wxj]

Is there a problem with this modification statement in the InsertManagerial Postgres. cpp file?
if (!is_first_trace) {
traces_query << " ON CONFLICT (trace_id) DO UPDATE SET "
<< "mc_seqno_end = EXCLUDED.mc_seqno_end, "
<< "end_lt = EXCLUDED.end_lt, "
<< "end_utime = EXCLUDED.end_utime, "
<< "state = EXCLUDED.state, "
<< "pending_edges_ = EXCLUDED.pending_edges_, "
<< "edges_ = EXCLUDED.edges_, "
<< "nodes_ = EXCLUDED.nodes_ "
<< "WHERE traces.end_lt < EXCLUDED.end_lt;\n";

It should be like this, right?
traces_query << " ON CONFLICT (trace_id) DO UPDATE SET "
<< "mc_seqno_end = EXCLUDED.mc_seqno_end, "
<< "end_lt = EXCLUDED.end_lt, "
<< "end_utime = EXCLUDED.end_utime, "
<< "state = EXCLUDED.state, "
<< "pending_edges_ = EXCLUDED.pending_edges_, "
<< "edges_ = EXCLUDED.edges_, "
<< "nodes_ = EXCLUDED.nodes_ "
<< "WHERE traces.end_lt < EXCLUDED.end_lt or traces.mc_seqno_end < EXCLUDED.mc_seqno_end ;\n";

[qnduncan]

We have a similar issue with the logs below. Is it an expected behavior?

[ 3][t 1][2025-01-08 01:11:06.965144282][TraceAssembler.cpp:109][!trace_assembler]       Pending traces: 1991043 Pending edges: 1283 Broken traces: 0
[ 3][t 3][2025-01-08 01:11:07.247597894][IndexScheduler.cpp:235][!indexscheduler]       43733166 / 43733166     0.346347/s (0s) Q[0M, 0b, 0t, 0m, 0T]
[ 3][t 1][2025-01-08 01:11:08.753295040][InsertManagerBase.cpp:81][!insertmanager]      Inserting batch[mb=1, b=5, txs=66, msgs=114, traces=38]
[ 1][t 1][2025-01-08 01:11:10.428188784][IndexScheduler.cpp:131][!indexquery]   Failed to fetch seqno 43733169: [Error : 651 : (0,2000000000000000,48754743): ltdb: block not found]
[ 2][t 1][2025-01-08 01:11:10.428211787][IndexScheduler.cpp:142][!indexscheduler]       Rescheduling seqno 43733169
[ 3][t 6][2025-01-08 01:11:10.755530429][InsertManagerBase.cpp:81][!insertmanager]      Inserting batch[mb=1, b=3, txs=36, msgs=58, traces=24]
[ 1][t 6][2025-01-08 01:11:11.278366114][IndexScheduler.cpp:131][!indexquery]   Failed to fetch seqno 43733169: [Error : 651 : (0,2000000000000000,48754743): ltdb: block not found]
[ 2][t 6][2025-01-08 01:11:11.278412922][IndexScheduler.cpp:142][!indexscheduler]       Rescheduling seqno 43733169
[ 3][t 4][2025-01-08 01:11:12.757798959][InsertManagerBase.cpp:81][!insertmanager]      Inserting batch[mb=1, b=6, txs=91, msgs=161, traces=61]
[ 3][t 1][2025-01-08 01:11:15.761199176][InsertManagerBase.cpp:81][!insertmanager]      Inserting batch[mb=1, b=4, txs=56, msgs=93, traces=42]
[ 3][t 2][2025-01-08 01:11:16.965583794][TraceAssembler.cpp:109][!trace_assembler]       Pending traces: 1991059 Pending edges: 1287 Broken traces: 0
[ 3][t 2][2025-01-08 01:11:17.258088277][IndexScheduler.cpp:235][!indexscheduler]       43733170 / 43733170     0.398737/s (0s) Q[0M, 0b, 0t, 0m, 0T]