participant try to vet the admin dars again and fail with outdated cost errors

[nicu-da]

Later edit

https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.namespace_name%3D%22validator-pool-001%22%0AjsonPayload.%22trace-id%22%3D%2262ec14b11aad6ac778b241529ac01d0d%22;pinnedLogId=2025-12-10T22:27:54.161615398Z%2Fxsvtowh5fgk0z6ea;summaryFields=resource%252Flabels%252Fcontainer_name:false:32:beginning;cursorTimestamp=2025-12-10T22:28:08.055819829Z;duration=P2D?project=cnu-mainnet

which leads to https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.namespace_name%3D%22validator-pool-001%22%0AjsonPayload.%22trace-id%22%3D%22389f4b0899bb67d20cc2297aafa5ba86%22;summaryFields=resource%252Flabels%252Fcontainer_name:false:32:beginning;cursorTimestamp=2025-12-10T23:58:14.855555370Z;aroundTime=2025-12-11T00:00:00.000Z;duration=PT5M?project=cnu-mainnet

this also happens without an lsu, after a HDM for example, and I assume it will also happen when new participant onboard to a sync

Initial report

after a LSU is performed

participants connect to the new PS

the participants try to upload the admin dars again and vet it again

the vetting transaction fails because the cost is outdated

this should already be vetted

https://drive.google.com/file/d/1tiZFwn_T09aiSz5zQxPgvy6dg5XKTesW/view?usp=drive_link

2025-12-09T13:36:39.012Z [⋮] INFO - c.d.c.t.TopologyStateProcessor:participant=sv1Participant/psid=global-domain::1220119e44dd::34-0 (bb3a062c06ba84fe24d3e6496cb3a507-MessageDispatcher.handle-ff82587279435d67-48d91d64b5f563bb) - Persisted topology transactions (SequencedTime(2025-12-09T13:36:38.684032Z), EffectiveTime(2025-12-09T13:36:38.934032Z)):
ValidatedTopologyTransaction(
  SignedTopologyTransaction(
    TopologyTransaction(
      SynchronizerUpgradeAnnouncement(successorSynchronizerId = global-domain::1220119e44dd...::34-1, upgradeTime = 2025-12-09T13:37:38.510309Z),
      serial = 1,
      operation = Replace,
      hash = SHA-256:f6250432c5a7...
    ),
    signatures = Seq(12205f6ec6e8..., 1220b2a0f9a8..., 1220beabe7ae..., 1220fcb36492...)
  )
) 
2025-12-09T13:36:49.429Z [⋮] INFO - c.d.c.t.TopologyStateProcessor:participant=sv1Participant/psid=global-domain::1220119e44dd::34-0 (47136809f7e1b4331a9ea18b1674ad8e-MessageDispatcher.handle-78dee7960ab6fe50-a05302c2a5444754) - Persisted topology transactions (SequencedTime(2025-12-09T13:36:49.262540Z), EffectiveTime(2025-12-09T13:36:49.512540Z)):
ValidatedTopologyTransaction(
  SignedTopologyTransaction(
    TopologyTransaction(
      SequencerConnectionSuccessor(sequencerId = SEQ::sv3::1220d37f600d..., synchronizerId = global-domain::1220119e44dd..., connection = localhost:27308),
      serial = 1,
      operation = Replace,
      hash = SHA-256:99f205fd85af...
    ),
    signatures = 1220d37f600d...
  )
) 
2025-12-09T13:36:49.435Z [⋮] DEBUG - c.d.c.p.t.SequencerConnectionSuccessorListener:participant=sv1Participant (47136809f7e1b4331a9ea18b1674ad8e-MessageDispatcher.handle-78dee7960ab6fe50-a05302c2a5444754) - Checking whether the participant can migrate Synchronizer 'global' from global-domain::1220119e44dd9767c1b54ebebe11dae807896aba2e1d674c7ce9c7a7865fdd7f0e97::34-0 to global-domain::1220119e44dd...::34-1 
2025-12-09T13:36:49.437Z [⋮] DEBUG - c.d.c.p.t.SequencerConnectionSuccessorListener:participant=sv1Participant (47136809f7e1b4331a9ea18b1674ad8e-MessageDispatcher.handle-78dee7960ab6fe50-a05302c2a5444754) - Configured sequencer connections: Set(SEQ::sv1::1220a2dc2d13..., SEQ::sv2::122063861133..., SEQ::sv3::1220d37f600d..., SEQ::sv4::1220698395c2...) 
2025-12-09T13:36:49.451Z [⋮] DEBUG - c.d.c.p.t.SequencerConnectionSuccessorListener:participant=sv1Participant (47136809f7e1b4331a9ea18b1674ad8e-MessageDispatcher.handle-78dee7960ab6fe50-a05302c2a5444754) - Successors are currently known for: Map(SEQ::sv3::1220d37f600d... -> SequencerConnectionSuccessor(sequencerId = SEQ::sv3::1220d37f600d..., synchronizerId = global-domain::1220119e44dd..., connection = localhost:27308)) 
2025-12-09T13:36:49.453Z [⋮] DEBUG - c.d.c.p.t.SequencerConnectionSuccessorListener:participant=sv1Participant (47136809f7e1b4331a9ea18b1674ad8e-MessageDispatcher.handle-78dee7960ab6fe50-a05302c2a5444754) - Some sequencer have not yet announced their endpoints on the successor synchronizer: Set(SEQ::sv1::1220a2dc2d13..., SEQ::sv2::122063861133..., SEQ::sv4::1220698395c2...) 
2025-12-09T13:36:49.631Z [⋮] DEBUG - c.d.c.p.t.SequencerConnectionSuccessorListener:participant=sv1Participant (c5ce57383f6c5f4b0cc4b41a46e0e400-MessageDispatcher.handle-ff5161311b3afa4c-ce0a3c0879b23914) - Checking whether the participant can migrate Synchronizer 'global' from global-domain::1220119e44dd9767c1b54ebebe11dae807896aba2e1d674c7ce9c7a7865fdd7f0e97::34-0 to global-domain::1220119e44dd...::34-1 
2025-12-09T13:36:49.637Z [⋮] DEBUG - c.d.c.p.t.SequencerConnectionSuccessorListener:participant=sv1Participant (c5ce57383f6c5f4b0cc4b41a46e0e400-MessageDispatcher.handle-ff5161311b3afa4c-ce0a3c0879b23914) - Successors are currently known for: Map(SEQ::sv3::1220d37f600d... -> SequencerConnectionSuccessor(sequencerId = SEQ::sv3::1220d37f600d..., synchronizerId = global-domain::1220119e44dd..., connection = localhost:27308), SEQ::sv2::122063861133... -> SequencerConnectionSuccessor(sequencerId = SEQ::sv2::122063861133..., synchronizerId = global-domain::1220119e44dd..., connection = localhost:27208)) 
2025-12-09T13:36:49.740Z [⋮] DEBUG - c.d.c.p.t.SequencerConnectionSuccessorListener:participant=sv1Participant (bd4317e84f3563560df917c19f0bfca3-MessageDispatcher.handle-8d83e87744c1642e-5da28e2e1becea17) - Successors are currently known for: Map(SEQ::sv3::1220d37f600d... -> SequencerConnectionSuccessor(sequencerId = SEQ::sv3::1220d37f600d..., synchronizerId = global-domain::1220119e44dd..., connection = localhost:27308), SEQ::sv2::122063861133... -> SequencerConnectionSuccessor(sequencerId = SEQ::sv2::122063861133..., synchronizerId = global-domain::1220119e44dd..., connection = localhost:27208), SEQ::sv1::1220a2dc2d13... -> SequencerConnectionSuccessor(sequencerId = SEQ::sv1::1220a2dc2d13..., synchronizerId = global-domain::1220119e44dd..., connection = localhost:27108)) 
2025-12-09 13:36:49,773 [canton-env-ec-1140] DEBUG c.d.c.p.t.SequencerConnectionSuccessorListener:participant=sv1Participant tid:4721bbf8cde8e65d8192cdc5c12d6667 - Checking whether the participant can migrate Synchronizer 'global' from global-domain::1220119e44dd9767c1b54ebebe11dae807896aba2e1d674c7ce9c7a7865fdd7f0e97::34-0 to global-domain::1220119e44dd...::34-1
2025-12-09 13:36:49,773 [canton-env-ec-1140] DEBUG c.d.c.p.t.SequencerConnectionSuccessorListener:participant=sv1Participant tid:4721bbf8cde8e65d8192cdc5c12d6667 - Configured sequencer connections: Set(SEQ::sv1::1220a2dc2d13..., SEQ::sv2::122063861133..., SEQ::sv3::1220d37f600d..., SEQ::sv4::1220698395c2...)
2025-12-09T13:36:49.773Z [⋮] DEBUG - c.d.c.p.t.SequencerConnectionSuccessorListener:participant=sv1Participant (4721bbf8cde8e65d8192cdc5c12d6667-MessageDispatcher.handle-eb5f3eea9b2bef51-d1539cc7b8309b23) - Checking whether the participant can migrate Synchronizer 'global' from global-domain::1220119e44dd9767c1b54ebebe11dae807896aba2e1d674c7ce9c7a7865fdd7f0e97::34-0 to global-domain::1220119e44dd...::34-1 
2025-12-09T13:36:49.773Z [⋮] DEBUG - c.d.c.p.t.SequencerConnectionSuccessorListener:participant=sv1Participant (4721bbf8cde8e65d8192cdc5c12d6667-MessageDispatcher.handle-eb5f3eea9b2bef51-d1539cc7b8309b23) - Configured sequencer connections: Set(SEQ::sv1::1220a2dc2d13..., SEQ::sv2::122063861133..., SEQ::sv3::1220d37f600d..., SEQ::sv4::1220698395c2...) 
2025-12-09 13:36:49,774 [canton-env-ec-1179] DEBUG c.d.c.p.t.SequencerConnectionSuccessorListener:participant=sv1Participant tid:4721bbf8cde8e65d8192cdc5c12d6667 - Successors are currently known for: Map(SEQ::sv3::1220d37f600d... -> SequencerConnectionSuccessor(sequencerId = SEQ::sv3::1220d37f600d..., synchronizerId = global-domain::1220119e44dd..., connection = localhost:27308), SEQ::sv2::122063861133... -> SequencerConnectionSuccessor(sequencerId = SEQ::sv2::122063861133..., synchronizerId = global-domain::1220119e44dd..., connection = localhost:27208), SEQ::sv1::1220a2dc2d13... -> SequencerConnectionSuccessor(sequencerId = SEQ::sv1::1220a2dc2d13..., synchronizerId = global-domain::1220119e44dd..., connection = localhost:27108), SEQ::sv4::1220698395c2... -> SequencerConnectionSuccessor(sequencerId = SEQ::sv4::1220698395c2..., synchronizerId = global-domain::1220119e44dd..., connection = localhost:27408))
2025-12-09T13:36:49.774Z [⋮] DEBUG - c.d.c.p.t.SequencerConnectionSuccessorListener:participant=sv1Participant (4721bbf8cde8e65d8192cdc5c12d6667-MessageDispatcher.handle-eb5f3eea9b2bef51-d1539cc7b8309b23) - Successors are currently known for: Map(SEQ::sv3::1220d37f600d... -> SequencerConnectionSuccessor(sequencerId = SEQ::sv3::1220d37f600d..., synchronizerId = global-domain::1220119e44dd..., connection = localhost:27308), SEQ::sv2::122063861133... -> SequencerConnectionSuccessor(sequencerId = SEQ::sv2::122063861133..., synchronizerId = global-domain::1220119e44dd..., connection = localhost:27208), SEQ::sv1::1220a2dc2d13... -> SequencerConnectionSuccessor(sequencerId = SEQ::sv1::1220a2dc2d13..., synchronizerId = global-domain::1220119e44dd..., connection = localhost:27108), SEQ::sv4::1220698395c2... -> SequencerConnectionSuccessor(sequencerId = SEQ::sv4::1220698395c2..., synchronizerId = global-domain::1220119e44dd..., connection = localhost:27408)) 
2025-12-09 13:36:49,775 [canton-env-ec-1145] DEBUG c.d.c.p.t.SequencerConnectionSuccessorListener:participant=sv1Participant tid:4721bbf8cde8e65d8192cdc5c12d6667 - New set of sequencer connections for successors: List(GrpcSequencerConnection(sequencerAlias = Sequencer 'Digital-Asset-Eng-3', sequencerId = SEQ::sv3::1220d37f600d..., endpoints = http://localhost:27308), GrpcSequencerConnection(sequencerAlias = Sequencer 'Digital-Asset-Eng-2', sequencerId = SEQ::sv2::122063861133..., endpoints = http://localhost:27208), GrpcSequencerConnection(sequencerAlias = Sequencer 'Digital-Asset-2', sequencerId = SEQ::sv1::1220a2dc2d13..., endpoints = http://localhost:27108), GrpcSequencerConnection(sequencerAlias = Sequencer 'Digital-Asset-Eng-4', sequencerId = SEQ::sv4::1220698395c2..., endpoints = http://localhost:27408))
2025-12-09 13:36:49,775 [canton-env-ec-1145] DEBUG c.d.c.p.s.d.DbSynchronizerConnectionConfigStore:participant=sv1Participant tid:4721bbf8cde8e65d8192cdc5c12d6667 - Inserting connection for (Synchronizer 'global', global-domain::1220119e44dd9767c1b54ebebe11dae807896aba2e1d674c7ce9c7a7865fdd7f0e97::34-1) into the store
2025-12-09T13:36:49.775Z [⋮] DEBUG - c.d.c.p.t.SequencerConnectionSuccessorListener:participant=sv1Participant (4721bbf8cde8e65d8192cdc5c12d6667-MessageDispatcher.handle-eb5f3eea9b2bef51-d1539cc7b8309b23) - New set of sequencer connections for successors: List(GrpcSequencerConnection(sequencerAlias = Sequencer 'Digital-Asset-Eng-3', sequencerId = SEQ::sv3::1220d37f600d..., endpoints = http://localhost:27308), GrpcSequencerConnection(sequencerAlias = Sequencer 'Digital-Asset-Eng-2', sequencerId = SEQ::sv2::122063861133..., endpoints = http://localhost:27208), GrpcSequencerConnection(sequencerAlias = Sequencer 'Digital-Asset-2', sequencerId = SEQ::sv1::1220a2dc2d13..., endpoints = http://localhost:27108), GrpcSequencerConnection(sequencerAlias = Sequencer 'Digital-Asset-Eng-4', sequencerId = SEQ::sv4::1220698395c2..., endpoints = http://localhost:27408)) 
2025-12-09T13:36:49.775Z [⋮] DEBUG - c.d.c.p.s.d.DbSynchronizerConnectionConfigStore:participant=sv1Participant (4721bbf8cde8e65d8192cdc5c12d6667-MessageDispatcher.handle-eb5f3eea9b2bef51-d1539cc7b8309b23) - Inserting connection for (Synchronizer 'global', global-domain::1220119e44dd9767c1b54ebebe11dae807896aba2e1d674c7ce9c7a7865fdd7f0e97::34-1) into the store 
2025-12-09 13:36:49,799 [canton-env-ec-69] DEBUG c.d.c.p.s.SynchronizerConnectionsManager:participant=sv1Participant tid:4721bbf8cde8e65d8192cdc5c12d6667 - About to perform handshake with synchronizer: global-domain::1220119e44dd...::34-1
2025-12-09T13:36:49.799Z [⋮] DEBUG - c.d.c.p.s.SynchronizerConnectionsManager:participant=sv1Participant (4721bbf8cde8e65d8192cdc5c12d6667-MessageDispatcher.handle-eb5f3eea9b2bef51-d1539cc7b8309b23) - About to perform handshake with synchronizer: global-domain::1220119e44dd...::34-1 
2025-12-09 13:36:49,803 [canton-env-ec-1136] DEBUG c.d.c.p.s.SynchronizerConnectionsManager:participant=sv1Participant tid:4721bbf8cde8e65d8192cdc5c12d6667 - Performing handshake with synchronizer with id global-domain::1220119e44dd9767c1b54ebebe11dae807896aba2e1d674c7ce9c7a7865fdd7f0e97::34-1 and config: SynchronizerConnectionConfig(
  synchronizer = Synchronizer 'global',
  sequencerConnections = SequencerConnections(
    connections = Map(
      Sequencer 'Digital-Asset-Eng-3' -> GrpcSequencerConnection(sequencerAlias = Sequencer 'Digital-Asset-Eng-3', sequencerId = SEQ::sv3::1220d37f600d..., endpoints = http://localhost:27308),
      Sequencer 'Digital-Asset-Eng-2' -> GrpcSequencerConnection(sequencerAlias = Sequencer 'Digital-Asset-Eng-2', sequencerId = SEQ::sv2::122063861133..., endpoints = http://localhost:27208),
      Sequencer 'Digital-Asset-2' -> GrpcSequencerConnection(sequencerAlias = Sequencer 'Digital-Asset-2', sequencerId = SEQ::sv1::1220a2dc2d13..., endpoints = http://localhost:27108),
      Sequencer 'Digital-Asset-Eng-4' -> GrpcSequencerConnection(sequencerAlias = Sequencer 'Digital-Asset-Eng-4', sequencerId = SEQ::sv4::1220698395c2..., endpoints = http://localhost:27408)
    ),
    sequencer trust threshold = 2,
    sequencer liveness margin = 1,
    submission request amplification = SubmissionRequestAmplification(factor = 2, patience = 10s),
    sequencer connection pool delays = SequencerConnectionPoolDelays(minRestartDelay = 0.01s, maxRestartDelay = 10s, warnValidationDelay = 20s, subscriptionRequestDelay = 1s)
  ),
  manualConnect = false,
  physicalSynchronizerId = global-domain::1220119e44dd...::34-1,
  timeTracker = SynchronizerTimeTrackerConfig(minObservationDuration = 10s)
)
2025-12-09T13:36:49.805Z [⋮] DEBUG - c.d.c.s.GrpcSequencerConnectionXPoolFactory:participant=sv1Participant/synchronizerAlias=Synchronizer 'global' (4721bbf8cde8e65d8192cdc5c12d6667-MessageDispatcher.handle-eb5f3eea9b2bef51-d1539cc7b8309b23) - poolConfig = SequencerConnectionXPoolConfig(
  connections = Seq(
    ConnectionXConfig(name = 'Digital-Asset-Eng-3-0', endpoint = http://localhost:27308, transportSecurity = false, customTrustCertificates = false, expectedSequencerId = SEQ::sv3::1220d37f600d...),
    ConnectionXConfig(name = 'Digital-Asset-Eng-2-0', endpoint = http://localhost:27208, transportSecurity = false, customTrustCertificates = false, expectedSequencerId = SEQ::sv2::122063861133...),
    ConnectionXConfig(name = 'Digital-Asset-2-0', endpoint = http://localhost:27108, transportSecurity = false, customTrustCertificates = false, expectedSequencerId = SEQ::sv1::1220a2dc2d13...),
    ConnectionXConfig(name = 'Digital-Asset-Eng-4-0', endpoint = http://localhost:27408, transportSecurity = false, customTrustCertificates = false, expectedSequencerId = SEQ::sv4::1220698395c2...)
  ),
  trustThreshold = 2,
  minRestartConnectionDelay = 0.01s,
  maxRestartConnectionDelay = 10s,
  warnConnectionValidationDelay = 20s,
  expectedPSIdO = global-domain::1220119e44dd...::34-1
) 
2025-12-09T13:36:49.808Z [⋮] DEBUG - c.d.c.c.s.g.SequencerInfoLoader:participant=sv1Participant (4721bbf8cde8e65d8192cdc5c12d6667-MessageDispatcher.handle-eb5f3eea9b2bef51-d1539cc7b8309b23) - Loading sequencer info entries with 4 connections (Digital-Asset-Eng-3=localhost:27308;Digital-Asset-Eng-2=localhost:27208;Digital-Asset-2=localhost:27108;Digital-Asset-Eng-4=localhost:27408), parallelism 4, threshold Some(3), synchronizer global 
2025-12-09T13:36:50.006Z [⋮] DEBUG - c.d.c.c.s.g.SequencerInfoLoader:participant=sv1Participant (4721bbf8cde8e65d8192cdc5c12d6667-MessageDispatcher.handle-eb5f3eea9b2bef51-d1539cc7b8309b23) - Loaded sufficiently many sequencer info entries (3) in synchronizer Synchronizer 'global' 
2025-12-09T13:36:50.144Z [⋮] DEBUG - c.d.c.s.c.t.GrpcSequencerClientTransportPekko:participant=sv1Participant/psid=global-domain::1220119e44dd::34-1/sequencerAlias=Digital-Asset-Eng-3 (4721bbf8cde8e65d8192cdc5c12d6667-MessageDispatcher.handle-eb5f3eea9b2bef51-d1539cc7b8309b23) - Request downloadTopologyStateForInitHash has succeeded for sequencer. 
2025-12-09T13:36:50.155Z [⋮] INFO - c.d.c.s.c.BftTopologyForInitDownloader$:participant=sv1Participant/psid=global-domain::1220119e44dd::34-1 (4721bbf8cde8e65d8192cdc5c12d6667-MessageDispatcher.handle-eb5f3eea9b2bef51-d1539cc7b8309b23) - Expecting topology state for init with hash TopologyStateForInitHashResponse(SHA-256:0b31faa7070c...) 
2025-12-09T13:36:50.155Z [⋮] DEBUG - c.d.c.s.c.RichSequencerClientImpl:participant=sv1Participant/psid=global-domain::1220119e44dd::34-1 (4721bbf8cde8e65d8192cdc5c12d6667-MessageDispatcher.handle-eb5f3eea9b2bef51-d1539cc7b8309b23) - Attempting to download topology state for init from SEQ::sv2::122063861133... (already tried: Set()) 
2025-12-09T13:36:50.212Z [⋮] DEBUG - c.d.c.s.c.t.GrpcSequencerClientTransportPekko:participant=sv1Participant/psid=global-domain::1220119e44dd::34-1/sequencerAlias=Digital-Asset-Eng-2 (4721bbf8cde8e65d8192cdc5c12d6667-MessageDispatcher.handle-eb5f3eea9b2bef51-d1539cc7b8309b23) - Downloaded topology state for initialization with last change timestamp at Some(2025-12-09T13:34:21.862471Z): 11 transactions 
2025-12-09T13:36:50.213Z [⋮] INFO - c.d.c.s.c.BftTopologyForInitDownloader$:participant=sv1Participant/psid=global-domain::1220119e44dd::34-1 (4721bbf8cde8e65d8192cdc5c12d6667-MessageDispatcher.handle-eb5f3eea9b2bef51-d1539cc7b8309b23) - Successfully downloaded topology state for init with hash matching expected SHA-256:0b31faa7070c... 
2025-12-09 13:36:50,259 [canton-env-ec-1136] INFO  c.d.c.p.t.SequencerConnectionSuccessorListener:participant=sv1Participant tid:4721bbf8cde8e65d8192cdc5c12d6667 - Handshake with global-domain::1220119e44dd...::34-1 was successful
2025-12-09T13:36:50.259Z [⋮] INFO - c.d.c.p.t.SequencerConnectionSuccessorListener:participant=sv1Participant (4721bbf8cde8e65d8192cdc5c12d6667-MessageDispatcher.handle-eb5f3eea9b2bef51-d1539cc7b8309b23) - Handshake with global-domain::1220119e44dd...::34-1 was successful 
2025-12-09T13:37:39.457Z [⋮] DEBUG - c.d.c.p.s.CantonSyncService:participant=sv1Participant (f98e5494f3fbb5e7ccaf8d9b3d11e8d7-MessageDispatcher.handle-bb5c7a0f6c470b14-753f6f519556ed70) - Checking whether admin workflows need to be vetted still. 
2025-12-09T13:37:39.460Z [⋮] DEBUG - c.d.c.p.s.d.DbPhysicalSyncPersistentState$$anon$1:participant=sv1Participant/psid=global-domain::1220119e44dd::34-1 (f98e5494f3fbb5e7ccaf8d9b3d11e8d7-MessageDispatcher.handle-bb5c7a0f6c470b14-753f6f519556ed70) - Attempting to build, sign, and Replace VettedPackages(
2025-12-09T13:37:39.511Z [⋮] DEBUG - c.d.c.t.s.d.DbTopologyStore:participant=sv1Participant/psid=global-domain::1220119e44dd::34-1 (75d6c4617bbc5f0ed547941eb942d8d4-SequencerClient.sendAsync-468b220715006391-360bdf3b6ffa48df) - Querying transactions as of 2025-12-09T13:34:22.112472Z for types Set(SynchronizerTrustCertificate) with filters for uids sv1::1220b2a0f9a8...; op Replace 
2025-12-09 13:37:39,529 [canton-env-ec-1168] WARN  c.d.c.s.c.t.GrpcSequencerClientTransportPekko:participant=sv1Participant/psid=global-domain::1220119e44dd::34-1/sequencerAlias=Digital-Asset-2 tid:75d6c4617bbc5f0ed547941eb942d8d4 - Request failed for sequencer.

[rgugliel-da]

Code: https://github.com/hyperledger-labs/splice/pull/3337/files#diff-52cff2d2e4c16ae483b97a794e15875f19462ede6f44780818b9c47416c3b02b

[rgugliel-da]

Additional insight from Christian: "So AFAIR the changes I did were related to an issue that connecting in "handshake only" would incorrectly trigger the vetting of the admin workflows. The change made it that we trigger the vetting only when we actually start the synchronizer.
From a quick look at the synchronizer start, regarding topology, it seems we only wait for the participant to be in the topology before proceeding."

[nicu-da]

2025-12-11T14:03:20.338Z [⋮] DEBUG - c.d.c.l.a.ApiRequestLogger:participant=aliceParticipant (0dc0690a9babc56c4486b1965b7ec9c7---1fa7b358c8b9131e) - Request com.digitalasset.canton.admin.participant.v30.SynchronizerConnectivityService/ReconnectSynchronizer by grpc:/127.0.0.1:59304: auth claims: Claims(List(ClaimPublic, ClaimAdmin, ClaimActAsAnyParty),None,None,None,None,false) 
2025-12-11T14:03:20.338Z [⋮] DEBUG - c.d.c.l.a.ApiRequestLogger:participant=aliceParticipant (0dc0690a9babc56c4486b1965b7ec9c7---1fa7b358c8b9131e) - Request com.digitalasset.canton.admin.participant.v30.SynchronizerConnectivityService/ReconnectSynchronizer by grpc:/127.0.0.1:59304: received a message ReconnectSynchronizerRequest(global, false) 
2025-12-11T14:03:20.339Z [⋮] DEBUG - c.d.c.p.s.SynchronizerConnectionsManager:participant=aliceParticipant (0dc0690a9babc56c4486b1965b7ec9c7---1fa7b358c8b9131e) - Trying to connect PAR::aliceValidator::122069394552... to Synchronizer 'global' 
2025-12-11T14:03:20.339Z [⋮] DEBUG - c.d.c.p.s.SynchronizerConnectionsManager:participant=aliceParticipant (0dc0690a9babc56c4486b1965b7ec9c7---1fa7b358c8b9131e) - About to connect to synchronizer: global 
2025-12-11T14:03:20.340Z [⋮] DEBUG - c.d.c.p.s.SynchronizerConnectionsManager:participant=aliceParticipant (0dc0690a9babc56c4486b1965b7ec9c7---1fa7b358c8b9131e) - Connecting to synchronizer with id UnknownPhysicalSynchronizerId config: SynchronizerConnectionConfig(
  synchronizer = Synchronizer 'global',
  sequencerConnections = SequencerConnections(
    connections = Map(
      Sequencer 'Digital-Asset-2' -> GrpcSequencerConnection(sequencerAlias = Sequencer 'Digital-Asset-2', endpoints = http://localhost:5108),
      Sequencer 'Digital-Asset-Eng-2' -> GrpcSequencerConnection(sequencerAlias = Sequencer 'Digital-Asset-Eng-2', endpoints = http://localhost:5208),
      Sequencer 'Digital-Asset-Eng-3' -> GrpcSequencerConnection(sequencerAlias = Sequencer 'Digital-Asset-Eng-3', endpoints = http://localhost:5308),
      Sequencer 'Digital-Asset-Eng-4' -> GrpcSequencerConnection(sequencerAlias = Sequencer 'Digital-Asset-Eng-4', endpoints = http://localhost:5408)
    ),
    sequencer trust threshold = 2,
    sequencer liveness margin = 1,
    submission request amplification = SubmissionRequestAmplification(factor = 2, patience = 10s),
    sequencer connection pool delays = SequencerConnectionPoolDelays(minRestartDelay = 0.01s, maxRestartDelay = 10s, warnValidationDelay = 20s, subscriptionRequestDelay = 1s)
  ),
  manualConnect = false,
  timeTracker = SynchronizerTimeTrackerConfig(minObservationDuration = 10s)
) 
2025-12-11T14:03:20.340Z [⋮] DEBUG - c.d.c.s.GrpcSequencerConnectionXPoolFactory:participant=aliceParticipant/synchronizerAlias=Synchronizer 'global' (0dc0690a9babc56c4486b1965b7ec9c7---1fa7b358c8b9131e) - poolConfig = SequencerConnectionXPoolConfig(
  connections = Seq(
    ConnectionXConfig(name = 'Digital-Asset-2-0', endpoint = http://localhost:5108, transportSecurity = false, customTrustCertificates = false),
    ConnectionXConfig(name = 'Digital-Asset-Eng-2-0', endpoint = http://localhost:5208, transportSecurity = false, customTrustCertificates = false),
    ConnectionXConfig(name = 'Digital-Asset-Eng-3-0', endpoint = http://localhost:5308, transportSecurity = false, customTrustCertificates = false),
    ConnectionXConfig(name = 'Digital-Asset-Eng-4-0', endpoint = http://localhost:5408, transportSecurity = false, customTrustCertificates = false)
  ),
  trustThreshold = 2,
  minRestartConnectionDelay = 0.01s,
  maxRestartConnectionDelay = 10s,
  warnConnectionValidationDelay = 20s
) 
2025-12-11T14:03:22.033Z [⋮] DEBUG - c.d.c.p.s.CantonSyncService:participant=aliceParticipant (0dc0690a9babc56c4486b1965b7ec9c7---1fa7b358c8b9131e) - Vetting admin workflows on global-domain::1220562a6b2e... 
2025-12-11T14:03:22.033Z [⋮] DEBUG - c.d.c.p.s.CantonSyncService:participant=aliceParticipant (0dc0690a9babc56c4486b1965b7ec9c7---1fa7b358c8b9131e) - Checking whether admin workflows need to be vetted still. 
2025-12-11T14:03:22.035Z [⋮] DEBUG - c.d.c.p.s.d.DbPhysicalSyncPersistentState$$anon$1:participant=aliceParticipant/psid=global-domain::1220562a6b2e::34-0 (0dc0690a9babc56c4486b1965b7ec9c7---1fa7b358c8b9131e) - Attempting to build, sign, and Replace VettedPackages(
  participantId = PAR::aliceValidator::122069394552...,
  packages = Seq(
    VettedPackage(packageId = 6f8e6085f576..., unbounded),
    VettedPackage(packageId = 60c61c542207..., unbounded),
    VettedPackage(packageId = a1fa18133ae4..., unbounded),
    VettedPackage(packageId = cae345b5500e..., unbounded),
    VettedPackage(packageId = c3bb0c5d0479..., unbounded),
    VettedPackage(packageId = fcee8dfc1b81..., unbounded),
    VettedPackage(packageId = c280cc3ef501..., unbounded),
    VettedPackage(packageId = 9e70a8b3510d..., unbounded),
    VettedPackage(packageId = 91e167fa7a25..., unbounded),
    VettedPackage(packageId = b70db8369e1c..., unbounded),
    VettedPackage(packageId = f181cd661f7a..., unbounded),
    VettedPackage(packageId = bde4bd30749e..., unbounded),
    VettedPackage(packageId = 19f0df5fdaf5..., unbounded),
    VettedPackage(packageId = ab068e2f920d..., unbounded),
    VettedPackage(packageId = 86d888f34152..., unbounded),
    VettedPackage(packageId = de2cc2f90eb5..., unbounded),
    VettedPackage(packageId = 99ea07e101ed..., unbounded),
    VettedPackage(packageId = 5aee9b21b8e9..., unbounded),
    VettedPackage(packageId = bfda48f9aa2c..., unbounded),
    VettedPackage(packageId = 52854220dc19..., unbounded),
    VettedPackage(packageId = fa79192fe1cc..., unbounded),
    VettedPackage(packageId = 6da1f43a10a1..., unbounded),
    VettedPackage(packageId = 590736e6f7bc..., unbounded),
    VettedPackage(packageId = 0e4a572ab1fb..., unbounded),
    VettedPackage(packageId = d095a2ccf6dd..., unbounded),
    VettedPackage(packageId = ee33fb70918e..., unbounded),
    VettedPackage(packageId = e7e0adfa881e..., unbounded),
    VettedPackage(packageId = 7adc4c2d07fa..., unbounded),
    VettedPackage(packageId = 3cde94fe9be5..., unbounded),
    VettedPackage(packageId = e5411f3d75f0..., unbounded)
  )
) with serial None()  
2025-12-11T14:03:22.037Z [⋮] DEBUG - c.d.c.t.s.d.DbTopologyStore:participant=aliceParticipant/psid=global-domain::1220562a6b2e::34-0 (0dc0690a9babc56c4486b1965b7ec9c7---1fa7b358c8b9131e) - Querying transactions as of 9999-12-31T23:59:59.999999Z for types Set(VettedPackages) with filters for uids aliceValidator::122069394552...; op Replace 
2025-12-11T14:03:22.039Z [⋮] INFO - c.d.c.p.a.s.a.PackageUpgradeValidator:participant=aliceParticipant (0dc0690a9babc56c4486b1965b7ec9c7---1fa7b358c8b9131e) - Typechecking upgrades for lineage of package-name canton-builtin-admin-workflow-ping. 
2025-12-11T14:03:22.039Z [⋮] INFO - c.d.c.p.a.s.a.PackageUpgradeValidator:participant=aliceParticipant (0dc0690a9babc56c4486b1965b7ec9c7---1fa7b358c8b9131e) - Typechecking upgrades for lineage of package-name canton-builtin-admin-workflow-ping succeeded. 
2025-12-11T14:03:22.046Z [⋮] INFO - c.d.c.t.TopologyStateProcessor:participant=aliceParticipant/psid=global-domain::1220562a6b2e::34-0 (0dc0690a9babc56c4486b1965b7ec9c7---1fa7b358c8b9131e) - Enqueued topology transactions:
ValidatedTopologyTransaction(
  SignedTopologyTransaction(
    TopologyTransaction(
      VettedPackages(
        participantId = PAR::aliceValidator::122069394552...,
        packages = Seq(
          VettedPackage(packageId = 6f8e6085f576..., unbounded),
          VettedPackage(packageId = 60c61c542207..., unbounded),
          VettedPackage(packageId = a1fa18133ae4..., unbounded),
          VettedPackage(packageId = cae345b5500e..., unbounded),
          VettedPackage(packageId = c3bb0c5d0479..., unbounded),
          VettedPackage(packageId = fcee8dfc1b81..., unbounded),
          VettedPackage(packageId = c280cc3ef501..., unbounded),
          VettedPackage(packageId = 9e70a8b3510d..., unbounded),
          VettedPackage(packageId = 91e167fa7a25..., unbounded),
          VettedPackage(packageId = b70db8369e1c..., unbounded),
          VettedPackage(packageId = f181cd661f7a..., unbounded),
          VettedPackage(packageId = bde4bd30749e..., unbounded),
          VettedPackage(packageId = 19f0df5fdaf5..., unbounded),
          VettedPackage(packageId = ab068e2f920d..., unbounded),
          VettedPackage(packageId = 86d888f34152..., unbounded),
          VettedPackage(packageId = de2cc2f90eb5..., unbounded),
          VettedPackage(packageId = 99ea07e101ed..., unbounded),
          VettedPackage(packageId = 5aee9b21b8e9..., unbounded),
          VettedPackage(packageId = bfda48f9aa2c..., unbounded),
          VettedPackage(packageId = 52854220dc19..., unbounded),
          VettedPackage(packageId = fa79192fe1cc..., unbounded),
          VettedPackage(packageId = 6da1f43a10a1..., unbounded),
          VettedPackage(packageId = 590736e6f7bc..., unbounded),
          VettedPackage(packageId = 0e4a572ab1fb..., unbounded),
          VettedPackage(packageId = d095a2ccf6dd..., unbounded),
          VettedPackage(packageId = ee33fb70918e..., unbounded),
          VettedPackage(packageId = e7e0adfa881e..., unbounded),
          VettedPackage(packageId = 7adc4c2d07fa..., unbounded),
          VettedPackage(packageId = 3cde94fe9be5..., unbounded),
          VettedPackage(packageId = e5411f3d75f0..., unbounded)
        )
      ),
      serial = 1,
      operation = Replace,
      hash = SHA-256:2abcc6bb8d2b...
    ),
    signatures = 122069394552...
  )
) 
2025-12-11T14:07:00.029Z [⋮] DEBUG - c.d.c.t.QueueBasedSynchronizerOutbox:participant=aliceParticipant/psid=global-domain::1220562a6b2e::34-1 (1929e6fd48f9eb6e683396cbc7efb31a-flush_outbox--f20d77e5ed9d08d7) - Invoked flush with queue size 1 
2025-12-11T14:07:00.154Z [⋮] DEBUG - c.d.c.t.s.d.DbTopologyStore:sequencer=sv4StandaloneSequencer/store=global-domain::1220562a6b2e::34-1 (1929e6fd48f9eb6e683396cbc7efb31a-Sequencer.sendAsyncSigned-c72fba1ae3cae5f6-41456153a253cb31) - Querying transactions as of 0001-01-01T00:00:00.000002Z for types Set(OwnerToKeyMapping) with filters for uids sv4::1220e8af9696...; op Replace 
2025-12-11 14:07:00,162 [canton-env-ec-482] ERROR c.d.c.s.s.b.b.c.m.a.AvailabilityModule:sequencer=sv4StandaloneSequencer/psid=global-domain::1220562a6b2e::34-1 tid:1929e6fd48f9eb6e683396cbc7efb31a - Failed to sign local batches: Vector(KeyNotAvailable(owner = SEQ::sv4::1220e8af9696..., key purpose = signing, timestamp = 0001-01-01T00:00:00.000002Z, candidates = Seq()))
2025-12-11T14:07:00.162Z [⋮] ERROR - c.d.c.s.s.b.b.c.m.a.AvailabilityModule:sequencer=sv4StandaloneSequencer/psid=global-domain::1220562a6b2e::34-1 (1929e6fd48f9eb6e683396cbc7efb31a-Sequencer.sendAsyncSigned-c72fba1ae3cae5f6-41456153a253cb31) - Failed to sign local batches: Vector(KeyNotAvailable(owner = SEQ::sv4::1220e8af9696..., key purpose = signing, timestamp = 0001-01-01T00:00:00.000002Z, candidates = Seq())) [
workaround does not work, the participant tries to vet on a the reconnect and the end result is even worse as it bricks the sequencer fully.
things to fix:
sync this in the participant, not sure why it does not see the old topology transaction already
the sequencer should not break that hard if it cannot find the signing keys 

so this confirms it even if we add the switch to the new ps later it does not help.
what's worse is that it fully breaks in teh sequencer this time and crashes the sequencer.

full logs