StateAccessViolation: Shared Memory buffer is not in the correct state

[pavel1murat]

we are running artdaq in a configuration with 6 boardreaders(BR) on 3 nodes, an event builder (EB09), and one data logger,
reading three tracker stations at an input rate of 1 - 2.5 kHz using an emulated CFO to throttle down the rate.

1-2 min into the the run, however after hundreds of thousands of processes events, the event builder reports an inconsistency in the shared memory bookkeeping and stops receiving events.

the problem doesn't get excited when we read only one station, in a configuration with two boardreaders, one event builder, and a data logger , all running on the same node.

The diagnostics about duplicate events is likely related to the CFO handling of the EMW train, with the null heartbeat in the end of the pulse train resulting in an event with a duplicate EW tag, and it doesn't cause any problem for the configuration with two BRs

%MSG-e eb09_SharedMemoryEventManager:  Early 06-Mar-2026 18:03:33 CST pre-events SharedMemoryEventManager.cc:1153
Event 84084 has already been completed and released to art! Check configuration for inconsistent Fragment count per event!
%MSG
%MSG-e eb09_SharedMemoryEventManager:  Early 06-Mar-2026 18:03:33 CST pre-events SharedMemoryEventManager.cc:278
Dropping fragment with sequence id 84084 and fragment id 0 because data taking has already passed this event.
%MSG
%MSG-e SharedMemoryManager:  Early 06-Mar-2026 18:03:33 CST pre-events SharedMemoryManager.cc:43
A signal of type 11 was caught by SharedMemoryManager. Detaching all Shared Memory segments, then proceeding with default handlers!
%MSG
%MSG-e SharedMemoryManager:  Early 06-Mar-2026 18:03:33 CST pre-events SharedMemoryManager.cc:1301
StateAccessViolation: Shared Memory buffer is not in the correct state! (expected Writing, actual Full)
%MSG
%MSG-e SharedMemoryManager:  Early 06-Mar-2026 18:03:33 CST pre-events SharedMemoryManager.cc:65
Restoring default signal handler
%MSG
%MSG-e SharedMemoryManager:  Early 06-Mar-2026 18:03:33 CST pre-events SharedMemoryManager.cc:1301
eb09SharedMemoryEventManager: getBufferForSequenceID_ returned -1 in DoneWritingFragment. This indicates a possible mismatch between expected Fragment count and the actual number of Fragments received.
%MSG
%MSG-e SharedMemoryManager:  Early 06-Mar-2026 18:03:33 CST pre-events SharedMemoryManager.cc:43
A signal of type 6 was caught by SharedMemoryManager. Detaching all Shared Memory segments, then proceeding with default handlers!
%MSG

[pavel1murat]

the issue is resolved and can be closed.

• the immediate cause: in case of an empty read , the user boardreader plugin was returning to the caller (artdaq boardreader) inconsistently formed fragments, which artdaq was failing to properly discard. The user code should not have been returning any fragments at all
• the underlying cause: absence of the artdaq documentation. Specification of what artdaq expects to be returned from the user code in special cases should be a part of artdaq user manual.

[corrodis]

Thanks a lot!
Shouldn't this error still be caught by the event builder and not make it choke? Aka, events/fragments should be dropped, but shouldn't the system be able to continue with the "good" data?