PrimitiveWitness tx gets stuck in mempool

[dan-da]

I sent a tx and it has been stuck in the mempool for about 2 hours, and judging from logs and top the node is not making any attempt to do anything about it.

hardware: the machine has 64 logical cores and 256 Gb RAM.

Soon after sending I find this in the log, indicating that proving failed:

2025-12-11T03:09:56.046419162Z  INFO ThreadId(43) neptune_cash::application::loops::main_loop::proof_upgrader: Proof-upgrader: Start producing single proof
...
2025-12-11T03:14:48.055864147Z ERROR ThreadId(43) neptune_cash::application::loops::main_loop::proof_upgrader: UpgradeProof job failed. error: external proving process failed: out-of-process triton-vm proving job terminated without exit code. Possibly killed by OS. You might not have enough RAM to construct this proof.
2025-12-11T03:14:48.05588367Z ERROR ThreadId(43) neptune_cash::application::loops::main_loop::proof_upgrader: Consider lowering your proving capability to proof collection, in case it is set higher.

After that, there is never another attempt. So the TX just hangs out in the mempool until cleared or node is restarted.

I think the node needs to be more aggressive about attempting to get self-initiated TX sent.

In this particular case, the failure may have been related to the fact I had copied some large data into shared mem. So its possible the prover did run out of RAM. Even still, that is a temporary, recoverable event. And if the node kept trying every 10 minutes or something, it would have upgraded after I freed up enough mem.