Load shedding as back pressure from DA

[mzabaluev]

The v1 load shedding as implemented in #255 is leaky: it does not account for transactions that don't make it to the DA for any reason, including the DA light node garbage collection implemented in #628.

The proposed solution is to move the pressure valve entirely onto m1-da-light-node. The simplest way I see to implement this is to move the in-flight counter and the load shedding logic to m1-da-light-node mempool. The transaction pipe task in the node will add transactions to its in-memory mempool and try to submit blobs to the da-light-node, but an "enhance your calm" error status from the latter will cause it to reject incoming transactions with a "mempool is full" error until da-light-node is able to accept blobs again.

[l-monninger]

The v1 load shedding as implemented in #255 is leaky: it does not account for transactions that don't make it to the DA for any reason, including the DA light node garbage collection implemented in #628.

Yeah, it's leaky. GC plus this in particular equals problems. Even without GC, any other fallibility results in leakiness.

A couple additional ideas:

1. We could GC the in-flight itself.
2. Pressure value entirely on the m1-da-light-node is tricky to report MempoolFull from because of asynchronicity. Short term, we could turn off the in-flight limit and just allow the transaction to fail and timeouts to expire.

[l-monninger]

I think I would probably GC the in-flight itself tbh. Allows us to keep working with these as independent systems and still report errors.

[mzabaluev]

2. Pressure value entirely on the m1-da-light-node is tricky to report MempoolFull from because of asynchronicity.

The idea is, it does not matter which transactions we reject on ingress when the "mempool is full" condition is reached. The da-light-node reporting as full is just as good a trigger as an internal counter. This would also help prevent unlimited growth in da-light-node mempool storage from going unnoticed (if GC does not work adequately for some reason). However, it's not an innate limit on m1-da-light-node capacity if we assume sufficiently large storage for the rocksdb mempool.

[l-monninger]

The issue is the actual API response can't happen at that stage. It either has to happen on submission, mempool query, or VM handling.