RFE: Add support for `author_poolStatus` RPC

[vatsa287]

Add a new author_poolStatus to get the current status of the transaction pool for which the extrinsic is being sent to.

Implemented at paritytech/polkadot-sdk#7334.

Description

This RFE is for adding a new RPC author_poolStatus which shows the current status of the substrate transaction pool.

During one of the implementation of a queue which submits batch transaction in thousands, I did not find a way of getting the status of the pool. We set the pool limit in transactions and its size with --pool-limit & --pool-kbytes respectively.
I wanted a way to find if the transaction pool is nearing its limit or not before any new bulk transactions are sent to the pool so the transaction do not get bounced, which is my current case.

I have used author_pendingExtrinsics RPC but it does not give the bytes occupied in the pool. Since bytes occupied is also another crucial factor along with count for the limit to be enforced whether the pool is full or not.
Another downside of using this RPC is that that response is too big adding unnecessary load to the nodes and to parse the data as well.

Since we are already using status() method at various places of TransactionPool trait. Send the data as a RPC too, benefitting with low response size.

In future I wish to add the current limits on count and size to this response as well. For now I will assume that transaction-pool will be full when ready.count + future.count(10% of ready.count) > pool_limit or ready.total_bytes + future.total_bytes(10% of ready.total_bytes) > pool_limit_bytes.

Review Notes

Utilise the status() method of TransactionPool trait to expose that data as part of a new RPC call named poolStatus as part of author.
I have added tests to demonstrate the same.

I have been recommended that this RPC must be approved here for the PR at polkadot-sdk to be merged.
paritytech/polkadot-sdk#7334 (review)

Signed-off-by: Shreevatsa N i.mnshreevatsa@gmail.com

[vatsa287]

@tomaka @lexnv @jsdw Can you please review this RFE?
Thanks!

[niklasad1]

Hey @vatsa287

This spec is an successor to the "legacy rpcs" that isn't specifically aimed towards polkadot-sdk implementation details and there is no author API in the current specification. Currently, the spec is aimed toward interacting with substrate-based nodes which may be a light client such e.g. smoldot.

The proposed RPC you want would fit best in the transactionWatch API

I understand that it would nice to have such thingy to get the transaction pool status but I'm not convinced that such implement detail belong in this spec.

Similar to #138

@tomaka thoughts?

[jsdw]

If a call like this existed then I'd guess it would live under the sudo prefix, so maybe sudo_unstable_poolStatus to begin with. The idea of it being in sudo being that you are asking for details from a specific node.

(see https://paritytech.github.io/json-rpc-interface-spec/ for details of the new spec)

I wanted a way to find if the transaction pool is nearing its limit or not before any new bulk transactions are sent to the pool so the transaction do not get bounced, which is my current case.

There's a limit to how quickly these transactions can be included in the network, so my inclination here would be to have some sort of throttling approach to the submitting of transactions such that you back off a bit if they are not accepted and retry (I'm not certain what sort of error you get back in this case, but I think something like ImmediatelyDropped to signal that the transaction couldn't enter the pool due to a limit).

You could also consider breaking transactions into smaller batches so that there is more granularity with respect to how many transactions are accepted onto the pool and can be processed at one time.

[tomaka]

I wanted a way to find if the transaction pool is nearing its limit or not before any new bulk transactions are sent to the pool so the transaction do not get bounced, which is my current case.

I don't really see why it is advantageous to be able to know ahead of time whether a transaction is going to be accepted, as opposed to getting back some feedback after having sent it like is currently the case.

In any real life situation, you must monitor your transactions if you want to be sure that they have been included in the finalized chain.
It is not possible to just "fire and forget" transactions unless you accept the risk that your transactions won't be included. Given these constraints, re-submitting transactions that get rejected because the pool is full isn't a huge additional burden.

In addition to this, a fundamental flaw in the proposal is that it assumes that the size of the transaction pool only increases when you send transactions to your node through JSON-RPC. That's not the case. Transactions enter the pool because they are broadcasted by other nodes, or in case of a chain re-org.

Because of this, author_poolStatus is fundamentally racy. Even if the function says that there are 2 MiBs free in the pool, it doesn't guarantee that there will still be 2 MiBs free in the pool when you later send your transaction (even if it is just moments later). In other words, even with author_poolStatus, you would still need to handle the possibility where submitting a transaction returns an error because the pool is full. Calling author_poolStatus doesn't help.