[Full Node / RPC] Low RPS (~7) and high latency under high block size load (~30 MB), despite sufficient system resources

[elvensejo]

Our celestia-app v6.4.2 full node (running in the Docker container) is experiencing a severe RPC performance bottleneck under the current ~30 MB block size load.

The Problem:
While the node has ample disk space, network bandwidth (
Testing download speed................................................................................
Download: 5795.31 Mbit/s
Testing upload speed......................................................................................................
Upload: 2986.41 Mbit/s), and CPU/RAM resources (confirmed after the team's previous advice), the RPC interface becomes a bottleneck. We are limited to ~7 Requests Per Second, and latency for data-fetching calls is very high.

Core Question:
This suggests a limitation in the RPC layer's data processing speed for large blocks. Are there:

Specific configuration tweaks for celestia-app to improve RPC throughput?

Optimal Docker/OS tuning parameters for this use case?

Any immediate workarounds or upcoming optimizations for this scenario?

We need guidance to improve performance before the 128mb/6s blocks.

[qubelabsio]

Hello! Have you already updated your config according to the recommended changes for the v6 upgrade here?

Could you also confirm that you are running version v6.4.2 and provide a snippet of your logs or the specific RPC commands that are failing? Seeing the exact errors could potentially help the team debug this faster.

[icyverso-create]

Thank you for the response and the recommendations!

Regarding your questions:
Yes, we are running v6.4.2. We have updated our configuration according to the recommended changes for the v6 upgrade, but unfortunately, we haven't seen a significant improvement in serving the large blocks.

Current status:
The network load has recently normalized, and block sizes have decreased. For new, smaller blocks, the node is performing well again.

However, we are still facing a performance issue with the historical blocks produced during the "test period" (approximately between heights 8970620 and 9040425). These blocks remain large (~30MB), and the RPC response time for them is still unacceptably high (30+ seconds), even with the updated configs.

Are there any additional "under-the-hood" tweaks that could help serve these specific large historical blocks faster?

[rach-id]

Hey @elvensejo, could you please share the specs of your machine here, whether you're running other docker containers on the same machine, and if you can run this tool https://github.com/celestiaorg/celestia-app/tree/main/tools/cpu_requirements#docker using docker and share the result.