Performance issues / New block age lag

[valamidev]

Describe the bug

With having the latest Sonic Node running since a while, I have to say compared to other EVM Nodes that it use more resource than any other chain does, even those which processing 10-20x more transactions.

Also Sonic node is rarely running with sync the chain head, it does usually run behind 4-60s behind.

[flolege]

Yes we can confirm this. We recently see more and more New block age lags.

Mostly our nodes run normally with block ages between 600 and 900ms, but occasionally they stall for some seconds, and we can see no New Block log entries in this period. Then the node starts catching up again, and suddenly processes new blocks rapidly until head. These periods can take longer - node catches up, produces some normal block ages, then falls behind again, etc.

We can see no obvious resource issues during this periods, RAM is 50%, CPU at 10-20%.

Sometimes this behavior happens already at node startup when the node is syncing. Everything runs smooth until block age around 15 seconds (away from head), then it stalls - and continues only seconds and often minutes later again. This goes a few times, until node catches up completely.

Also worth to mention is that this does not happen to all of our nodes at the same time. E.g. one is running normally, while the other has the issue at the same time. I tried to pprof and trace the hell out of it, but i can see no obvious issue at the moment.

EDIT: We happened to record the behavior, just look for yourself. CPU/RAM everything looks fine, no resource draining going on. We checked the nvme disk load via iostat, nothing unusual. The node recovers at the end of the clip.

Stall.mp4

[CRossel87a]

Does the team recommend any optimal locations to help reduce block age?

[flolege]

I think the stall is caused on the eth p2p level, by multiple p2p nodes connecting at the exact same time. It may cause backpressure on the gossip/ dagprocessor because all the peers share their event data at the same time, but I can not see the overload check triggering the suspend before stalls, so this is still weird.

Here are some screenshots supporting the theory, first the connected events in the trace view:

And here we see the p2p runPeer calls running at the same time.

I wonder how can that happen, why isn't that limited by the max peers setting - why can so many nodes connect at the same time?
And for the sonic team, can't this effect on the gossip system be mitigated somehow?

EDIT: I am reading the trace view wrong. It shows that the routines did something at the same time, but it may not be connecting. These peers were already connected I guess. Still, it would really be great to have an opinion on the issue from the sonic team.

[sonicseg]

We would like you to answer these followup questions:

Is the higher resources usage in terms of network, disk space, disk IO, or cpu usage? Or all of them?

Is the node running as a validator or as an archive?

If as an archive, is there a high RPC load on the node? If so, what RPC method mix is observed? Prometheus metrics collected from the node could help us understand details.

Can you provide a list of peers the node is connected to? Ideally, can the user provide the node ID (ENR address) itself, so that we can locate it in the network and construct a graph of its network neighborhood?

All the information you can collect and share will help us optimize the client further and improve not only the user experience and network speed, but also the footprint a node has on a host system.

[vanpt1114]

I'm also facing this issue. I am running an API node in the Singapore region.

[blockmandev]

The validator client is still using old LevelDB code from Fantom Opera instead of the new Carmen (SonicDB). I think someone forgot to push the new SonicDB into the validator client, or the validator docs need to be updated.

[LuisPH3]

The validator client is still using old LevelDB code from Fantom Opera instead of the new Carmen (SonicDB). I think someone forgot to push the new SonicDB into the validator client, or the validator docs need to be updated.

Carmen is the state db, not involved at this stage. The sonic client does use Carmen.