[Bug] Revisit which (unique) consensus and peering logs should be which log levels

[vicsn]

🐛 Bug Report

In order to speed up debugging consensus issues, we should give logs more appropriate log levels.

E.g. this should be a WARN (and there should not be another log saying the same thing):

snarkos[18506]: 2025-07-08T17:59:11.185621Z DEBUG snarkos_node_router::heartbeat: No connected peers

With that, we could first search for ERROR logs, then WARN, the DEBUG, etc.

[ljedrz]

Happy to tackle this once #3713 is in.

[vicsn]

Another finding on this topic on recent release v4.2.0, a bunch of benign TRACE level logs are ERRORS. There is some overlap here with other existing issues, but I'll just list them out:

failed: Dropping connection request from '35.88.249.212:4130' (already connecting)│
failed: '35.88.249.212:4130' disconnected before sending "Message::ChallengeRespons│
failed: Skipping request to connect to self                                       │
Too many connections; refusing to connect to
failed: Connection reset by peer (os error 104)                                     │
tower_http::trace::on_failure: response failed classification=Status code: 500 Internal Server Error latency=0 ms      │
consensus: Unable to advance to the next block - Failed to speculate on transactions - Failed to post-rati│
bft::bft: BFT failed to advance the subdag for round 444 - Failed to speculate on transactions - Failed to│
ft::helpers::storage: Failed to insert certificate '7288515620661834..' from block 334 - Certificate for round 769 already exists in storage (gc = 666)       │
 snarkos_node_tcp::protocols::reading: can't read from 54.190.70.172:47311: invalid data                 │
 snarkos_node_tcp::tcp: Failed to connect with 34.216.155.81:60927: broken pipe                          │
55.81:51165' disconnected before sending "Event::ChallengeRequest"                                       │
this author for round 835 already exists in storage (gc = 732)                                           │
ft::helpers::storage: Failed to insert certificate '7337963959614824..' from block 361 - Certificate with this author for round 835 already exists in storage
 snarkos_node_tcp::tcp: Failed to connect with 204.76.203.28:54430: frame size too big                   │
 snarkos_node_tcp::protocols::handshake: handshake with 35.91.105.76:4130 timed out                      │
onsensus: Unable to advance to the next block - Block has invalid height. Expected 378, but got 377.     │
ft::bft: BFT failed to advance the subdag for round 876 - Block has invalid height. Expected 378, but got│
onsensus: Unable to advance to the next block - Batch(es) in the block point(s) to a certificate ... that is not associated with a previous block

At least the following errors are unclear and need better logging, perhaps some of the above as well:

ft::bft: BFT failed to receive the callback for round 876 - channel closed                               │
th 44.247.152.199:35977 failed: Dropped '44.247.152.199:35977' for reason: ProtocolViolation             │

[kaimast]

snarkos_node_bft::bft: BFT failed to receive the callback for round 876 - channel closed

#3858 should fix the majority of these. Right now, there seem to be cases where a task terminates without sending a result through the callback channel, for example due to a panic.