Improve Logging Logic

[0xNeshi]

Logging the processes in our service will drastically improve maintainability, for both us and package users, especially because the service is multi-threaded.

Investigate exact crate to rely on for logging, maybe first investigate tracing.

---

EDIT: We got some feedback on logs, and they generally currently seem to be annoying, as they just clutter the output.

Quote:

Based on my experience, I feel like lots of logs are redundant and i always set RUST_LOG=info,event_scanner=warn 😄 . Since I have my own logs each time there is a message / notification. And lots of logs from scanner which is INFO level is like should be DEBUG / TRACE level for me, e.g. the current block ranges to scan, the current raw logs scanned...

And i have app level logs for each message / notificaiton / error, so probly what i care is, if there is anything unexpected.

For current INFO level logs, i do feel it's redundant like the block ranges stuffs. And if there is a lot of raw logs for one shot, the log size will be huge that is annoying

---

NOTE: all crossed out items are marked as so because they actually apply to app code, not library code (event-scanner is a library)

Some general guidelines that we need to discuss and agree upon:

• the point of logs is:
  • notify the dev what the scanner is doing
  • notify the dev what data the scanner is processing
  • create history of events leading up to an error/bug
  • notify the dev when something that can affect them happens
• write meaningful log entries
• use appropriate log levels:
  • error - unexpected terminal failure or one that breaks invariants; needs to be debugged and fixed
    • Don't log returned errors - devs will handle them explicitly, logging is redundant
  • warning - abnormal behavior e.g. deep reorg detected (devs are notified about reorgs, but deeps ones are concerning), fallback failed (BUT do not log on retries, as this is internal expected behavior)
  • info - should probably not be used, as users generally set their own logs; maybe use it for major state changes, e.g. successful primary/fallback provider switch
  • debug/trace - I'm not sure what we could log that's useful to devs. But whatever it is, it should be sparse, maybe even opt-in (add a tracing feature flag)
• logs should indicate where they originated (block range scanner, event scanner, maybe even the exact function?)
• (optional) logs should contain some sort of trace ID, to enable tracking the chain of events and possibly indicating in which thread/task logs are being emitted