Revisiting log levels

[mratsim]

#1779 #1785

Description/Philosophy

• Fatal: ???
• Error: Bugs or critical unexpected behaviors, node cannot proceed with task, consistency compromised
• Warning: Expected behavior (file not found), node cannot proceed with task but can recover
• Notice: Start/quit, validator funds, own attestations, own blocks, finality checkpoints, (any) slashings
• Info: Important networking issues
• Debug: Common networking (new peers, kick peers, timeouts), consensus stages, block/attestation validation
• Trace: Keep-alive, routine schedule

Note: it is important not to fill disk with logs, this prevent analyzing them visually.
Logs done at high frequency should be summarized even at trace level to avoid drowning other subsystem
For example they can use an uint8 counter with

proc myHighFreqProc() =
  var counter {.threadvar.}: uint8
  if counter == 255:
    trace "Total of 255 myHighFreqProc call"
  counter += 1

For the keep alive / control message that were done by Gossipsub: https://github.com/status-im/nim-libp2p/pull/381/files

TODO

The current color of the notice level is confusing, it seems like an issue that needs to be addressed and seems to sit in-between warning and error.

[arnetheduck]

let's also add a doc/logging.txt outlining the general strategy - in the wiki, it gets lost (or if we need to migrate from github)