Log spam

[Jakob3xD]

Hello,
we updated our blackbox exporter and are now getting a lot of log messages due to the current implementation of slog introduced with #1311 as it removes the default none logging option.

We have tens of thousands of probes running each minute. Some of them are expected to fail but will cause the logger to print an error message. We are now producing over 14k of logs per second.

I can see two possible solutions for this where I prefer the first one.

1. Adjust the log level for some message

Currently, a probe failure is logged with level ERROR. IMO this should be level DEBUG (or WARN). My interpretation of level ERROR is that blackbox exporter itself is having an issue. Like failed to read config file, failed to unmarshal config file, failed binding to port. Things that are real issues. A failing probe can have different reasons, but does not affect the blackbox exporter.

2. Add back the none logging option

Add the none option by writing logs to io.Discard like in NewNopLogger().
Instead, it would also be possible to add a flag to define where to write the logs. For example stderr, stdout, /dev/null.