Structured logging

[silasdavis]

Many of the log statements have a semi-structured format with key=value substrings. It is useful operationally to have distinct key-values for downstream loggers like elasticsearch, logstash,etc.

I propose we swap out the Go std logger for an interface like:

https://github.com/go-kit/kit/blob/master/log/log.go#L11

or

https://github.com/inconshreveable/log15/blob/master/handler.go#L19

And use one of those libraries to handle the logging. Both work well. The former has simplest possible interface, the latter a little bit more structure, but both essentially transmit the same information.

I also personally much prefer to not share a global logging instance and pass it down the stack through options, but I appreciate it is a matter of taste. I think we could optionally set it on the factory and have it passed down from there.

Will submit PR if you agree.

[vmihailenco]

Let's try Logger from go-kit - it looks very simple. I guess we don't need to pull whole go-kit just for the interface and can copy paste only interface with the reference to go-kit somewhere.

[sruehl]

stumbled over that: https://github.com/go-logr/logr
might be the best option as it's pluggable to a lot of logging framework

[lzap]

Go now has the new slog experimental package: https://pkg.go.dev/golang.org/x/exp/slog