Log all significant events in a standard format

[joefutrelle]

When ERDDAP is running it doesn't appear to generate a logfile providing information about what operations it's doing and whether or not they're succeeding. I propose that it continuously generate logfiles that

1. Contain one line for each event, starting with the date and time
2. Are organized into log levels such as INFO, WARN, ERROR, DEBUG so that they can be configured at an appropriate level for the operational use case

This would enable integrating ERDDAP monitoring and metrics into off-the-shelf systems that perform log analysis, to facilitate real-time monitoring as well as other use cases such as debugging.

[BobSimons]

That's a good idea. It's a big job because there are lots of possible places to generate log messages. I may implement it in parallel with the current logging system because there are also advantages to an unstructured log (e.g., multiline messages).

If you have a suggested log format, please let me know.

[joefutrelle]

I was discussing this with a sysadmin here who pointed me to RFC 5424 which is used for syslogs.

https://docs.ruckuswireless.com/fastiron/08.0.60/fastiron-08060-monitoringguide/GUID-88F338BA-B7BF-485C-B1DE-7418710452A6.html

[kfiresmith]

Hello. I'm the sysadmin who mentioned Syslog enhanced log format (RFC 5424). Thinking about this more, I think that this is likely the job of something like Log4J or any other language-specific logging library. Ideally the logging library would be implemented and configured to create logs in a structure that conforms to the Linux FHS and other norms.

For example, you might create a path at /var/log/erddap, and populate it with something like erddap-master.log in 5424 format, and place other logs alongside it that better handle multi-line (which I believe that syslog can sorta handle so I'd read up on the spec as well...), such as foo-multi.json.

The goal with the main log is to get things into an industry standard format that can be easily forwarded to anyone's log concentration, storage, and analysis system (eg: Splunk, Graylog, ElasticStack, DataDog).

Having things be written in a standard format will allow an on-host forwarder such as rsyslog or syslog-ng to scoop up the lines and ship them off to central log collectors.

That's my quick take at least, as an ops guy.