Support prometheus-enabled zookeeper mntr metrics

[hdhoang]

Hi, we run some bitnami-packaged ZK3.7. We enable prometheus support on it, and the mntr data format changes to prometheus/openmetric exposition format:

#  echo mntr | nc zookeeper.local 2181 | head
zk_version      3.7.0-e3704b390a6697bfdf4b0bef79e3da7a4f6bac4b, built on 2021-03-17 09:46 UTC
zk_server_state follower
zk_peer_state   following - broadcast
zk_read_commit_proc_issued{quantile="0.5"}      1.0
zk_read_commit_proc_issued_count        1.9583359E7
zk_read_commit_proc_issued_sum  1.3896375E7

In particular, recorded stats may now have scientific notation, instead of float-only.

zk_packets_received     1.6308337E7
zk_packets_sent 1.6853626E7

I have worked around this locally by augmenting #332 with a isScientific guard before quitting. However, the recorded state file fails to match regex at

Nagios-Plugins/check_zookeeper.pl

Lines 312 to 319 in 248dc2c

	if($last_line =~ /^(\d+(?:\.\d+)?)\s+
	(\d+)\s+
	(\d+)\s+
	(\d+)\s*$/x){
	$last_timestamp = $1;
	$last_stats{"zk_outstanding_requests"} = $2,
	$last_stats{"zk_packets_received"} = $3,
	$last_stats{"zk_packets_sent"} = $4,

Thus we always see

missing or incorrect state file stats, should have been reset now and available from next run

So far, I can't find any confirmation that enabling prometheus provider change mntr format at https://zookeeper.apache.org/doc/current/zookeeperMonitor.html#Prometheus

Please help to support this format, thanks.

[HariSekhon]

Can you please raise a Pull Request with what you have done with the isScientific guard, then I'll have a look and add to it to see if I can solve this.

It would be helpful if you could post the values.yaml you used for your bitnami-packaged ZooKeeper to reproduce this too.