Evaluate switch to coreos/prometheus-operator

[solsson]

Can it solve rules management?

[solsson]

The value is far from clear, and after the initial apply I need to manage not only config files but also CRDs. It makes me suspicious that the annotations that trigger scraping are not mentioned in docs. Did they do away with those? Well, this was just a 5 minute check, I might revisit later.

[solsson]

Learned from #3 that https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus is the place to start. Despite the path contrib it actually looks maintained.

[solsson]

To what degree does the Prometheus community agree with prometheus-operator/kube-prometheus#16 (comment)?

Edit: tried a mailing list thread - https://groups.google.com/forum/#!topic/prometheus-users/pRSQiTVxCMc

[solsson]

For now we could use a custom prometheus.yaml, and later transition to ServiceMonitor if/when it has merits:

• Prometheus Operator with custom prometheus.yml prometheus-operator/prometheus-operator#671 (comment)
• Documentation: Add docs on how to use a custom configuration prometheus-operator/prometheus-operator#678

[solsson]

Managing the secret manually works. Would be great if the operator supported ServiceMonitor and custom config blocks though. I don't see a reason why the two service discovery methods can't be combined, but I'll dig into that if/when we grow out of the annotations.

[solsson]

Would be great if ...

I take this, and b68f715, back. Instead I'll add a Prometheus named custom. It's actually quite convenient to have data separated into scoped instances anyway. Will probably use another Prometheus named kafka to isolate those metrics. Until you get a hang of all the variables it will really help to have them in separate instances.