[stable/concourse] Moves prometheus annotation to deployment

[cirocosta]

What this PR does / why we need it:

Previously, Prometheus annotations were added to the web service,
which is meant to represent the whole replica set of pods, leaving
no pod-specific labels to the metrics gathered, defeating the
purpose of some of them (that are meant to be container-specific).

By moving them to deployment, we're able to collect them per-pod,
instead of per-service.

Which issue this PR fixes

Special notes for your reviewer:

Although it was possible to still have the effects of this change working without making the annotation per-deployment (by not setting the concourse.web.prometheus.enabled flag and manually including the annotations + env variables), I'm confident that going this route makes more sense.

Checklist

[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

• DCO signed
• Chart Version bumped
• Variables are documented in the README.md

cc @frodenas / @william-tran

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[william-tran]

From https://groups.google.com/forum/#!topic/prometheus-users/jDEzVHUUFzw it looks like prometheus scrapes each pod behind the service, and from #6338 and #6422 it looks like it is desirable to have it on the service to support prometheus-operator. Not having used that myself, can you add anything to that @2color @richardalberto?

[william-tran]

But, @cirocosta if you've observed differences in pod scraping vs service scraping that obfuscate some metrics, please add that to the discussion

[paulczar]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cirocosta, paulczar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• stable/concourse/OWNERS [paulczar]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[william-tran]

@paulczar I do not appreciate you lgtm-ing while this PR is being discussed. Why did you do that?

[paulczar]

sorry about that, as you know there was a backlog of PRs for the concourse chart, I was trying to clear through a bunch that were approved but had version conflicts as i could fix the conflicts through the web ui. Clearly I either missed the ongoing discussion here, Will figure out on slack with you if we want to revert or leave as is.

[william-tran]

To continue this discussion; I think we need to keep the labels where they are; though prometheus-operator can use other mechanisms for discovering things to scrape, I assume there might be some configurations out there that use these service level annotations. This assumption should be true considering that was the original reason why they were added to the service, but again I'd like actual users @2color @richardalberto to chime in.