Optional observation tag for labels is incompatible with Prometheus registry

[hpoettker]

Bug description

The ObservationEnvironmentRepositoryObservationConvention sets the tags for environment class, profile, application, and label on the observations only when the respective values are non-blank.

This is a problem for the Prometheus registry as Prometheus expects the same set of tags to be set with every observation.
When an observation with a different set of tags compared to previous observations is registered, the new observation will be dropped.

An example would be a client request that asks for specific label after previously only requests without a specific label have been requested.
When the Prometheus registry is used, the requests for a specific label will be dropped in this scenario.

Steps to reproduce

A full minimal complete verifiable example can be found here: https://github.com/hpoettker/config-server-reproducer

In short, a reproducing app can be created as follows:

• create a new project with the following dependencies:
  • Spring Cloud Config Server
  • Spring Boot Starter Actuator
  • Prometheus Registry
• add @EnableConfigServer
• expose the actuator's Prometheus end-point in the application.yml

Then, start up the application in the profile native, and execute the following requests:

curl http://localhost:8080/application/default

followed by

curl http://localhost:8080/application/default/label

The client will receive the expected responses but the the following warning will be logged:

The meter (MeterId{name='spring.cloud.config.environment.find.active', tags=[tag(spring.cloud.config.environment.application=application),tag(spring.cloud.config.environment.class=org.springframework.cloud.config.server.environment.SearchPathCompositeEnvironmentRepository),tag(spring.cloud.config.environment.label=label),tag(spring.cloud.config.environment.profile=default)]}) registration has failed:
Prometheus requires that all meters with the same name have the same set of tag keys.
There is already an existing meter named 'spring.cloud.config.environment.find.active' containing tag keys [spring.cloud.config.environment.application, spring.cloud.config.environment.class, spring.cloud.config.environment.profile].
The meter you are attempting to register has keys [spring.cloud.config.environment.application, spring.cloud.config.environment.class, spring.cloud.config.environment.label, spring.cloud.config.environment.profile].
Note that subsequent logs will be logged at debug level.

When checking the response of the end-point for the meter spring_cloud_config_environment_find_active, the second request with the label is missing from the statistics.