Prometheus Proxy

Prometheus is an excellent systems monitoring and alerting toolkit, which uses a pull model for collecting metrics. The pull model is problematic when a firewall separates a Prometheus server and its metrics endpoints. Prometheus Proxy enables Prometheus to reach metrics endpoints running behind a firewall and preserves the pull model.

The prometheus-proxy runtime comprises 2 services:

proxy: runs in the same network domain as Prometheus server (outside the firewall) and proxies calls from Prometheus to the agent behind the firewall.
agent: runs in the same network domain as all the monitored hosts/services/apps (inside the firewall). It maps the scraping queries coming from the proxy to the actual /metrics scraping endpoints of the hosts/services/apps.

Here's a simplified network diagram of how the deployed proxy and agent work:

Endpoints running behind a firewall require a prometheus-agent (the agent) to be run inside the firewall. An agent can run as a stand-alone server, embedded in another java server, or as a java agent. Agents connect to a prometheus-proxy (the proxy) and register the paths for which they will provide data. One proxy can work one or many agents.

Requirements

Requires Java 11 or newer.

CLI Usage

Download the proxy and agent uber-jars from here.

Start a proxy with:

java -jar prometheus-proxy.jar

Start an agent with:

java -jar prometheus-agent.jar -Dagent.proxy.hostname=mymachine.local --config https://raw.githubusercontent.com/pambrose/prometheus-proxy/master/examples/myapps.conf

If the prometheus-proxy were running on a machine named mymachine.local and the agent.pathConfigs value in the myapps.conf config file had the contents:

agent {
  pathConfigs: [
    {
      name: "App1 metrics"
      path: app1_metrics
      url: "http://app1.local:9100/metrics"
    },
    {
      name: "App2 metrics"
      path: app2_metrics
      url: "http://app2.local:9100/metrics"
    },
    {
      name: "App3 metrics"
      path: app3_metrics
      url: "http://app3.local:9100/metrics"
    }
  ]
}

then the prometheus.yml scrape_config would target the three apps with:

If the endpoints were restricted with basic auth/bearer authentication, you could either include the basic-auth credentials in the URL with: http://user:pass@hostname/metrics or they could be configured with basic_auth/bearer_token in the scrape-config.

The prometheus.yml file would include:

scrape_configs:
  - job_name: 'app1 metrics'
    metrics_path: '/app1_metrics'
    bearer_token: 'eyJ....hH9rloA'
    static_configs:
      - targets: [ 'mymachine.local:8080' ]
  - job_name: 'app2 metrics'
    metrics_path: '/app2_metrics'
    basic_auth:
        username: 'user'
        password: 's3cr3t'
    static_configs:
      - targets: [ 'mymachine.local:8080' ]
  - job_name: 'app3 metrics'
    metrics_path: '/app3_metrics'
    static_configs:
      - targets: [ 'mymachine.local:8080' ]

Docker Usage

The docker images are available via:

docker pull pambrose/prometheus-proxy:1.14.2
docker pull pambrose/prometheus-agent:1.14.2

Start a proxy container with:

docker run --rm -p 8082:8082 -p 8092:8092 -p 50051:50051 -p 8080:8080 \
        --env ADMIN_ENABLED=true \
        --env METRICS_ENABLED=true \
        pambrose/prometheus-proxy:1.14.2

Start an agent container with:

docker run --rm -p 8083:8083 -p 8093:8093 \
        --env AGENT_CONFIG='https://raw.githubusercontent.com/pambrose/prometheus-proxy/master/examples/simple.conf' \
        pambrose/prometheus-agent:1.14.2

Using the config file simple.conf, the proxy and the agent metrics would be available from the proxy on localhost at:

If you want to use a local config file with a docker container (instead of the above HTTP-served config file), use the docker mount option. Assuming the config file prom-agent.conf is in your current directory, run an agent container with:

docker run --rm -p 8083:8083 -p 8093:8093 \
    --mount type=bind,source="$(pwd)"/prom-agent.conf,target=/app/prom-agent.conf \
    --env AGENT_CONFIG=prom-agent.conf \
    pambrose/prometheus-agent:1.14.2

Note: The WORKDIR of the proxy and agent images is /app, so make sure to use /app as the base directory in the target for --mount options.

Embedded Agent Usage

If you are running a JVM-based program, you can run with the agent embedded directly in your app and not use an external agent:

EmbeddedAgentInfo agentInfo = startAsyncAgent("configFile.conf", true);

Configuration

The proxy and agent use the Typesafe Config library for configuration. Highlights include:

support for files in three formats: Java properties, JSON, and a human-friendly JSON superset (HOCON)
config files can be files or urls
config values can come from CLI options, environment vars, Java system properties, and/or config files.
config files can reference environment variables

All the proxy and agent properties are described here. The only required argument is an agent config value, which should have an agent.pathConfigs value.

Proxy CLI Options

Options	ENV VAR Property	Default	Description
--config, -c	PROXY_CONFIG		Agent config file or url
--port, -p	PROXY_PORT proxy.http.port	8080	Proxy listen port
--agent_port, -a	AGENT_PORT proxy.agent.port	50051	gRPC listen port for agents
--admin, -r	ADMIN_ENABLED proxy.admin.enabled	false	Enable admin servlets
--admin_port, -i	ADMIN_PORT proxy.admin.port	8092	Admin servlets port
--debug, -b	DEBUG_ENABLED proxy.admin.debugEnabled	false	Enable proxy debug servlet on admin port
--metrics, -e	METRICS_ENABLED proxy.metrics.enabled	false	Enable proxy metrics
--metrics_port, -m	METRICS_PORT proxy.metrics.port	8082	Proxy metrics listen port
--sd_enabled	SD_ENABLED proxy.service.discovery.enabled	false	Service discovery endpoint enabled
--sd_path	SD_PATH proxy.service.discovery.path	"discovery"	Service discovery endpoint path
--sd_target_prefix	SD_TARGET_PREFIX proxy.service.discovery.targetPrefix	"http://localhost:8080/"	Service discovery target prefix
--cert, -t	CERT_CHAIN_FILE_PATH proxy.tls.certChainFilePath		Certificate chain file path
--key, -k	PRIVATE_KEY_FILE_PATH proxy.tls.privateKeyFilePath		Private key file path
--trust, -s	TRUST_CERT_COLLECTION_FILE_PATH proxy.tls.trustCertCollectionFilePath		Trust certificate collection file path
--version, -v			Print version info and exit
--usage, -u			Print usage message and exit
-D			Dynamic property assignment

Agent CLI Options

Options	ENV VAR Property	Default	Description
--config, -c	AGENT_CONFIG		Agent config file or url (required)
--proxy, -p	PROXY_HOSTNAME agent.proxy.hostname		Proxy hostname (can include :port)
--name, -n	AGENT_NAME agent.name		Agent name
--admin, -r	ADMIN_ENABLED agent.admin.enabled	false	Enable admin servlets
--admin_port, -i	ADMIN_PORT agent.admin.port	8093	Admin servlets port
--debug, -b	DEBUG_ENABLED agent.admin.debugEnabled	false	Enable agent debug servlet on admin port
--metrics, -e	METRICS_ENABLED agent.metrics.enabled	false	Enable agent metrics
--metrics_port, -m	METRICS_PORT agent.metrics.port	8083	Agent metrics listen port
--consolidated, -o	CONSOLIDATED agent.consolidated	false	Enable multiple agents per registered path
--timeout	SCRAPE_TIMEOUT_SECS agent.scrapeTimeoutSecs	15	Scrape timeout time (seconds)
--max_retries	SCRAPE_MAX_RETRIES agent.scrapeMaxRetries	0	Scrape maximum retries (0 disables scrape retries)
--chunk	CHUNK_CONTENT_SIZE_KBS agent.chunkContentSizeKbs	32	Threshold for chunking data to Proxy and buffer size (KBs)
--gzip	MIN_GZIP_SIZE_BYTES agent.minGzipSizeBytes	1024	Minimum size for content to be gzipped (bytes)
--trust_all_x509	TRUST_ALL_X509_CERTIFICATES agent.http.enableTrustAllX509Certificates	false	Disable SSL verification for agent https endpoints
--cert, -t	CERT_CHAIN_FILE_PATH agent.tls.certChainFilePath		Certificate chain file path
--key, -k	PRIVATE_KEY_FILE_PATH agent.tls.privateKeyFilePath		Private key file path
--trust, -s	TRUST_CERT_COLLECTION_FILE_PATH agent.tls.trustCertCollectionFilePath		Trust certificate collection file path
--override	OVERRIDE_AUTHORITY agent.tls.overrideAuthority		Override authority (for testing)
--version, -v			Print version info and exit
--usage, -u			Print usage message and exit
-D			Dynamic property assignment

Misc notes:

If you want to customize the logging, include the java arg -Dlogback.configurationFile=/path/to/logback.xml
JSON config files must have a .json suffix
Java Properties config files must have a .properties or .prop suffix
HOCON config files must have a .conf suffix
Option values are evaluated in the order: CLI, environment vars, and finally config file vals
Property values can be set as a java -D arg to or as a proxy or agent jar -D arg
For more information about the proxy service discovery options, see the Prometheus documentation

Admin Servlets

These admin servlets are available when the admin servlet is enabled:

/ping
/threaddump
/healthcheck
/version

The admin servlets can be enabled with the ADMIN_ENABLED environment var, the --admin CLI option, or with the proxy.admin.enabled and agent.admin.enabled properties.

The debug servlet can be enabled with the DEBUG_ENABLED environment var, the --debug CLI option , or with the proxy.admin.debugEnabled and agent.admin.debugEnabled properties. The debug servlet requires that the admin servlets are enabled. The debug servlet is at: /debug on the admin port.

Descriptions of the servlets are here. The path names can be changed in the configuration file. To disable an admin servlet, assign its property path to "".

Adding TLS to Agent-Proxy Connections

Agents connect to a proxy using gRPC. gRPC supports TLS with or without mutual authentication. The necessary certificate and key file paths can be specified via CLI args, environment variables and configuration file settings.

The gRPC docs describe how to setup TLS. The repo includes the certificates and keys necessary to test TLS support.

Running TLS without mutual authentication requires these settingss:

certChainFilePath and privateKeyFilePath on the proxy
trustCertCollectionFilePath on the agent

Running TLS with mutual authentication requires these settingss:

certChainFilePath, privateKeyFilePath and trustCertCollectionFilePath on the proxy
certChainFilePath, privateKeyFilePath and trustCertCollectionFilePath on the agent

Running with TLS

Run a proxy and an agent with TLS (no mutual auth) using the included testing certs and keys with:

java -jar prometheus-proxy.jar --config examples/tls-no-mutual-auth.conf
java -jar prometheus-agent.jar --config examples/tls-no-mutual-auth.conf

Run a proxy and an agent docker container with TLS (no mutual auth) using the included testing certs and keys with:

docker run --rm -p 8082:8082 -p 8092:8092 -p 50440:50440 -p 8080:8080 \
    --mount type=bind,source="$(pwd)"/testing/certs,target=/app/testing/certs \
    --mount type=bind,source="$(pwd)"/examples/tls-no-mutual-auth.conf,target=/app/tls-no-mutual-auth.conf \
    --env PROXY_CONFIG=tls-no-mutual-auth.conf \
    --env ADMIN_ENABLED=true \
    --env METRICS_ENABLED=true \
    pambrose/prometheus-proxy:1.14.2

docker run --rm -p 8083:8083 -p 8093:8093 \
    --mount type=bind,source="$(pwd)"/testing/certs,target=/app/testing/certs \
    --mount type=bind,source="$(pwd)"/examples/tls-no-mutual-auth.conf,target=/app/tls-no-mutual-auth.conf \
    --env AGENT_CONFIG=tls-no-mutual-auth.conf \
    --env PROXY_HOSTNAME=mymachine.lan:50440 \
    --name docker-agent \
    pambrose/prometheus-agent:1.14.2

Note: The WORKDIR of the proxy and agent images is /app, so make sure to use /app as the base directory in the target for --mount options.

Scraping HTTPS Endpoints

Disable SSL verification for agent https endpoints with the TRUST_ALL_X509_CERTIFICATES environment var, the --trust_all_x509 CLI option, or the agent.http.enableTrustAllX509Certificates property.

Scraping a Prometheus Instance

It's possible to scrape an existing prometheus server using the /federate endpoint. This enables using the existing service discovery features already built into Prometheus.

An example config can be found in federate.conf.

Grafana

Grafana dashboards for the proxy and agent are here.

Nginx Support

Nginx does not work as a reverse proxy for prometheus-proxy.

Name		Name	Last commit message	Last commit date
Latest commit History 218 Commits
bin		bin
docs		docs
etc		etc
examples		examples
gradle/wrapper		gradle/wrapper
grafana		grafana
logback		logback
src		src
testing/certs		testing/certs
.codebeatignore		.codebeatignore
.codeclimate.yml		.codeclimate.yml
.dockerignore		.dockerignore
.editorconfig		.editorconfig
.gitignore		.gitignore
.travis.yml		.travis.yml
License.txt		License.txt
Makefile		Makefile
README.md		README.md
build.gradle		build.gradle
gradle.properties		gradle.properties
gradlew		gradlew
gradlew.bat		gradlew.bat
jitpack.yml		jitpack.yml
settings.gradle		settings.gradle

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Prometheus Proxy

Requirements

CLI Usage

Docker Usage

Embedded Agent Usage

Configuration

Proxy CLI Options

Agent CLI Options

Admin Servlets

Adding TLS to Agent-Proxy Connections

Running with TLS

Scraping HTTPS Endpoints

Scraping a Prometheus Instance

Grafana

Nginx Support

Related Links

About

Releases

Packages

Languages

License

rakhbari/prometheus-proxy

Folders and files

Latest commit

History

Repository files navigation

Prometheus Proxy

Requirements

CLI Usage

Docker Usage

Embedded Agent Usage

Configuration

Proxy CLI Options

Agent CLI Options

Admin Servlets

Adding TLS to Agent-Proxy Connections

Running with TLS

Scraping HTTPS Endpoints

Scraping a Prometheus Instance

Grafana

Nginx Support

Related Links

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages