Skip to content

Collecting and Visualizing Metrics

Jump to bottom
Rico edited this page Jan 10, 2023 · 10 revisions

If you ever ran a server or hosted some application yourself, you probably already understand the value of effective metrics and their visualization. When I created this project, I wanted to track how many certificates it processes and identify any errors missed logs. That's why I added support for exporting Prometheus-style metrics.

This document provides a comprehensive guide on setting up a monitoring stack from scratch and integrating certstream-server-go metrics into a Prometheus database for visualization with Grafana. For the purpose of simplicity, we'll utilize docker-compose on a linux server.

Prerequisites

This document assumes you know docker (and docker-compose) and the linux command line. It requires you to have docker and docker-compose (either v1 or v2) installed on your system. You should also have basic network/socket knowledge for setting up the listen addresses of the services.

Setting up the directories

Let's begin by creating a new directory that'll hold our configuration and application data. As an example we will use /opt/prometheus-grafana as the base directory. This directory will contain the docker-compose.yml file, as well as three config directories for application configuration files and two data directories used for persisting and backing up the data (e.g. prometheus db).

You can use these commands to create empty directories and config files:

mkdir prometheus prometheus_data grafana grafana_data certstream

touch prometheus/prometheus.yml grafana/config.monitoring certstream/config.yaml

The final directory layout should look something like this:

.
├── docker-compose.yml
├── certstream/
│   └── config.yaml
├── grafana/
│   └── config.monitoring
├── grafana_data/
├── prometheus/
│   └── prometheus.yml
└── prometheus_data/

Please ensure that the directories have matching read, write, and execute permissions for the users specified in the docker-compose.yml file. Alternatively, you may create individual users for each service and grant them specific permissions on the directories.

Creating config files

certstream-server-go

The certstream config is probably the easiest. The full documentation on the config parameters can be found here.

In the config.yml (certstream directory) you can enable the prometheus exporter in the config like so:

certstream/config.yaml:

webserver:
  listen_addr: "0.0.0.0"
  listen_port: 80
  full_url: "/full-stream"
  lite_url: "/"
  domains_only_url: "/domains-only"
  cert_path: ""
  cert_key_path: ""

prometheus:
  enabled: true
  listen_addr: "0.0.0.0"
  listen_port: 80
  metrics_url: "/metrics"

Enter the interface and port for the /metrics endpoint. When using docker, the interface and port are only used within the docker container. It is recommended to use 0.0.0.0 as the IP when running in a docker container. Don't forget to also configure the port in the docker-compose.yml file. If both prometheus and webserver are set to the same IP, a second port does not need to be configured. Once enabled, you should be able to access metrics via http from the /metrics endpoint.

Prometheus

In order to store our metrics, we need to configure Prometheus to scrape our /metrics endpoint. I suggest increasing the scrape_interval to 5s to have more detailed statistics.

prometheus/prometheus.yml:

global:
  scrape_interval:     5s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'Prometheus'
    static_configs:
      - targets: ['127.0.0.1:9090']
  - job_name: 'certstream'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'
    static_configs:
      # For usage in our docker stack, the target must be the name as the certstream service specified in the docker-compose.yml
      - targets: ['certstream']

Enter the IP/port combination or the hostname in the targets field. In case your prometheus/grafana stack runs on another machine and you need to connect to the /metrics endpoint via https, make sure to add the scheme: https to the prometheus config. You can find the full Prometheus documentation here

Grafana

Grafana barely needs any configuration out of the box. At least give it some information about the domain and the root url the site runs on. Since we're using the docker version, we're advised to configure Grafana using environment variables.

grafana/config.monitoring:

GF_SERVER_DOMAIN=grafana.example.com
GF_SERVER_ROOT_URL="https://grafana.example.com/"

You can extend the Grafana config by some mailing parameters, in case you want to send out mails from your Grafana instance.

GF_SMTP_ENABLED=true
GF_SMTP_HOST=mail.example.com
GF_SMTP_PASSWORD=examplePassword
GF_SMTP_USER=example
GF_SMTP_FROM_ADDRESS="grafana@example.com"
GF_SMTP_FROM_NAME="Grafana Example"

The whole documentation can be found here.

docker-compose

The final component is the docker-compose.yml file, which ties all of our services together into a single, manageable stack. You can find the docker-compose.yml in the docker directory of this GitHub repository.

wget -O docker-compose.yml https://raw.githubusercontent.com/d-Rickyy-b/certstream-server-go/master/docker/docker-compose.metrics.yml

Feel free to change the ports to your likings. A comprehensive guide on docker networking can be found in the official docker docs.

As mentioned above, make sure the directories have sufficient read, write, and execute permissions for the users running the services. You can specify the uid:gid in the docker-compose.yml if needed.

To ensure stability, the image versions are pinned. If you wish to update a service (e.g. Grafana), simply update the version number in the image tag (image: grafana/grafana:9.3.1) and run the following command: docker-compose pull && docker-compose stop && docker-compose up --force-recreate --build -d && docker image prune. This will pull the new version and recreate the stack. Alternatively, you can update individual containers by stopping and recreating them individually.

Starting our stack

In order to start the monitoring stack including the certstream service, run docker-compose up -d (or docker compose up -d on compose v2).

Setup Grafana

Upon the initial startup of Grafana, you can log in using the username and password admin. You will be prompted to change the password immediately.

To configure the Prometheus data source, navigate to the Data sources tab in the settings and click on "Add data source". In the URL field, enter http://prometheus:9090 (note that this must be the same name as specified as the service name in the docker-compose.yml file).

Finally, you can import the dashboard. Go to the dashboard page and click on New > Import. Enter the ID 17700. After the stack has been running for some time, you should be rewarded with a dashboard filled with sweet statistics.

Dashboard View

Clone this wiki locally