README.md

monitor-promdiscovery

Overview

The monitor-promdiscovery tool generate file-based service discovery files for Prometheus, for the monitor-exporter and icinga2-exporter The tool is typically run from cron or equivalent tool to check OP5 Monitor or Icinga2 for hosts service that should be scraped for performance metrics by the exporters.
For a host to be detected, it must be a part of a hostgroup or hostgroups that is defined in the configuration file.

Flow

1. monitor-promdiscovery is executed at regular intervals, e.g. every minute, by cron.
2. monitor-promdiscovery performs a filter query against the configured Monitor instance for all hosts that are part of a specific hostgroup. All hosts in the hostgroup(s) will be scraped for all their services' performance data monitor-exporter or icinga2-exporter.
3. monitor-promdiscovery checks against the configured file-based discovery if any host from the above call is new or not existing anymore.
4. Only if there is not a match the file-based discovery file will be updated
5. If the file is updated, Prometheus will reload the configuration file.

Running

Only use configuration file

python -m monitor_promdiscovery -f ./config.yml

Override system property in the configuration file

python -m monitor_promdiscovery -f ./config.yml -s <system>

Where can be either icinga2 or monitor

Installing

1. Check out / clone the git repo.

2. Install dependencies

   pip install -r requirements.txt

3. Build a distribution

   python setup.py sdist

4. Install locally

   pip install dist/monitor-promdiscovery-X.Y.Z.tar.gz

Configuration

The monitor-promdiscovery configuration file should be rather self-explanatory.

NB! Only icinga2 or op5monitor has to be configured. If both are used, use the -s switch instead of configure system in the property file. Using the below config executing without specifiying -s , the icinga2 entry will be used.

# The system can be either op5monitor or icinga2 
system: icinga2

op5monitor:  
  url: https://monitor.local  
  user: monitor  
  passwd: monitor  
  # Connection timeout - default 5
  # timeout: 5
  # Verify ssl - default False
  # verify: False

  # The hostgroup used to select hosts as targets  
  hostgroup: prometheus    
  prometheus:  
    # path where to create the file-based discovery file - must be set  
    sd_file: <sd_directory>/monitor_sd.yml  
    # Additional labels to tag metrics with - optional
    labels:  
      source: monitor  
      env: prod

icinga2:
  url: https://127.0.0.1:5665
  user: root
  passwd: cf593406ffcfd2ef
  # Connection timeout - default 5
  # timeout: 5
  # Verify ssl - default False
  # verify: False

  # The hostgroup used to select hosts as targets
  hostgroup: 
    - prometheus
    - linux_servers
  prometheus:
    # path where to create the file-based discovery file - must be set
    sd_file: /home/andersh/programs/prometheus/sd/icinga2_sd.yml
    # Additional labels to tag metrics with - optional
    labels:
      source: icinga2
      env: prod

logger:
  # Path and name for the log file. If not set sent to stdout
  logfile: monitor-promdiscovery.log
  # Format if day will have current day as post fix
  # format: day
  # Log level
  #level: INFO

The hostgroup property can be both a single value or a list of different hostgroups as shown above.

Prometheus configuration

The Prometheus configuration file would typical have the following job configuration in the scrape_configs section for the monitor-exporter:

  - job_name: 'op5'
    scrape_interval: 2m
    metrics_path: /metrics
    file_sd_configs:
    - files:
      - 'sd/monitor_sd.yml'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        # The address for the monitor-exporter
        replacement: localhost:5000

And the format for icinga2 is almost the same:

  - job_name: 'icinga2'
    scrape_interval: 2m
    metrics_path: /metrics
    file_sd_configs:
    - files:
      - 'sd/icinga2_sd.yml'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        # The address for the monitor-exporter
        replacement: localhost:5000

The file_sd_configs must have the same filename as created by the tool, as defined in the configuration file.

System requirement

Python 3.6

For required packages, please review requirements.txt