docker-metricbeat-example

Docker and Metricbeat configurations examples. Learn how to use Docker and Metricbeat to monitor you host system, all your nodes, your docker containers and all your (distributed) services.

Warning

This project is built for example and testing.
DO NOT use any part of it in production unless you know exactly what you are doing.

Software versions

Name	Version
Docker	17.10.0-ce (f4ffd25) (Tue Oct 17 19:05:05 2017)
Docker-compose	1.17.0, build ac53b73
Elasticsearch	6.0.0
Metricbeat	6.0.0
Kibana	6.0.0
Apache	2.4.29-alpine
MongoDB	3.5.13-jessie
MySQL	8.0
Nginx	1.13.7-alpine
RabbitMQ	3.6.14-management-alpine
Redis	3.2.11-alpine

Install

Clone the repository

git clone git@github.com:ypereirareis/docker-metricbeat-example.git && cd docker-metricbeat-example

Setup your host (sudo/root needed). You shoud have something like this.

$ make setup
=> ACLs on /var/run/docker.sock OK
vm.max_map_count = 262144
=> vm.max_map_count=262144 OK
2105e36d174f6ad3d0d23cac5769454d60a51fa8998fc0fbe946c4e713f1493c
Building metricbeat-host
Step 1/6 : FROM docker.elastic.co/beats/metricbeat:6.0.0
 ---> bc8ff400feac
Step 2/6 : ARG METRICBEAT_FILE=metricbeat.yml
 ---> Using cache
 ---> ebd68e798d93
Step 3/6 : COPY ${METRICBEAT_FILE} /usr/share/metricbeat/metricbeat.yml
 ---> Using cache
 ---> e24c78b4b7e8
Step 4/6 : USER root
 ---> Using cache
 ---> 84b29d7e9635
Step 5/6 : RUN mkdir /var/log/metricbeat     && chown metricbeat /usr/share/metricbeat/metricbeat.yml     && chmod go-w /usr/share/metricbeat/metricbeat.yml     && chown metricbeat /var/log/metricbeat
 ---> Using cache
 ---> 0d1485e2e4c1
Step 6/6 : USER metricbeat
 ---> Using cache
 ---> 7c5172b2912c
Successfully built 7c5172b2912c
Successfully tagged dockermetricbeatexample_metricbeat-host:latest
elasticsearch uses an image, skipping
Building metricbeat
Step 1/6 : FROM docker.elastic.co/beats/metricbeat:6.0.0
 ---> bc8ff400feac
Step 2/6 : ARG METRICBEAT_FILE=metricbeat.yml
 ---> Using cache
 ---> ebd68e798d93
Step 3/6 : COPY ${METRICBEAT_FILE} /usr/share/metricbeat/metricbeat.yml
 ---> Using cache
 ---> 19c48873ed8e
Step 4/6 : USER root
 ---> Using cache
 ---> fa6d8c6138d6
Step 5/6 : RUN mkdir /var/log/metricbeat     && chown metricbeat /usr/share/metricbeat/metricbeat.yml     && chmod go-w /usr/share/metricbeat/metricbeat.yml     && chown metricbeat /var/log/metricbeat
 ---> Using cache
 ---> b691cc7eb5bf
Step 6/6 : USER metricbeat
 ---> Using cache
 ---> 5e02571e6fb2
Successfully built 5e02571e6fb2
Successfully tagged dockermetricbeatexample_metricbeat:latest
kibana uses an image, skipping

Host Monitoring

We are assuming your docker0 interface IP is: inet 172.17.0.1 netmask 255.255.0.0 broadcast 0.0.0.0 If not the case, please adjust the configuration in docker-compose.yml file for service metricbeat-host.

metricbeat-host:
  ...
  environment:
    - HOST_ELASTICSEARCH=elasticsearch:9222
    - HOST_KIBANA=kibana:5666
  extra_hosts:
    - "elasticsearch:172.17.0.1" # The IP of docker0 interface to access host from container
    - "kibana:172.17.0.1" # The IP of docker0 interface to access host from container
  network_mode: host # Mandatory to monitor host filesystem, memory, processes,...

Start monitoring your host.

$ make start-monitoring-host 
Creating metricbeat-elasticsearch ... 
Creating metricbeat-elasticsearch ... done
metricbeat-elasticsearch is up-to-date
Creating metricbeat-kibana ... 
Creating metricbeat-kibana ... done
Waiting for elasticsearch...
Creating metricbeat-metricbeat-host ... 
Creating metricbeat-metricbeat-host ... done

You can check everything is OK, and you should have 3 containers running...
Be careful Elasticsearch and Kibana ports are exposed on 0.0.0.0 network (every IP address).
Default Metricbeat dashboard are automatically loaded into Kibana (setup.dashboards.enabled: true)

$ docker ps                 
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                              NAMES
e25a76b4f1e4        dockermetricbeatexample_metricbeat-host               "/usr/local/bin/do..."   2 minutes ago       Up 2 seconds                                           metricbeat-metricbeat-host
27668d971ddf        docker.elastic.co/kibana/kibana:6.0.0                 "/bin/bash /usr/lo..."   2 minutes ago       Up 2 minutes        0.0.0.0:5666->5601/tcp             metricbeat-kibana
af93d8214167        docker.elastic.co/elasticsearch/elasticsearch:6.0.0   "/usr/local/bin/do..."   2 minutes ago       Up 2 minutes        9300/tcp, 0.0.0.0:9222->9200/tcp   metricbeat-elasticsearch

If everything is fine, you should be able to access Kibana, and Monitoring dashboard:

Kibana => http://127.0.0.1:5666
Dashboard list => http://127.0.0.1:5666/app/kibana#/dashboards?_g=()
System Overview => http://127.0.0.1:5666/app/kibana#/dashboard/Metricbeat-system-overview?_g=()

Host Dashboard

Docker Dashboard

Services Monitoring

First of all you need to start the Metricbeat container to monitor services.

make start-monitoring

You should now have an extra Metricbeat container for services (called metricbeat-metricbeat-services).

$ docker ps
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                              NAMES
d934ee72db19        dockermetricbeatexample_metricbeat                    "/usr/local/bin/do..."   4 minutes ago       Up 4 minutes                                           metricbeat-metricbeat-services
...

Start/Stop all services

make start-all
make stop-all

Start/Stop services one by one

Apache

Compose file: ./docker-compose.apache.yml
Start: make start-apache
Stop: make stop-apache

4f4cb6e72a39    httpd:2.4   "httpd-foreground"  11 minutes ago  Up About a minute   80/tcp  metricbeat-apache

MongoDB

Compose file: ./docker-compose.mongodb.yml
Start: make start-mongodb
Stop: make stop-mongodb

a379163bc90d    mongo   "docker-entrypoint..."   11 minutes ago    Up About a minute   27017/tcp    metricbeat-mongodb

MySQL

Compose file: ./docker-compose.mysql.yml
Start: make start-mysql
Stop: make stop-mysql

3ae8e0c7c1e3    mysql:8.0   "docker-entrypoint..."   11 minutes ago   Up About a minute   3306/tcp metricbeat-mysql

Nginx

Compose file: ./docker-compose.nginx.yml
Start: make start-nginx
Stop: make stop-nginx

da1a01c36c4c    nginx:1.13.7-alpine "nginx -g 'daemon ..."   11 minutes ago   Up About a minute   80/tcp  metricbeat-nginx

RabbitMq

Compose file: ./docker-compose.rabbitmq.yml
Start: make start-rabbitmq
Stop: make stop-rabbitmq

6c072660a008    bijukunjummen/rabbitmq-server:3.6.5 "/bin/sh -c /opt/r..."   14 minutes ago      Up 2 seconds   4369/tcp, 5672/tcp, 9100-9105/tcp, 25672/tcp, 127.0.0.1:1234->15672/tcp   metricbeat-rabbitmq

Redis

Compose file: ./docker-compose.redis.yml
Start: make start-redis
Stop: make stop-redis

52be9d662f8f    redis:3.2.11-alpine "docker-entrypoint..."   11 minutes ago Up About a minute   6379/tcp    metricbeat-redis

Others

More to come

Clean everything

A simple command to remove all containers related to Metricbeat

$ make clean
d934ee72db19
3ae8e0c7c1e3
a379163bc90d
6c072660a008
52be9d662f8f
da1a01c36c4c
4f4cb6e72a39
a400ee0b914a
43d22a57fde4
c1ca7e0d7e6a
All METRICBEAT containers removed !

(Re)start everything

$ make install

LICENSE

MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

docker-metricbeat-example

Software versions

Install

Host Monitoring

Services Monitoring

Start/Stop all services

Start/Stop services one by one

Apache

MongoDB

MySQL

Nginx

RabbitMq

Redis

Others

Clean everything

(Re)start everything

LICENSE

About

Releases

Packages

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 5 Commits
config		config
docker/metricbeat		docker/metricbeat
img		img
scripts		scripts
.gitignore		.gitignore
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
docker-compose.apache.yml		docker-compose.apache.yml
docker-compose.mongodb.yml		docker-compose.mongodb.yml
docker-compose.mysql.yml		docker-compose.mysql.yml
docker-compose.nginx.yml		docker-compose.nginx.yml
docker-compose.rabbitmq.yml		docker-compose.rabbitmq.yml
docker-compose.redis.yml		docker-compose.redis.yml
docker-compose.yml		docker-compose.yml

License

ypereirareis/docker-metricbeat-example

Folders and files

Latest commit

History

Repository files navigation

docker-metricbeat-example

Software versions

Install

Host Monitoring

Services Monitoring

Start/Stop all services

Start/Stop services one by one

Apache

MongoDB

MySQL

Nginx

RabbitMq

Redis

Others

Clean everything

(Re)start everything

LICENSE

About

Topics

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages