Skip to content

Latest commit

 

History

History
85 lines (75 loc) · 3.41 KB

README.md

File metadata and controls

85 lines (75 loc) · 3.41 KB
Raw

Monitoring Stack

The IUDX Monitoring stack uses Prometheus, Grafana, Micrometer, Node Exporter, Docker Daemon to capture API, System metrics. It integrates with telegram to publish alerts from Grafana based on predefined rules.

Monitoring-stack-Installation

Generate secret files

  1. Make a copy of sample secrets directory.
cp -r example-secrets/secrets .
  1. Generate required secrets using following script:
./create-secrets.sh
  1. Make sure /tmp/metrics-targets directory is created on the node where Prometheus is being deployed. It needs to be recreated with every node restart as it is a temporary directory.
mkdir /tmp/metrics-targets
  1. Config Telegrambot for grafana's alerts is detailed here. Then appropiately define the environment file secrets/grafana-env-secret. The template is defined as follow: Please do not include comments and substitute appropiate correct values in the placeholders <placholder>.
GF_SERVER_ROOT_URL=https://<grafana-domain-name>/
GF_SERVER_DOMAIN=<grafana-domain-name>
TELEGRAM_CHAT_ID=<telegram-chat-id>
TELEGRAM_BOT_TOKEN=<telegram-chat-token>
  1. Configure servers to be monitored for certificate expiry, server up status, as targets in blackbox-targets.yml. See below for an example.
- targets:
    - https://rs.io.test/apis
    - https://catalogue.io.test/
  1. secrets directory after generation of secrets
secrets/
├── configs
│   └── blackbox-targets.yaml
├── .grafana.env
└── passwords
    ├── grafana-super-admin-passwd
    └── grafana-super-admin-username

Assign node labels

docker node update --label-add monitoring_node=true <hostname/ID>

Define Appropriate values of resources

Define Appropriate values of resources -

  • CPU
  • RAM
  • PID limit in mon-stack.resources.yaml for grafana, prometheus, loki, promtail, blackbox as shown in sample resource-values file for here

Deploy

Deploy monitoring stack:

./install.sh

Description

  • Installs Vertx_sd, Prometheus, Loki, Grafana, blackbox swarm services with replicas as one at node with "node.labels.monitoring_node==true" .
  • Promtail service installed in global mode i.e. all nodes have one promtail task running.

Note

  1. Grafana creates super admin when it is run for the first time, and the password is saved to db (i.e. grafana-volume). Subsequent running/restarting the docker with new admin credentials doesn't overwrite the password stored in Grafana db.
  2. Pipeline stages might be different for each application , this can be done using match stage
  3. mon-stack.yaml contains additional service vertx_sd, which discover vertx instances from zookeeper for prometheus.
  4. The grafana is now secured through centralised nginx.
  5. If you need to expose/access grafana HTTP port or have custom stack configuration( see here for example configuration of 'mon-stack.custom.yaml' file). You can bring up as follows.
docker stack deploy -c mon-stack.yaml -c mon-stack.resources.yaml -c mon-stack.custom.yaml mon-stack

This is generally useful in local,dev/test environment.