Nodies Monitoring is a customizable and extensible monitoring solution for monitoring host machine metrics, container metrics, node status and node logs.
Our project is split into a server (monitoring) stack, and a client (exporter) stack.
Please see architecture documentation for a deeper analysis of the server/client stack.
TBD instructions on how to contribute to this project.
NOTE: These dependencies are required on both the server and client stack.
Note: Tested and recommended installation on Ubuntu 22.04.1 LTS host OS
Python
Docker
Uninstall existing docker
sudo apt-get remove docker docker-engine docker.io containerd runcInstall required packages
sudo apt-get update
sudo apt-get install \
ca-certificates \
curl \
gnupg \
lsb-releaseAdd docker official GPG key
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpgSetup docker repo
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/nullUpdate apt package index
sudo apt-get updateInstall latest docker
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-pluginClone the repository
sudo git clone https://github.com/baaspoolsllc/nodies_monitoring.gitChange the current working directory to the cloned repository
cd nodies_monitoringInstall python3-pip
apt install python3-pipInstall requirements
pip3 install -r requirements.txtNOTE: This step is required on both the server and the client stack. When setting up this stack, you should follow the principles of least privilege when allowing users to access your exposed ports by setting up network or host based firewalls.
- Run
sudo python3 setup.pyto generate a templatesettings.yml.newfile. - Update
clients.promtail.loki_endpointwith the ip address of the host that will run the monitoring stack (loki, grafana, minio, prometheus, alertmanager) - Update
server.prometheus.exporter_endpointwith the endpoints any services of the exporter_stack (blockchain_exporter, cadvisor, node_exporter, promtail) - Update
server.alerts.contactpointswith the contact points of your alert recievers to send grafana-managed alerts to. - Update
clients.promtail.log_root_pathwith the root path of your nginx logs. - Rename
settings.yml.newtosettings.yml
On your monitoring server:
- Run setup.py again to save your configurations and set folder permissions.
sudo python3 setup.py- Edit the
settings.yml.newfile
sudo vi settings.yml.new- Rename the file
settings.yml.newtosettings.yml
sudo mv settings.yml.new settings.yml- Create the
chains.jsonfile - Mandatory
echo "[]" > clients/bcexporter/config/chains.json- Execute the setup.py once again
sudo python3 setup.py- Change directory into the server subfolder, and boot up all server services
cd server && docker compose up -d- Launch a browser and navigate to
http://<host>:<port>e.g http://192.168.1.2:3000
Default username and password for Grafana is admin:admin
NOTE: Sudo is required here to set some folder permissions
On your exporter server:
- Include your blockchain accessible endpoints inside a
chains.json(./templates/chains.json) file. An example file is provided for you inchains.example.json.
- Ensure that your access logs are formatted with the following format:
log_format json_combined escape=json
'{'
'"region":"us-east-1",'
'"time_local":"$time_local",'
'"request":"$scheme://$host$request_uri",'
'"method":"$request_method",'
'"protocol":"$server_protocol",'
'"status":"$status",'
'"request_body":"$request_body",'
'"body_bytes_sent":"$body_bytes_sent",'
'"http_referrer":"$http_referer",'
'"http_user_agent":"$http_user_agent",'
'"upstream_addr":"$upstream_addr",'
'"upstream_status":"$upstream_status",'
'"remote_addr":"$remote_addr",'
'"remote_user":"$remote_user",'
'"upstream_response_time":"$upstream_response_time",'
'"upstream_connect_time":"$upstream_connect_time",'
'"upstream_header_time":"$upstream_header_time",'
'"request_time":"$request_time"'
'}';
Note: Replace the region value with a unique region identifer, i.e us-east-1, eu-central-1, etc based off where your nginx server is located.
- Run setup.py again to save your configurations and set folder permissions.
sudo python3 setup.py- Edit the
settings.yml.newfile
sudo vi settings.yml.new- Rename the file
settings.yml.newtosettings.yml
sudo mv settings.yml.new settings.yml- Create the
chains.jsonfile - Mandatory
echo "[]" > clients/bcexporter/config/chains.json- Execute the setup.py once again
sudo python3 setup.py- Modify
clients.promtail.log_root_pathin thesettings.ymlfile to reflect your nginx log directory - Change directory into the clients subfolder, and boot up all client services
cd clients && docker compose up -dNOTE: Sudo is required here to set some folder permissions
setup.py has an optional CLI flag that allows control over which clients are ran on the exporter stack. For example, if you don't want to run log aggregation, then you can disable the log shipper promtail.
sudo python3 setup.py --clients blockchain_exporter cadvisor node_exporter
Default datasources have been provisioned in ./templates/datasources
To add additional datasources, please refer to grafana datasource documentation
Default dashboards have been provisioned in ./server/grafana/dashboards
To add additional dashboards, please refer to grafana dashboard documentation
Default alerting has been provisioned in ./server/grafana_provisioning/alerting
To add additional alerting, please refer to grafana alerting documentation
For any inquiries, please reach out to PoktBlade(PoktBlade#5970) or poktdachi(dachi#0005) on the pokt discord
