Monitoring for keep-test #3353
Merged
Monitoring for keep-test #3353
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- added deployment for prometheus - added deployment for grafana - added storageclass to insert persistent storage using cloud provider - prepared PersistentVolumeClaim both for prometheus and grafana - prepared basic service using NodePort to use with Ingress - preaparing README file with instructions
- reorganized conf for prometheus.yaml to have working discovery - added jobs for grana and prometheus because grafana has default dashes for grafana monitoring - modified data of configmap for grafana to add additional settins like dashboards conf and json files for creating those dashboards - added securityContext for grafana deployment - adjusted `volumeMounts` and `volumes` to have grafana config working with dashboards already created - used `items` to have one configmap and be able to choose wich file added dynamically should be put where - made some cosmetic name changes
- renamed configmap and splitted into two files for prom a& grafana - created kustomization.yaml file for further config templates - removed `NodePort` - renamed some objects according to last review - prepared basic README file describing whole process how set everythings up - adjusted client-dashboard.json file by adding local keep-test bootstrap services & external one fro azure - changed `job=clients` in search path for client-dashboard to `type=client`
This is the current configuration running in the cluster.
By default Prometheus stores data from the last 15 days. We need to increase that to 1 year. See https://prometheus.io/docs/prometheus/latest/storage/
Kustomization let's us define config files as standalone files and then combine them into a config map.
The configuration in this commit was copied from a tutorial to play with prometheus. We could use some parts of it but we don't need it right now. We should revisit the code removed in this commit when we will be working on supporting more kubernetes metrics monitoring.
pdyraga
approved these changes
Oct 10, 2022
| #### keep-discovered-nodes | ||
|
|
||
| The nodes to monitor are discovered with | ||
| link:https://github.com/keep-network/prometheus-sd[Prometheus Custom Service Discovery]. |
There was a problem hiding this comment.
Should we link to a specific branch? That repo is empty currently.
There was a problem hiding this comment.
It won't be empty once we merge keep-network/prometheus-sd#1
pdyraga
added a commit
that referenced
this pull request
Oct 25, 2022
In this PR we add a configuration of monitoring for the production environment. The configuration is based on a couple of PRs deployed and tested on the keep-test environment: #3353 #3357 #3373 The following resources are exposed: Public Dashboard: https://public.monitoring.threshold.network Grafana: https://monitoring.threshold.network/grafana Prometheus: https://monitoring.threshold.network/prometheus
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements a basic monitoring setup for
keep-testenvironment.We set up Prometheus and Grafana to monitor Keep Nodes running on test environment.
For details please see README.adoc.
Please note that we introduce the usage of kustomization tool. The tool is super helpful for config map generation and consistent resource labeling.
When exploring Prometheus configuration we touched Kubernetes cluster monitoring but decided to remove it from this PR, as the solution is far from the final. We should revisit the code removed in 2a78f10 in the future.
Refs #3012
Refs #3279
Refs #3149