Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provision a basic PSQL dashboard for grafana #290

Open
wants to merge 3 commits into
base: monitoring
Choose a base branch
from

Conversation

reosarevok
Copy link
Member

This is based on https://grafana.com/grafana/dashboards/14114-postgres-overview/ with an extra check for max query duration that seemed interesting, and is mostly intended as a proof of concept for provisioning dashboards. We can further improve the dashboard as needed.

This is based on https://grafana.com/grafana/dashboards/14114-postgres-overview/
with an extra check for max query duration that seemed interesting,
and is mostly intended as a proof of concept for provisioning dashboards.
We can further improve the dashboard as needed.
@mwiencek
Copy link
Member

This actually works quite nicely for me, though we could build on top of the metrics you added by creating a basic dashboard for them. That could also be used as an example for how to add additional dashboards to the repo. Here's one that includes a panel for the artist count:

{
  "__inputs": [
    {
      "name": "DS_PROMETHEUS",
      "label": "Prometheus",
      "description": "",
      "type": "datasource",
      "pluginId": "prometheus",
      "pluginName": "Prometheus"
    }
  ],
  "__elements": {},
  "__requires": [
    {
      "type": "grafana",
      "id": "grafana",
      "name": "Grafana",
      "version": "11.4.0"
    },
    {
      "type": "datasource",
      "id": "prometheus",
      "name": "Prometheus",
      "version": "1.0.0"
    },
    {
      "type": "panel",
      "id": "timeseries",
      "name": "Time series",
      "version": ""
    }
  ],
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": null,
  "links": [],
  "panels": [
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "description": "Number of rows in the artist table.",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "displayName": "rows",
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 0
      },
      "id": 1,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "pluginVersion": "11.4.0",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "disableTextWrap": false,
          "editorMode": "builder",
          "expr": "artist_count",
          "fullMetaSearch": false,
          "includeNullMetadata": true,
          "legendFormat": "__auto",
          "range": true,
          "refId": "A",
          "useBackend": false
        }
      ],
      "title": "Artist Count",
      "type": "timeseries"
    }
  ],
  "schemaVersion": 40,
  "tags": [],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "browser",
  "title": "Table Row Counts",
  "uid": "cea3e99siyr5sa",
  "version": 2,
  "weekStart": ""
}
  • For comparing with Solr metrics like number of documents per Solr core, we probably only need a metric for each entity table corresponding to a Solr core?
  • Currently these row count metrics are collected every 30s, which seems too frequent.

As a start, monitor the amount of rows on sir-indexed tables.

Includes a dashboard with gauges for every table; I don't see
a reason why it would be useful to have these be line charts since
there's no reason we should expect huge jumps, it's just good
to have a clear idea of which tables are bigger with the numbers.
@reosarevok reosarevok force-pushed the provision-psql-dashboard branch from c40a2c7 to cbb90c9 Compare January 16, 2025 13:23
There seems to be no good reason why we would keep hitting the DB
every 30 seconds to get the counts. 5 minutes seems more than enough.

My understanding is that if I set min_interval here to 300s (5m)
it will just keep the value for that long and keep responding with it,
however often prometheus asks.
@reosarevok
Copy link
Member Author

That seems like a good start. I expanded and provisioned the dashboard, adding all the counts that seem to be relevant as per https://github.com/metabrainz/sir/blob/e9e63641cd103c29a1aca456fb870d9f7d508774/sir/schema/__init__.py - and changed sql-exporter to only check stuff every 5 minutes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants