feat(logs): add alloy + loki (#451)

* feat(logs): add alloy + loki * fix(metrics) misc metrics fixes & cleanup (#479) * fix and clean up metrics * allow prom to be accessed by external services --------- Co-authored-by: x19 <[email protected]> * add prom version --------- Co-authored-by: Sebastian <[email protected]>
Syndica · Jan 6, 2025 · 74d3c43 · 74d3c43
1 parent 47a84ab
commit 74d3c43
Show file tree

Hide file tree

Showing 11 changed files with 565 additions and 740 deletions.
diff --git a/.gitignore b/.gitignore
@@ -10,6 +10,7 @@
 /kcov-output
 /results
 /validator
+/logs
 
 # desktop app config
 .DS_Store

diff --git a/metrics/README.md b/metrics/README.md
@@ -1,98 +1,56 @@
 original src: https://github.com/docker/awesome-compose/tree/master/prometheus-grafana
 
-## Setup 
+## Setup
 
-requirements: 
-- `docker compose` 
-  - [https://docs.docker.com/engine/install/ubuntu/](https://docs.docker.com/engine/install/ubuntu/)
-- either mac or linux supported 
-
-modify `/etc/hosts` to include the following line: 
-```
-127.0.0.1 prometheus
-```
+requirements:
 
+- `docker compose`
+  - [https://docs.docker.com/engine/install/ubuntu/](https://docs.docker.com/engine/install/ubuntu/)
+- macos or linux (windows untested/unsupported)
 
-## Running 
+## Running
 
-mac: `docker compose -f compose_mac.yaml up -d`
-linux: `docker compose -f compose_linux.yaml up -d`
+mac: `docker compose up -d`
+linux: `docker compose up -d`
 
 - grafana will be accessable on `localhost:3000`
   - note: `username: admin password: grafana`
 - prometheus will be accessable on `localhost:9090`
-- sig metrics will be published to localhost:12345 (if you change this on the sig cli, you will 
-need to also modify the prometheus `target` to point to the different port).
+- sig metrics will be published to localhost:12345 (if you change this on the sig cli, you will
+  need to also modify the prometheus `target` to point to the different port).
 
 ## Shutting down
 
-mac: `docker compose -f compose_mac.yaml down`
-linux: `docker compose -f compose_linux.yaml down`
+mac: `docker compose down`
+linux: `docker compose down`
 
-## Compose sample
-### Prometheus & Grafana
+## Project Structure
 
-Project structure:
 ```
 .
-├── compose_linux.yaml
-├── compose_mac.yaml
+├── docker-compose.yml
+├── alloy -- this scrapes logs/ and pushes to loki
+├── loki -- log database
 ├── grafana
 │   └── dashboards/ -- this is where the sig dashboard lives (will need to copy .json export of dashboard from running container and push through git for any dashboard changes)
 │   └── datasources/ -- this points to prometheus docker
 ├── prometheus
-│   └── prometheus.yml 
+│   └── prometheus.yml
 └── README.md
 ```
 
-[_compose.yaml_](compose.yaml)
-```
-services:
-  prometheus:
-    image: prom/prometheus
-    ...
-    ports:
-      - 9090:9090
-  grafana:
-    image: grafana/grafana
-    ...
-    ports:
-      - 3000:3000
-```
-The compose file defines a stack with two services `prometheus` and `grafana`.
-When deploying the stack, docker compose maps port the default ports for each service to the equivalent ports on the host in order to inspect easier the web interface of each service.
-Make sure the ports 9090 and 3000 on the host are not already in use.
+## Run with logs
 
-## Deploy with docker compose
-
-```
-$ docker compose up -d
-Creating network "prometheus-grafana_default" with the default driver
-Creating volume "prometheus-grafana_prom_data" with default driver
-...
-Creating grafana    ... done
-Creating prometheus ... done
-Attaching to prometheus, grafana
-
-```
+`./zig-out/bin/sig gossip -n testnet 2>&1 | tee -a logs/sig.log`
 
 ## Expected result
 
-Listing containers must show two containers running and the port mapping as below:
 ```
 $ docker ps
-CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
-dbdec637814f        prom/prometheus     "/bin/prometheus --c…"   8 minutes ago       Up 8 minutes        0.0.0.0:9090->9090/tcp   prometheus
-79f667cb7dc2        grafana/grafana     "/run.sh"                8 minutes ago       Up 8 minutes        0.0.0.0:3000->3000/tcp   grafana
-```
-
-Navigate to `http://localhost:3000` in your web browser and use the login credentials specified in the compose file to access Grafana. It is already configured with prometheus as the default datasource.
-
-![page](output.jpg)
-
-Navigate to `http://localhost:9090` in your web browser to access directly the web interface of prometheus.
-
-Stop and remove the containers. Use `-v` to remove the volumes if looking to erase all data.
-```
-$ docker compose down -v
+CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS         PORTS                                       NAMES
+948f31ee975a   prom/prometheus        "/bin/prometheus --c…"   32 seconds ago   Up 3 seconds   0.0.0.0:9090->9090/tcp, :::9090->9090/tcp   prometheus
+adc6fb731842   grafana/grafana        "/run.sh"                32 seconds ago   Up 3 seconds   0.0.0.0:3000->3000/tcp, :::3000->3000/tcp   grafana
+e698bc98a061   grafana/alloy:v1.3.1   "/bin/alloy run --se…"   32 seconds ago   Up 3 seconds   0.0.0.0:3200->3200/tcp, :::3200->3200/tcp   alloy
+f774e615e5b1   grafana/loki:3.0.0     "/usr/bin/loki --con…"   32 seconds ago   Up 3 seconds   0.0.0.0:3100->3100/tcp, :::3100->3100/tcp   loki
+27ee173b0491   prom/node-exporter     "/bin/node_exporter"     32 seconds ago   Up 3 seconds   0.0.0.0:9100->9100/tcp, :::9100->9100/tcp   node-exporter
 ```
diff --git a/metrics/alloy/config.alloy b/metrics/alloy/config.alloy
@@ -0,0 +1,16 @@
+local.file_match "local_files" {
+    path_targets = [{"__path__" = "/var/log/alloy/*.log"}]
+    sync_period = "1s"
+}
+
+loki.source.file "log_scrape" {
+    targets    = local.file_match.local_files.targets
+    forward_to = [loki.write.grafana_loki.receiver]
+    tail_from_end = true
+}
+
+loki.write "grafana_loki" {
+    endpoint {
+        url = "http://loki:3100/loki/api/v1/push"
+    }
+}
diff --git a/metrics/compose_mac.yaml b/metrics/compose_mac.yaml
diff --git a/metrics/compose_linux.yaml → metrics/docker-compose.yml b/metrics/compose_linux.yaml → metrics/docker-compose.yml
@@ -1,18 +1,48 @@
 services:
   prometheus:
-    network_mode: "host" # -- only linux
-    image: prom/prometheus
+    image: prom/prometheus:v3.1.0
     container_name: prometheus
-    command:
-      - "--config.file=/etc/prometheus/prometheus.yml"
+    restart: unless-stopped
     ports:
       - 9090:9090
-    restart: unless-stopped
     volumes:
       - ./prometheus:/etc/prometheus
       - prom_data:/prometheus
+    command:
+      - "--config.file=/etc/prometheus/prometheus.yml"
+      # "--log.level=debug" # Not receiving metrics? Consider uncommenting
+
+    # Prometheus needs to be able to connect to the host network as "sig" isn't
+    # containerised. No other service should talk to the host.
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
+  loki:
+    image: grafana/loki:3.0.0
+    container_name: loki
+    ports:
+      - "3100:3100"
+    volumes:
+      - ./loki:/etc/loki:ro
+    command:
+      - "--config.file=/etc/loki/loki.yml"
+
+  alloy:
+    image: grafana/alloy:v1.3.1
+    container_name: alloy
+    ports:
+      - "3200:3200"
+    volumes:
+      - "./alloy/config.alloy:/etc/alloy/config.alloy:ro"
+      - "../logs:/var/log/alloy:ro"
+    command:
+      [
+        "run",
+        "--server.http.listen-addr=0.0.0.0:3200",
+        "/etc/alloy/config.alloy",
+      ]
+
   grafana:
-    network_mode: "host" # -- only linux
     image: grafana/grafana
     container_name: grafana
     ports:
@@ -25,11 +55,13 @@ services:
       - ./grafana/datasources:/etc/grafana/provisioning/datasources
       - ./grafana/dashboards:/etc/grafana/provisioning/dashboards
       - ./grafana/dashboards:/var/lib/grafana/dashboards
+
   node-exporter:
     image: prom/node-exporter
     container_name: node-exporter
     ports:
       - 9100:9100
     restart: unless-stopped
+
 volumes:
   prom_data: