From 2fff3c4cce11b753f76c0d1b5c7aa35cf586ec10 Mon Sep 17 00:00:00 2001 From: Mark Nash Date: Mon, 21 Oct 2024 17:57:26 -0700 Subject: [PATCH 01/11] Update join.md to remove .html on the teams site --- docs/community/join.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/community/join.md b/docs/community/join.md index c604df1..8903e2b 100644 --- a/docs/community/join.md +++ b/docs/community/join.md @@ -22,7 +22,7 @@ This is a MUST do if you don't want to get left out! 3. Install Discord on your [computer](https://discord.com/download), [Android](https://play.google.com/store/apps/details?id=com.discord&hl=en_US&gl=US), or [iOS](https://apps.apple.com/us/app/discord-chat-talk-hangout/id985746746) device to always stay up-to-date. ## Chat with someone! -The best way to get involved with the network is to find someone that can direct you! Most of our work happens in teams. Here's how to [join a team](https://docs.seattlecommunitynetwork.org/community/teams.html). +The best way to get involved with the network is to find someone that can direct you! Most of our work happens in teams. Here's how to [join a team](https://docs.seattlecommunitynetwork.org/community/teams). ## Subscribe to our Google calendar! On our Google calendar we post regular occurring meetings and any impromptu events like social events, emergency repairs, site visits, etc. From ae23a0032cb09ef379b7a8ed8b11952a5224ba70 Mon Sep 17 00:00:00 2001 From: Mark Nash Date: Mon, 21 Oct 2024 18:00:09 -0700 Subject: [PATCH 02/11] Update community-networking-toolkit.md to remove a .html on a link --- docs/community/community-networking-toolkit.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/community/community-networking-toolkit.md b/docs/community/community-networking-toolkit.md index 4b3b96f..026462b 100644 --- a/docs/community/community-networking-toolkit.md +++ b/docs/community/community-networking-toolkit.md @@ -12,7 +12,7 @@ Here is a [spreadsheet](https://docs.google.com/spreadsheets/d/1-f0neQ0XBOndDnIj ## Ethernet Crimping Ethernet crimping is an important skill for network site installations. The length of an ethernet cable connection is only truly known once you're on site so it is useful to be able to quickly cut ethernet cable to the desired length and crimp it. -* [Our Guide to Crimping](https://docs.seattlecommunitynetwork.org/learn/cable-crimping.html) +* [Our Guide to Crimping](https://docs.seattlecommunitynetwork.org/learn/cable-crimping) * [Workshop Slides](https://docs.google.com/presentation/d/1HG5OcJysTicr_JHOlsKTB2ewWBVexrQDYv0Xn3_1hYA/edit?usp=sharing) * [Crimping Video](https://www.youtube.com/watch?v=WvP0D0jiyLg) * [Cable Testing Video (first 7 min only for basics)](https://www.youtube.com/watch?v=3tHvOLBp2zM) From 2b37b85723aab81d40d9c4f3d068b3a4e28e7bd9 Mon Sep 17 00:00:00 2001 From: Rudra Singh <136931703+rudra-singh1@users.noreply.github.com> Date: Fri, 15 Nov 2024 22:33:12 -0500 Subject: [PATCH 03/11] Create grafana_log_dashboard.md --- docs/infrastructure/grafana_log_dashboard.md | 287 +++++++++++++++++++ 1 file changed, 287 insertions(+) create mode 100644 docs/infrastructure/grafana_log_dashboard.md diff --git a/docs/infrastructure/grafana_log_dashboard.md b/docs/infrastructure/grafana_log_dashboard.md new file mode 100644 index 0000000..059c2a6 --- /dev/null +++ b/docs/infrastructure/grafana_log_dashboard.md @@ -0,0 +1,287 @@ +# Grafana Server Log Monitoring Dashboard Setup + +## Developed By: Rudra Prakash Singh, Esther Jang + +This guide walks you through the process of setting up a Grafana Server Log Monitoring Dashboard, including the installation and configuration of Grafana, Loki, and Promtail for log aggregation and visualization. It also covers setting up scripts for automating log retrieval and dashboard updates. + +--- + +## Step 1: Install Grafana via CLI + +1. **Install Grafana** on your server using the following instructions for Debian-based systems: + - Follow the installation guide here: [Grafana Installation on Debian](https://grafana.com/docs/grafana/latest/setup-grafana/installation/debian/) + +2. Once installed, start Grafana and make sure it's accessible via a port (default: `3000`). + +3. After Grafana is running, create an admin account. Once logged in, you'll land on the Grafana homepage. + +--- + +## Step 2: Install Loki + +1. **Install Loki** by following the official instructions for Docker: + - [Loki Installation via Docker](https://grafana.com/docs/loki/latest/setup/install/docker/) + +--- + +## Step 3: Create a Docker Compose File for Loki + +To set up Loki with Grafana, create a `docker-compose.yml` file: + +```yaml +version: "3" # The version can be removed if you're using a recent Docker Compose version. + +services: + grafana: + image: grafana/grafana:latest + ports: + - "3000:3000" + networks: + - loki + volumes: + - grafana-data:/path # Adding persistence for Grafana data + + loki: + image: grafana/loki:latest + ports: + - "3100:3100" + networks: + - loki + volumes: + - ./path # Local directory for Loki data + user: "10001" # Run Loki as the appropriate user to avoid permission issues + + promtail: + image: grafana/promtail:latest + command: "-config.file=/etc/promtail/config.yaml" + networks: + - loki + volumes: + - "/var/log:/var/log" # Mounting host logs + +networks: + loki: + +volumes: + grafana-data: # Declaring the volume for Grafana +``` + +--- + +## Step 4: Install Promtail + +1. **Install Promtail** by following the official instructions: + - [Promtail Installation Guide](https://grafana.com/docs/loki/latest/send-data/promtail/installation/) + +--- + +## Step 5: Configure Data Sources (Logs) + +To aggregate logs from other servers, we set up a method of transferring logs to the Grafana server. The logs are pulled into the Grafana server, where they are processed and rendered on a Grafana dashboard. + +### 5.1: Write a Bash Script to Pull Logs + +Here’s an example of a script (`transfer_logs.sh`) to pull logs from another server: + +```bash +#!/bin/bash + +# Define variables +REMOTE_SERVER="root@server_ip" # Remote server IP +SSH_KEY_PATH="path" # Path to SSH key +DEST_DIR="path" # Destination directory on the Grafana server + +# Define paths for logs +DOCKER_LOG="path/to/docker/log" +NGINX_LOG="path/to/nginx/log" +SYSLOG="path/to/syslog" + +# Create directories if they don't exist +mkdir -p "$DEST_DIR/docker" "$DEST_DIR/nginx" "$DEST_DIR/syslog" + +# Transfer logs +echo "Transferring Docker log..." +ssh -i $SSH_KEY_PATH $REMOTE_SERVER "cat $DOCKER_LOG" > "$DEST_DIR/docker/docker.log" + +echo "Transferring NGINX log..." +ssh -i $SSH_KEY_PATH $REMOTE_SERVER "cat $NGINX_LOG" > "$DEST_DIR/nginx/error.log" + +echo "Transferring syslog..." +ssh -i $SSH_KEY_PATH $REMOTE_SERVER "cat $SYSLOG" > "$DEST_DIR/syslog/syslog.log" + +echo "All logs have been successfully transferred." +``` + +--- + +### 5.2: Configure Promtail to Scrape Logs + +Next, create a `promtail-config.yaml` file to let Promtail know where the logs are located: + +```yaml +server: + http_listen_port: 9080 + grpc_listen_port: 0 + +positions: + filename: /tmp/positions.yaml + +clients: + - url: http://server_ip:3100/loki/api/v1/push # Loki's URL + +scrape_configs: + - job_name: docker_logs + static_configs: + labels: + job: docker + __path__: /path/to/logs/on/grafana/server/docker/*.log + - job_name: nginx_logs + static_configs: + labels: + job: nginx + __path__: /path/to/logs/on/grafana/server/nginx/*.log + - job_name: syslog_logs + static_configs: + labels: + job: syslog + __path__: /path/to/logs/on/grafana/server/syslog/*.log +``` + +Run Promtail with this configuration: + +```bash +sudo ./promtail-linux-amd64 -config.file=promtail-config.yaml +``` + +--- + +## Step 6: Configure Grafana to Use Loki + +1. Open Grafana’s homepage and click the **three vertical lines** in the top-left corner. + +2. Navigate to **Data Sources**, then click **Add Data Source**. + +3. Select **Loki** as the data source type. + +4. Define the **URL** of your Loki instance, which is typically `http://localhost:3100`. + +5. Click **Save & Test**. You should see a success message confirming the connection. + +--- + +## Step 7: Explore Data in Grafana + +1. Go to the **Explore** section in Grafana. + +2. Use label filters and run queries to see the logs coming from Promtail. The labels in the filters correspond to the `labels` defined in the `promtail-config.yaml`. + +--- + +## Step 8: Create a Dashboard + +1. On the left sidebar in Grafana, click **Dashboards** > **New Dashboard**. + +2. Click **Add Panel** to create a new panel. + +3. Under **Data Source**, select **Loki**. + +4. Write queries to retrieve logs for display. Use different labels to filter log data as needed. + +5. Choose a visualization type (e.g., **Logs**). + +6. Repeat the process to add more visualizations, then adjust the layout and size to suit your needs. + +7. Save the dashboard once you are satisfied with the layout. + +--- + +## Step 9: Automate Log Retrieval and Update Dashboard + +To make the process of pulling logs and updating the Grafana dashboard more automated, create two Python scripts: + +### 9.1: `transfer_server.py` + +This script triggers the Promtail process to start log scraping when accessed via an HTTP request: + +```python +from http.server import BaseHTTPRequestHandler, HTTPServer +import subprocess + +COMMAND = "sudo ./promtail-linux-amd64 -config.file=promtail-config.yaml" + +class RequestHandler(BaseHTTPRequestHandler): + def do_GET(self): + if self.path == "/run": + try: + subprocess.Popen(COMMAND, shell=True) + self.send_response(200) + self.end_headers() + self.wfile.write(b"Command executed successfully.") + except Exception as e: + self.send_response(500) + self.end_headers() + self.wfile.write(f"Error executing command: {e}".encode()) + else: + self.send_response(404) + self.end_headers() + self.wfile.write(b"Not found.") + +if __name__ == "__main__": + server_address = ('server_ip', 8081) + httpd = HTTPServer(server_address, RequestHandler) + print("Server running on http://server_ip. Access /run to execute the command.") + httpd.serve_forever() +``` + +### 9.2: `server.py` + +This script pulls the logs by executing the previously created bash script: + +```python +from http.server import BaseHTTPRequestHandler, HTTPServer +import subprocess + +class ScriptHandler(BaseHTTPRequestHandler): + def do_GET(self): + script_path = "/path/to/transfer_logs.sh" + + try: + subprocess.run([script_path], check=True) + self.send_response(200) + self.send_header("Content-type", "text/html") + self.end_headers() + self.wfile.write(b"Logs Pulled Successfully.") + except subprocess.CalledProcessError: + self.send_response(500) + self.send_header("Content-type", "text/html") + self.end_headers() + self.wfile.write(b"Failed to pull logs.") + +def run(server_class=HTTPServer, handler_class=ScriptHandler, port=8080): + server_address = ('', port) + httpd = server_class(server_address, handler_class) + print(f"Starting http server on port {port}") + httpd.serve_forever() + +if __name__ == "__main__": + run() +``` + +Run both Python scripts in the background using `nohup`: + +```bash +nohup python3 transfer_server.py > transfer_server.log 2>&1 & +nohup python3 server.py > server.log 2>&1 & +``` + +--- + +## Step 10: Add Buttons to Grafana Dashboard + +1. In Grafana, go to **Settings** > **Links**. + +2. Add the links to the above Python scripts with the appropriate port (`8080` for log pulling and `8081` for triggering Promtail). + +3. This will enable users to pull logs instantly as they wish. Make sure promtail activation button is run first when opening dashboard, and then pulling logs button after. + +--- From 27c0b29350797227bbb2507c01fed16a643c50a2 Mon Sep 17 00:00:00 2001 From: Rudra Singh <136931703+rudra-singh1@users.noreply.github.com> Date: Tue, 19 Nov 2024 11:46:27 -0800 Subject: [PATCH 04/11] Update grafana_log_dashboard.md --- docs/infrastructure/grafana_log_dashboard.md | 42 ++++++++++---------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/docs/infrastructure/grafana_log_dashboard.md b/docs/infrastructure/grafana_log_dashboard.md index 059c2a6..e910442 100644 --- a/docs/infrastructure/grafana_log_dashboard.md +++ b/docs/infrastructure/grafana_log_dashboard.md @@ -39,7 +39,7 @@ services: networks: - loki volumes: - - grafana-data:/path # Adding persistence for Grafana data + - grafana-data:/path # [CHANGE THIS LINE] Adding persistence for Grafana data loki: image: grafana/loki:latest @@ -48,16 +48,16 @@ services: networks: - loki volumes: - - ./path # Local directory for Loki data + - ./path # [CHANGE THIS LINE] Local directory for Loki data user: "10001" # Run Loki as the appropriate user to avoid permission issues promtail: image: grafana/promtail:latest - command: "-config.file=/etc/promtail/config.yaml" + command: "-config.file=/etc/promtail/config.yaml" # [CHANGE THIS LINE] networks: - loki volumes: - - "/var/log:/var/log" # Mounting host logs + - "/var/log:/var/log" # [CHANGE THIS LINE] Mounting host logs networks: loki: @@ -79,6 +79,12 @@ volumes: To aggregate logs from other servers, we set up a method of transferring logs to the Grafana server. The logs are pulled into the Grafana server, where they are processed and rendered on a Grafana dashboard. +Here’s the updated section of the `README` with the requested changes marked. The rest remains unchanged. + +--- + +## Step 5: Configure Data Sources (Logs) + ### 5.1: Write a Bash Script to Pull Logs Here’s an example of a script (`transfer_logs.sh`) to pull logs from another server: @@ -87,14 +93,14 @@ Here’s an example of a script (`transfer_logs.sh`) to pull logs from another s #!/bin/bash # Define variables -REMOTE_SERVER="root@server_ip" # Remote server IP -SSH_KEY_PATH="path" # Path to SSH key -DEST_DIR="path" # Destination directory on the Grafana server +REMOTE_SERVER="root@server_ip" # [CHANGE THIS LINE] Remote server IP +SSH_KEY_PATH="path" # [CHANGE THIS LINE] Path to SSH key +DEST_DIR="path" # [CHANGE THIS LINE] Destination directory on the Grafana server # Define paths for logs -DOCKER_LOG="path/to/docker/log" -NGINX_LOG="path/to/nginx/log" -SYSLOG="path/to/syslog" +DOCKER_LOG="path/to/docker/log" # [CHANGE THIS LINE] Path to Docker log on remote server +NGINX_LOG="path/to/nginx/log" # [CHANGE THIS LINE] Path to NGINX log on remote server +SYSLOG="path/to/syslog" # [CHANGE THIS LINE] Path to syslog on remote server # Create directories if they don't exist mkdir -p "$DEST_DIR/docker" "$DEST_DIR/nginx" "$DEST_DIR/syslog" @@ -127,30 +133,24 @@ positions: filename: /tmp/positions.yaml clients: - - url: http://server_ip:3100/loki/api/v1/push # Loki's URL + - url: http://server_ip:3100/loki/api/v1/push # [CHANGE THIS LINE] Loki's URL scrape_configs: - job_name: docker_logs static_configs: labels: job: docker - __path__: /path/to/logs/on/grafana/server/docker/*.log + __path__: /path/to/logs/on/grafana/server/docker/*.log # [CHANGE THIS LINE] Path to Docker logs - job_name: nginx_logs static_configs: labels: job: nginx - __path__: /path/to/logs/on/grafana/server/nginx/*.log + __path__: /path/to/logs/on/grafana/server/nginx/*.log # [CHANGE THIS LINE] Path to NGINX logs - job_name: syslog_logs static_configs: labels: job: syslog - __path__: /path/to/logs/on/grafana/server/syslog/*.log -``` - -Run Promtail with this configuration: - -```bash -sudo ./promtail-linux-amd64 -config.file=promtail-config.yaml + __path__: /path/to/logs/on/grafana/server/syslog/*.log # [CHANGE THIS LINE] Path to syslog logs ``` --- @@ -243,7 +243,7 @@ import subprocess class ScriptHandler(BaseHTTPRequestHandler): def do_GET(self): - script_path = "/path/to/transfer_logs.sh" + script_path = "/path/to/transfer_logs.sh" # [CHANGE THIS LINE] try: subprocess.run([script_path], check=True) From 09189f01414e87f09936faa2c6fb049c81797215 Mon Sep 17 00:00:00 2001 From: Rudra Singh <136931703+rudra-singh1@users.noreply.github.com> Date: Sat, 23 Nov 2024 14:00:32 -0800 Subject: [PATCH 05/11] Update grafana_log_dashboard.md --- docs/infrastructure/grafana_log_dashboard.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/infrastructure/grafana_log_dashboard.md b/docs/infrastructure/grafana_log_dashboard.md index e910442..4ffbbba 100644 --- a/docs/infrastructure/grafana_log_dashboard.md +++ b/docs/infrastructure/grafana_log_dashboard.md @@ -1,3 +1,7 @@ +--- +title: Grafana Log Dashboard Guide +--- + # Grafana Server Log Monitoring Dashboard Setup ## Developed By: Rudra Prakash Singh, Esther Jang From e96a79a6e9df2be4058f7e0cf8fcde57c3154b47 Mon Sep 17 00:00:00 2001 From: Rudra Singh <136931703+rudra-singh1@users.noreply.github.com> Date: Sat, 23 Nov 2024 14:01:14 -0800 Subject: [PATCH 06/11] Update grafana_log_dashboard.md --- docs/infrastructure/grafana_log_dashboard.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/infrastructure/grafana_log_dashboard.md b/docs/infrastructure/grafana_log_dashboard.md index e910442..a8f415b 100644 --- a/docs/infrastructure/grafana_log_dashboard.md +++ b/docs/infrastructure/grafana_log_dashboard.md @@ -1,3 +1,6 @@ +--- +title: Grafana Logging Dashboard Guide +--- # Grafana Server Log Monitoring Dashboard Setup ## Developed By: Rudra Prakash Singh, Esther Jang From f06952e62318b8be94fd701d575a5a3a39fcf3c9 Mon Sep 17 00:00:00 2001 From: Rudra Singh <136931703+rudra-singh1@users.noreply.github.com> Date: Sat, 23 Nov 2024 14:08:13 -0800 Subject: [PATCH 07/11] Update grafana_log_dashboard.md --- docs/infrastructure/grafana_log_dashboard.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/infrastructure/grafana_log_dashboard.md b/docs/infrastructure/grafana_log_dashboard.md index a8f415b..b9096de 100644 --- a/docs/infrastructure/grafana_log_dashboard.md +++ b/docs/infrastructure/grafana_log_dashboard.md @@ -1,6 +1,7 @@ --- title: Grafana Logging Dashboard Guide --- + # Grafana Server Log Monitoring Dashboard Setup ## Developed By: Rudra Prakash Singh, Esther Jang From fc26a69efaf91deb7271fc652ff4c2aefc3ab718 Mon Sep 17 00:00:00 2001 From: Rudra Singh <136931703+rudra-singh1@users.noreply.github.com> Date: Sat, 23 Nov 2024 14:47:54 -0800 Subject: [PATCH 08/11] Update and rename grafana_log_dashboard.md to grafana-log-dashboard.md --- ...{grafana_log_dashboard.md => grafana-log-dashboard.md} | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) rename docs/infrastructure/{grafana_log_dashboard.md => grafana-log-dashboard.md} (98%) diff --git a/docs/infrastructure/grafana_log_dashboard.md b/docs/infrastructure/grafana-log-dashboard.md similarity index 98% rename from docs/infrastructure/grafana_log_dashboard.md rename to docs/infrastructure/grafana-log-dashboard.md index dcfd4d0..13be1a3 100644 --- a/docs/infrastructure/grafana_log_dashboard.md +++ b/docs/infrastructure/grafana-log-dashboard.md @@ -1,10 +1,10 @@ --- -title: Grafana Logging Dashboard Guide +title: Grafana Log Dashboard Guide --- -# Grafana Server Log Monitoring Dashboard Setup +### Developed By: Rudra Prakash Singh, Esther Jang -## Developed By: Rudra Prakash Singh, Esther Jang +--- This guide walks you through the process of setting up a Grafana Server Log Monitoring Dashboard, including the installation and configuration of Grafana, Loki, and Promtail for log aggregation and visualization. It also covers setting up scripts for automating log retrieval and dashboard updates. @@ -287,5 +287,3 @@ nohup python3 server.py > server.log 2>&1 & 2. Add the links to the above Python scripts with the appropriate port (`8080` for log pulling and `8081` for triggering Promtail). 3. This will enable users to pull logs instantly as they wish. Make sure promtail activation button is run first when opening dashboard, and then pulling logs button after. - ---- From bf7d7dacbd4355e1442d98bc0bba1798a5988e8e Mon Sep 17 00:00:00 2001 From: Rudra Singh <136931703+rudra-singh1@users.noreply.github.com> Date: Sat, 23 Nov 2024 17:50:12 -0500 Subject: [PATCH 09/11] Add files via upload --- docs/tutorials/grafana-log-dashboard.md | 289 ++++++++++++++++++++++++ 1 file changed, 289 insertions(+) create mode 100644 docs/tutorials/grafana-log-dashboard.md diff --git a/docs/tutorials/grafana-log-dashboard.md b/docs/tutorials/grafana-log-dashboard.md new file mode 100644 index 0000000..13be1a3 --- /dev/null +++ b/docs/tutorials/grafana-log-dashboard.md @@ -0,0 +1,289 @@ +--- +title: Grafana Log Dashboard Guide +--- + +### Developed By: Rudra Prakash Singh, Esther Jang + +--- + +This guide walks you through the process of setting up a Grafana Server Log Monitoring Dashboard, including the installation and configuration of Grafana, Loki, and Promtail for log aggregation and visualization. It also covers setting up scripts for automating log retrieval and dashboard updates. + +--- + +## Step 1: Install Grafana via CLI + +1. **Install Grafana** on your server using the following instructions for Debian-based systems: + - Follow the installation guide here: [Grafana Installation on Debian](https://grafana.com/docs/grafana/latest/setup-grafana/installation/debian/) + +2. Once installed, start Grafana and make sure it's accessible via a port (default: `3000`). + +3. After Grafana is running, create an admin account. Once logged in, you'll land on the Grafana homepage. + +--- + +## Step 2: Install Loki + +1. **Install Loki** by following the official instructions for Docker: + - [Loki Installation via Docker](https://grafana.com/docs/loki/latest/setup/install/docker/) + +--- + +## Step 3: Create a Docker Compose File for Loki + +To set up Loki with Grafana, create a `docker-compose.yml` file: + +```yaml +version: "3" # The version can be removed if you're using a recent Docker Compose version. + +services: + grafana: + image: grafana/grafana:latest + ports: + - "3000:3000" + networks: + - loki + volumes: + - grafana-data:/path # [CHANGE THIS LINE] Adding persistence for Grafana data + + loki: + image: grafana/loki:latest + ports: + - "3100:3100" + networks: + - loki + volumes: + - ./path # [CHANGE THIS LINE] Local directory for Loki data + user: "10001" # Run Loki as the appropriate user to avoid permission issues + + promtail: + image: grafana/promtail:latest + command: "-config.file=/etc/promtail/config.yaml" # [CHANGE THIS LINE] + networks: + - loki + volumes: + - "/var/log:/var/log" # [CHANGE THIS LINE] Mounting host logs + +networks: + loki: + +volumes: + grafana-data: # Declaring the volume for Grafana +``` + +--- + +## Step 4: Install Promtail + +1. **Install Promtail** by following the official instructions: + - [Promtail Installation Guide](https://grafana.com/docs/loki/latest/send-data/promtail/installation/) + +--- + +## Step 5: Configure Data Sources (Logs) + +To aggregate logs from other servers, we set up a method of transferring logs to the Grafana server. The logs are pulled into the Grafana server, where they are processed and rendered on a Grafana dashboard. + +Here’s the updated section of the `README` with the requested changes marked. The rest remains unchanged. + +--- + +## Step 5: Configure Data Sources (Logs) + +### 5.1: Write a Bash Script to Pull Logs + +Here’s an example of a script (`transfer_logs.sh`) to pull logs from another server: + +```bash +#!/bin/bash + +# Define variables +REMOTE_SERVER="root@server_ip" # [CHANGE THIS LINE] Remote server IP +SSH_KEY_PATH="path" # [CHANGE THIS LINE] Path to SSH key +DEST_DIR="path" # [CHANGE THIS LINE] Destination directory on the Grafana server + +# Define paths for logs +DOCKER_LOG="path/to/docker/log" # [CHANGE THIS LINE] Path to Docker log on remote server +NGINX_LOG="path/to/nginx/log" # [CHANGE THIS LINE] Path to NGINX log on remote server +SYSLOG="path/to/syslog" # [CHANGE THIS LINE] Path to syslog on remote server + +# Create directories if they don't exist +mkdir -p "$DEST_DIR/docker" "$DEST_DIR/nginx" "$DEST_DIR/syslog" + +# Transfer logs +echo "Transferring Docker log..." +ssh -i $SSH_KEY_PATH $REMOTE_SERVER "cat $DOCKER_LOG" > "$DEST_DIR/docker/docker.log" + +echo "Transferring NGINX log..." +ssh -i $SSH_KEY_PATH $REMOTE_SERVER "cat $NGINX_LOG" > "$DEST_DIR/nginx/error.log" + +echo "Transferring syslog..." +ssh -i $SSH_KEY_PATH $REMOTE_SERVER "cat $SYSLOG" > "$DEST_DIR/syslog/syslog.log" + +echo "All logs have been successfully transferred." +``` + +--- + +### 5.2: Configure Promtail to Scrape Logs + +Next, create a `promtail-config.yaml` file to let Promtail know where the logs are located: + +```yaml +server: + http_listen_port: 9080 + grpc_listen_port: 0 + +positions: + filename: /tmp/positions.yaml + +clients: + - url: http://server_ip:3100/loki/api/v1/push # [CHANGE THIS LINE] Loki's URL + +scrape_configs: + - job_name: docker_logs + static_configs: + labels: + job: docker + __path__: /path/to/logs/on/grafana/server/docker/*.log # [CHANGE THIS LINE] Path to Docker logs + - job_name: nginx_logs + static_configs: + labels: + job: nginx + __path__: /path/to/logs/on/grafana/server/nginx/*.log # [CHANGE THIS LINE] Path to NGINX logs + - job_name: syslog_logs + static_configs: + labels: + job: syslog + __path__: /path/to/logs/on/grafana/server/syslog/*.log # [CHANGE THIS LINE] Path to syslog logs +``` + +--- + +## Step 6: Configure Grafana to Use Loki + +1. Open Grafana’s homepage and click the **three vertical lines** in the top-left corner. + +2. Navigate to **Data Sources**, then click **Add Data Source**. + +3. Select **Loki** as the data source type. + +4. Define the **URL** of your Loki instance, which is typically `http://localhost:3100`. + +5. Click **Save & Test**. You should see a success message confirming the connection. + +--- + +## Step 7: Explore Data in Grafana + +1. Go to the **Explore** section in Grafana. + +2. Use label filters and run queries to see the logs coming from Promtail. The labels in the filters correspond to the `labels` defined in the `promtail-config.yaml`. + +--- + +## Step 8: Create a Dashboard + +1. On the left sidebar in Grafana, click **Dashboards** > **New Dashboard**. + +2. Click **Add Panel** to create a new panel. + +3. Under **Data Source**, select **Loki**. + +4. Write queries to retrieve logs for display. Use different labels to filter log data as needed. + +5. Choose a visualization type (e.g., **Logs**). + +6. Repeat the process to add more visualizations, then adjust the layout and size to suit your needs. + +7. Save the dashboard once you are satisfied with the layout. + +--- + +## Step 9: Automate Log Retrieval and Update Dashboard + +To make the process of pulling logs and updating the Grafana dashboard more automated, create two Python scripts: + +### 9.1: `transfer_server.py` + +This script triggers the Promtail process to start log scraping when accessed via an HTTP request: + +```python +from http.server import BaseHTTPRequestHandler, HTTPServer +import subprocess + +COMMAND = "sudo ./promtail-linux-amd64 -config.file=promtail-config.yaml" + +class RequestHandler(BaseHTTPRequestHandler): + def do_GET(self): + if self.path == "/run": + try: + subprocess.Popen(COMMAND, shell=True) + self.send_response(200) + self.end_headers() + self.wfile.write(b"Command executed successfully.") + except Exception as e: + self.send_response(500) + self.end_headers() + self.wfile.write(f"Error executing command: {e}".encode()) + else: + self.send_response(404) + self.end_headers() + self.wfile.write(b"Not found.") + +if __name__ == "__main__": + server_address = ('server_ip', 8081) + httpd = HTTPServer(server_address, RequestHandler) + print("Server running on http://server_ip. Access /run to execute the command.") + httpd.serve_forever() +``` + +### 9.2: `server.py` + +This script pulls the logs by executing the previously created bash script: + +```python +from http.server import BaseHTTPRequestHandler, HTTPServer +import subprocess + +class ScriptHandler(BaseHTTPRequestHandler): + def do_GET(self): + script_path = "/path/to/transfer_logs.sh" # [CHANGE THIS LINE] + + try: + subprocess.run([script_path], check=True) + self.send_response(200) + self.send_header("Content-type", "text/html") + self.end_headers() + self.wfile.write(b"Logs Pulled Successfully.") + except subprocess.CalledProcessError: + self.send_response(500) + self.send_header("Content-type", "text/html") + self.end_headers() + self.wfile.write(b"Failed to pull logs.") + +def run(server_class=HTTPServer, handler_class=ScriptHandler, port=8080): + server_address = ('', port) + httpd = server_class(server_address, handler_class) + print(f"Starting http server on port {port}") + httpd.serve_forever() + +if __name__ == "__main__": + run() +``` + +Run both Python scripts in the background using `nohup`: + +```bash +nohup python3 transfer_server.py > transfer_server.log 2>&1 & +nohup python3 server.py > server.log 2>&1 & +``` + +--- + +## Step 10: Add Buttons to Grafana Dashboard + +1. In Grafana, go to **Settings** > **Links**. + +2. Add the links to the above Python scripts with the appropriate port (`8080` for log pulling and `8081` for triggering Promtail). + +3. This will enable users to pull logs instantly as they wish. Make sure promtail activation button is run first when opening dashboard, and then pulling logs button after. From ef2b108b40ec4e836c475ef58324e997ed006c84 Mon Sep 17 00:00:00 2001 From: Rudra Singh <136931703+rudra-singh1@users.noreply.github.com> Date: Sat, 23 Nov 2024 14:52:50 -0800 Subject: [PATCH 10/11] Update .pages --- docs/tutorials/.pages | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/tutorials/.pages b/docs/tutorials/.pages index b674ae4..8b5048d 100644 --- a/docs/tutorials/.pages +++ b/docs/tutorials/.pages @@ -7,4 +7,5 @@ nav: - epc-setup.md - enb-setup.md - proxmox-vaultwarden-deployment.md + - grafana-log-dashboard.md - librenms From 40dbdac8dbc7cd8d5e7369d7e4e52eb46d73709e Mon Sep 17 00:00:00 2001 From: Esther Jang Date: Sat, 23 Nov 2024 18:26:38 -0800 Subject: [PATCH 11/11] Update space.md --- docs/community/space.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/community/space.md b/docs/community/space.md index ac3f0c6..6ebf765 100644 --- a/docs/community/space.md +++ b/docs/community/space.md @@ -3,11 +3,10 @@ title: SCN Community Lab Space --- # Seattle Community Network Lab Space -## This page under construction! -We now have a newly opening community network lab and hack space, co-located with the gracious and welcoming [/dev/hack](https://devhack.net) hackerspace in Seattle, WA, with whom we share full mutual membership. +We now have a newly opened community network lab and hack space, co-located with the gracious and welcoming [/dev/hack](https://devhack.net) hackerspace in Seattle, WA, with whom we share mutual membership. Address: 4534 1/2 University Way NE, Seattle WA 98105 SCN Lab Space Membership is for 24/7 keyholder access to the space only, and is absolutely not required to volunteer with us. Members can bring in guests at any time. -Here are the current member [application](https://www.cognitoforms.com/TheSilentTaskForce1/SCNSpaceMembershipApplicationFormAndAgreement) and working draft of [policies](https://docs.google.com/document/d/1jHKDoY01IUAhroEsmL_nff-2he1DZx-30uwkQw4jY6A/edit). +Here are the current member [application](https://www.cognitoforms.com/TheSilentTaskForce1/SCNSpaceMembershipApplicationFormAndAgreement) and [policies](https://docs.google.com/document/d/1jHKDoY01IUAhroEsmL_nff-2he1DZx-30uwkQw4jY6A/edit).