go-watch-logs revamped - God speed (#6)

* (fix) re do watcher donot tail - tail ended up creating multiple pids for the process and not reliable moved to read the file, keep a track of the last line using on disk kvs alerts using summary of first and last line use file size to detect the log rotation support scanning of large files without keeping in memory * (feat) added teams msg * (feat) added tests * (docs) updated README for v2 * (ci) lint issues * (bench) added benchmarks * (ci) lint fix
rakutentech · Jun 4, 2024 · 15d4de3 · 15d4de3
1 parent 3c1a09a
commit 15d4de3
Show file tree

Hide file tree

Showing 19 changed files with 817 additions and 424 deletions.
diff --git a/.env.example b/.env.example
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -0,0 +1,29 @@
+name: goreleaser
+
+on:
+  push:
+    tags:
+      - '*'
+
+jobs:
+  goreleaser:
+    runs-on: ubuntu-latest
+    steps:
+      -
+        name: Checkout
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      - uses: kevincobain2000/action-gobrew@v2
+        with:
+          version: 'mod'
+
+      - name: Run GoReleaser
+        uses: goreleaser/goreleaser-action@v5
+        with:
+          workdir: app
+          distribution: goreleaser
+          version: latest
+          args: release --clean --skip=validate
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -0,0 +1,31 @@
+name: test
+
+on:
+  pull_request:
+  push:
+    tags-ignore:
+      - '**'
+    branches:
+      - '**'
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+      - uses: kevincobain2000/action-gobrew@v2
+        with:
+          version: 'mod'
+
+      - name: Install Tools
+        run: |
+          go install github.com/axw/gocov/gocov@latest
+          go install github.com/AlekSi/gocov-xml@latest
+          go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
+          curl -sLk https://raw.githubusercontent.com/kevincobain2000/cover-totalizer/master/install.sh | sh
+
+      - run: go mod tidy
+      - run: golangci-lint run ./...
+      - run: go build
+      - run: go test ./...
diff --git a/.gitignore b/.gitignore
@@ -10,11 +10,14 @@
 
 # Output of the go coverage tool, specifically when used with LiteIDE
 *.out
-# diskcache directory 
+# diskcache directory
 cache/*
 vendor/*
 
 tmp/
 
 .env
-*.pid
+*.pid
+*.log
+*.db
+.DS_Store
diff --git a/.golangci.yaml b/.golangci.yaml
@@ -0,0 +1,22 @@
+linters:
+  disable-all: true
+  # Enable specific linter
+  # https://golangci-lint.run/usage/linters/#enabled-by-default
+  enable:
+    - errcheck
+    - gosimple
+    - govet
+    - ineffassign
+    - staticcheck
+    - dupl
+    - errorlint
+    - exportloopref
+    - goconst
+    - gocritic
+    - gocyclo
+    - goprintffuncname
+    - gosec
+    - prealloc
+    - revive
+    - stylecheck
+    - whitespace
diff --git a/.goreleaser.yaml b/.goreleaser.yaml
@@ -0,0 +1,41 @@
+before:
+  hooks:
+    - go mod tidy
+builds:
+  -
+    main: main.go
+    env:
+      - CGO_ENABLED=0
+    goos:
+      - linux
+      - darwin
+      # - windows
+      - freebsd
+    goarch:
+      - amd64
+      - arm64
+      - arm
+    ignore:
+      - goos: windows
+        goarch: arm64
+      - goos: freebsd
+        goarch: arm64
+      - goos: windows
+        goarch: arm
+      - goos: freebsd
+        goarch: arm
+      - goos: darwin
+        goarch: arm
+archives:
+  -
+    format: binary
+    name_template: "{{ .ProjectName }}-{{ .Os }}-{{ .Arch }}"
+snapshot:
+  name_template: "{{ incpatch .Version }}-next"
+changelog:
+  sort: asc
+  filters:
+    exclude:
+      - '^docs:'
+      - '^test:'
+      - '^bin'
diff --git a/README.md b/README.md
@@ -1,119 +1,78 @@
-## What is ``go-watch-logs``
+<p align="center">
+  Go Watch Logs<br>
+  Monitor static logs file for patterns and send alerts to MS Teams<br>
+</p>
 
-1. ``go-watch-logs`` is a standalone CLI which can tail any console or API request and logs.
-2. You can configure ``go-watch-logs`` to send a notification whenever a regexp match is found on a tailed log file.
-3. ``go-watch-logs`` supports sending alert notifications to MS teams and via Email. For more info, see [go-alertnotification](https://github.com/rakutentech/go-alertnotification).
-4. ``go-watch-logs`` has an inbuilt throttler counter which allows you to set a limit to num of alert messages for a regexp within the duration.
+**Quick Setup:** One command to install Go and manage versions.
+
+**Hassle Free:** Doesn't require root or sudo.
+
+**Platform:** Supports (arm64, arch64, Mac, Mac M1, Ubuntu and Windows).
+
+**Flexible:** Works with any logs file, huge to massive, log rotation is supported.
+
+**Notify:** Supports MS Teams. Emails, Slack (coming soon).
 
 ### Install using go
 
 ```bash
-export GO111MODULE=on
-go get -u github.com/rakutentech/go-watch-logs
+go install github.com/rakutentech/go-watch-logs@latest
 go-watch-logs --help
 ```
 
-### Or Install using executable
+### Install using curl
 
 Use this method if go is not installed on your server
 
 ```bash
-git clone https://github.com/rakutentech/go-watch-logs/
-./bin/linux/go-watch-logs --help
-```
-
-## Run
-
-```bash
-cp .env.example .env
-set -a && . ./.env && set +a
-go-watch-logs --watch-file=/path/to/logs.log "regexp1" "regexp2" > /dev/null 2>&1 &
+curl -sL https://raw.githubusercontent.com/rakutentech/go-watch-logs/master/install.sh | sh
 ```
 
 ## Run it on a cron
 
-``go-watch-logs`` shouldn't kill itself even when the watching file is removed. It is capable of recovering itself.
-But it is good idea to add it to your cronjob.
 
 ```
-* * * * * set -a && . /path/to/.env && set +a && flock -n /tmp/go-watch-logs.lock go-watch-logs --watch-file=/path/to/logs.log "regexp1" "regexp2" > /dev/null 2>&1 &
+* * * * * go-watch-logs --file-path=my.log --match="error:pattern1|error:pattern2" --ms-teams-hook="https://outlook.office.com/webhook/xxxxx"
 ```
 
 
 **All done!**
 
 ## Help
 
-```
---help
-        Prints this Usage
---limit int
-        Limit to notify (default 10)
---seconds int
-        Limit notify per number of second (default 30)
---offset int
-	Offset Limit to ignore first few (default 0)
---watch-file string
-        Path to the file to tail
--ignore-regexp string
-        One regexp to ignore reporting --ignore-regexp="donotmatch1|donotmatch2"
--recovery-cmd string
-        Shell cmd to execute on match found (default "")
-
-Basic Usage:
-        go-watch-logs --limit=10 --seconds=30 --watch-file=/path/to/your.log  "regexp1" "regexp2"
-Description:
-        Will send max 10 notifications every 30 seconds for regexp1 matched per line in your.log file
-        And will send max 10 notifications every 30 seconds for regexp2 matched per line in your.log file
-```
-
-## Build it your self
-
 ```sh
-GOOS=linux GOARCH=amd64 go build watch_logs.go; mv watch_logs ./bin/linux/go-watch-logs
-GOOS=darwin GOARCH=amd64 go build watch_logs.go; mv watch_logs ./bin/mac/go-watch-logs
+  -db-path string
+    	path to db file (default "go-watch-logs.db")
+  -file-path string
+    	path to logs file
+  -ignore string
+    	ignore pattern
+  -match string
+    	match pattern
+  -ms-teams-hook string
+    	ms teams webhook
+  -version
+    	print version
 ```
 
+
 ----
 
 ## Performance Notes
 
-1. Load on http server
-
-![load.png](./docs/load.png)
-
-2. ``go-watch-logs`` on the ``ssl_access_log`` for ``HTTP/1.1" 50`` errors.
-
 ```
-go-watch-logs --limit=2 --seconds=60 --watch-file=/var/log/httpd/vhost/ssl_access_log "HTTP/1.1\" 50"
+BenchmarkReadFileAndMatchErrors-10    	   10328	    113576 ns/op	    8543 B/op	      49 allocs/op
+BenchmarkReadFileAndMatchErrors-10    	   10000	    115039 ns/op	    8543 B/op	      49 allocs/op
+BenchmarkSetAndGetLastLineNum-10      	 3247788	       369.2 ns/op	     490 B/op	       8 allocs/op
+BenchmarkSetAndGetLastLineNum-10      	 3224370	       370.3 ns/op	     490 B/op	       8 allocs/op
+BenchmarkLoadAndSaveState-10          	   14349	     74996 ns/op	   11479 B/op	     103 allocs/op
+BenchmarkLoadAndSaveState-10          	   15531	     79613 ns/op	   12916 B/op	     177 allocs/op
+BenchmarkLogRotation-10               	    9644	    128297 ns/op	   10326 B/op	      89 allocs/op
+BenchmarkLogRotation-10               	    9604	    126398 ns/op	   10295 B/op	      88 allocs/op
 ```
 
-3. ``go-watch-logs`` doesn't consume much memory.
-
-Check the RSS is the Resident Set Size and which is used to show how much memory is allocated to that process and is in RAM. VSZ is the Virtual Memory Size. It includes all memory that the process can access
-
-> Total memory used through out is **12720 KB** i.e. **12 MB**
-
-```s
-Address           Kbytes     RSS   Dirty Mode  Mapping
-total kB          709040   12720    6552
-```
-
-> Viewing httpd app and ``go-watch-logs`` pids with top command
-
-```s
-PID    USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM   TIME+ COMMAND
-108778 appuser+  20   0  375916  10340   4864 S   1.0  0.3   0:18.91 main
-108770 appuser+  20   0  375916  10352   4884 S   1.0  0.3   0:18.88 main
-125938 appuser+  20   0  496132   2600   1884 S   0.0  0.1   0:00.99 start_server
-125902 appuser+  20   0  496132   8008   1620 S   0.0  0.2   0:01.17 start_server
- 23545 appuser+  20   0  709036  12340   3920 S   0.0  0.3   0:01.65 go-watch-logs
- 45145 apache    20   0  142776   3052    640 S   0.0  0.1   0:09.71 httpd
- 99944 root      20   0  142776   6944   4544 S   0.0  0.2   0:15.16 httpd
-```
 
 ## Credits
 
 1. https://github.com/rakutentech/go-alertnotification
-2. https://github.com/hpcloud/tail
 
diff --git a/bin/linux/go-watch-logs b/bin/linux/go-watch-logs
diff --git a/docs/load.png b/docs/load.png
diff --git a/go.mod b/go.mod
@@ -1,13 +1,26 @@
 module github.com/rakutentech/go-watch-logs
 
-go 1.14
+go 1.21
+
+toolchain go1.22.3
+
+require (
+	github.com/rakutentech/go-alertnotification v1.1.3
+	github.com/stretchr/testify v1.9.0
+	github.com/tidwall/buntdb v1.3.1
+)
 
 require (
-	github.com/hpcloud/tail v1.0.0
-	github.com/k0kubun/pp v3.0.1+incompatible // indirect
-	github.com/mattn/go-colorable v0.1.8 // indirect
-	github.com/rakutentech/go-alertnotification v1.0.2
-	golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae // indirect
-	gopkg.in/fsnotify.v1 v1.4.7 // indirect
-	gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect
+	github.com/GitbookIO/diskache v0.0.0-20161028144708-bfb81bf58cb1 // indirect
+	github.com/GitbookIO/syncgroup v0.0.0-20200915204659-4f0b2961ab10 // indirect
+	github.com/davecgh/go-spew v1.1.1 // indirect
+	github.com/pmezard/go-difflib v1.0.0 // indirect
+	github.com/tidwall/btree v1.4.2 // indirect
+	github.com/tidwall/gjson v1.14.3 // indirect
+	github.com/tidwall/grect v0.1.4 // indirect
+	github.com/tidwall/match v1.1.1 // indirect
+	github.com/tidwall/pretty v1.2.0 // indirect
+	github.com/tidwall/rtred v0.1.2 // indirect
+	github.com/tidwall/tinyqueue v0.1.1 // indirect
+	gopkg.in/yaml.v3 v3.0.1 // indirect
 )