Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactoring for better separation of concerns #110

Merged
merged 12 commits into from
Apr 13, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,15 +1,22 @@
.PHONY: all build test docker
.PHONY: all build build-sql test docker

all: build test

DOCKER_IMAGE := checkup

build:
go fmt ./...
go mod tidy
mkdir -p builds/
go build -o builds/ ./cmd/...

build-sql:
go fmt ./...
go mod tidy
go build -o builds/ -tags sql ./cmd/...

test:
go test -race -count=1 -v ./...
go test -race -count=1 ./...

docker:
docker build --no-cache . -t $(DOCKER_IMAGE)
97 changes: 57 additions & 40 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,20 @@ Checkup was created by Matt Holt, author of the [Caddy web server](https://caddy

This tool is a work-in-progress. Please use liberally (with discretion) and report any bugs!

## Recent changes

Due to recent development, some breaking changes have been introduced:

- providers: the json config field `provider` was renamed to `type` for consistency,
- notifiers: the json config field `name` was renamed to `type` for consistency,
- sql: by default the sqlite storage engine is disabled (needs build with `-tags sql` to enable),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there's any way to call this out more aggressively, that would be great! Perhaps a CHANGELOG? My crons stopped running overnight due to (1) me using r.j3ss.co/checkup:latest, which tracks HEAD and (2) these changes. Specifically, the provider -> type change for storage caught me. For anyone relying on HEAD for their production checks, this is likely to bite them.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm very sorry about that. It was a risk I was aware of when I was cleaning up the code for this, and I just hoped people aren't blindly doing nightly/HEAD builds, and tried to make it as obvious as possible that there was a breaking change introduced by prominently updating the README at the very top.

Aside from adding a CHANGELOG file,

  1. we can decide on a branching model (master|develop, v0, v1...), AND
  2. we can remove tags as they are unused/out of date (new releases needed #113 Project status? #86 slack: unknown Notifier type #84 ...) OR
  3. we can tag a 0.3.0 before breaking changes and 0.4.0 after

And finally, we could automate the release process in line with what we choose above (#96). There are also a number of issues that could be solved by having a rolling release model:

The current releases are woefully out of date. I'm voting to remove them, keep the development on master, branch to v0 before the breaking changes (f4747239), and branch into v1 later when we feel the release is pretty stable and non-breaking.

@beyang thoughts?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@parkr sorry this bit you. We don't have a good sense of who's using this at the moment, so yes, we should probably adopt a versioning system as @titpetric proposes. @titpetric the model you propose sounds good to me. I'd propose this model:

  • master is the development branch
  • We adopt semver. I've pushed tags v1.0.0 (the last commit before @titpetric's recent changes) and v2.0.0 (the latest master commit). I've also pushed a 1.0 branch in case we want to cut any new patches off the older version

@titpetric feel free to push up a CHANGELOG file as well. Also not we did publish release notes for early versions of Checkup: https://github.com/sourcegraph/checkup/releases.


If you want to build the latest version, it's best to run:

- `make build` - builds checkup without sql support,
- `make build-sql` - builds checkup with pgsql/sqlite;

The resulting binary will be placed into `builds/checkup`.

## Intro

Expand Down Expand Up @@ -76,9 +89,9 @@ You can configure Checkup entirely with a simple JSON document. You should confi
// storage configuration goes here
},

"notifier": {
"notifiers": [
// notifier configuration goes here
}
]
}
```

Expand All @@ -90,7 +103,7 @@ Here are the configuration structures you can use, which are explained fully [in

#### HTTP Checkers

**[godoc: HTTPChecker](https://godoc.org/github.com/sourcegraph/checkup#HTTPChecker)**
**[godoc: HTTPChecker](https://godoc.org/github.com/sourcegraph/checkup/check/http)**

```js
{
Expand All @@ -104,7 +117,7 @@ Here are the configuration structures you can use, which are explained fully [in

#### TCP Checkers

**[godoc: TCPChecker](https://godoc.org/github.com/sourcegraph/checkup#TCPChecker)**
**[godoc: TCPChecker](https://godoc.org/github.com/sourcegraph/checkup/check/tcp)**

```js
{
Expand All @@ -116,7 +129,7 @@ Here are the configuration structures you can use, which are explained fully [in

#### DNS Checkers

**[godoc: DNSChecker](https://godoc.org/github.com/sourcegraph/checkup#DNSChecker)**
**[godoc: DNSChecker](https://godoc.org/github.com/sourcegraph/checkup/check/dns)**

```js
{
Expand All @@ -129,7 +142,7 @@ Here are the configuration structures you can use, which are explained fully [in

#### TLS Checkers

**[godoc: TLSChecker](https://godoc.org/github.com/sourcegraph/checkup#TLSChecker)**
**[godoc: TLSChecker](https://godoc.org/github.com/sourcegraph/checkup/check/tls)**

```js
{
Expand All @@ -142,11 +155,11 @@ Here are the configuration structures you can use, which are explained fully [in

#### Amazon S3 Storage

**[godoc: S3](https://godoc.org/github.com/sourcegraph/checkup#S3)**
**[godoc: S3](https://godoc.org/github.com/sourcegraph/checkup/check/s3)**

```js
{
"provider": "s3",
"type": "s3",
"access_key_id": "<yours>",
"secret_access_key": "<yours>",
"bucket": "<yours>",
Expand All @@ -159,11 +172,11 @@ S3 is the default storage provider assumed by the status page, so the only chang

#### File System Storage

**[godoc: FS](https://godoc.org/github.com/sourcegraph/checkup#FS)**
**[godoc: FS](https://godoc.org/github.com/sourcegraph/checkup/storage/fs)**

```js
{
"provider": "fs",
"type": "fs",
"dir": "/path/to/your/check_files",
"url": "http://127.0.0.1:2015/check_files"
}
Expand All @@ -180,11 +193,11 @@ Then fill out [config.js](https://github.com/sourcegraph/checkup/blob/master/sta

#### GitHub Storage

**[godoc: GitHub](https://godoc.org/github.com/sourcegraph/checkup#GitHub)**
**[godoc: GitHub](https://godoc.org/github.com/sourcegraph/checkup/storage/github)**

```js
{
"provider": "github",
"type": "github",
"access_token": "some_api_access_token_with_repo_scope",
"repository_owner": "owner",
"repository_name": "repo",
Expand All @@ -209,22 +222,22 @@ Where "dir" is a subdirectory within the repo to push all the check files. Setup

#### SQL Storage (sqlite3/PostgreSQL)

**[godoc: SQL](https://godoc.org/github.com/sourcegraph/checkup#SQL)**
**[godoc: SQL](https://godoc.org/github.com/sourcegraph/checkup/storage/sql)**

Postgres or sqlite3 databases can be used as storage backends.

sqlite database file configuration:
```js
{
"provider": "sql",
"type": "sql",
"sqlite_db_file": "/path/to/your/sqlite.db"
}
```

postgresql database file configuration:
```js
{
"provider": "sql",
"type": "sql",
"postgresql": {
"user": "postgres",
"dbname": "dbname",
Expand All @@ -251,7 +264,7 @@ Currently the status page does not support SQL storage.
Enable notifications in Slack with this Notifier configuration:
```js
{
"name": "slack",
"type": "slack",
"username": "username",
"channel": "#channel-name",
"webhook": "webhook-url"
Expand All @@ -260,6 +273,26 @@ Enable notifications in Slack with this Notifier configuration:

Follow these instructions to [create a webhook](https://get.slack.help/hc/en-us/articles/115005265063-Incoming-WebHooks-for-Slack).

#### Mail notifier

Enable E-mail notifications with this Notifier configuration:
```js
{
"type": "mail",
"from": "[email protected]",
"to": [ "[email protected]", "[email protected]" ],
"subject": "Custom subject line",
"smtp": {
"server": "smtp.example.com",
"port": 25,
"username": "username",
"password": "password"
}
}
```

The settings for `subject`, `smtp.port` (default to 25), `smtp.username` and `smtp.password` are optional.

## Setting up storage on S3

The easiest way to do this is to give an IAM user these two privileges (keep the credentials secret):
Expand Down Expand Up @@ -453,34 +486,18 @@ You can implement your own Checker and Storage types. If it's general enough, fe

### Building Locally

Requires Go v1.10 or newer.
Requires Go v1.13 or newer.

```bash
git clone [email protected]:sourcegraph/checkup.git
cd checkup/cmd/checkup/

# Install dependencies
go get -v -d

# Build binary
go build -v -ldflags '-s' -o ../../checkup

# Run tests
go test -race ../../
cd checkup
make
```

### Building with Docker
Building the SQL enabled version is done with `make build-sql`.

Linux binary:

```bash
git clone [email protected]:sourcegraph/checkup.git
cd checkup
docker pull golang:latest
docker run --net=host --rm \
-v `pwd`:/project \
-w /project golang bash \
-c "cd cmd/checkup; go get -v -d; go build -v -ldflags '-s' -o ../../checkup"
```
### Building a Docker image

This will create a checkup binary in the root project folder.
If you would like to run checkup in a docker container, building it is done by running `make docker`.
It will build the version without sql support. An SQL supported docker image is currently not provided,
but there's a plan to do that in the future.
29 changes: 29 additions & 0 deletions check.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
package checkup

import (
"encoding/json"
"fmt"

"github.com/sourcegraph/checkup/check/dns"
"github.com/sourcegraph/checkup/check/exec"
"github.com/sourcegraph/checkup/check/http"
"github.com/sourcegraph/checkup/check/tcp"
"github.com/sourcegraph/checkup/check/tls"
)

func checkerDecode(typeName string, config json.RawMessage) (Checker, error) {
switch typeName {
case dns.Type:
return dns.New(config)
case exec.Type:
return exec.New(config)
case http.Type:
return http.New(config)
case tcp.Type:
return tcp.New(config)
case tls.Type:
return tls.New(config)
default:
return nil, fmt.Errorf(errUnknownCheckerType, typeName)
}
}
36 changes: 28 additions & 8 deletions dnschecker.go → check/dns/dns.go
Original file line number Diff line number Diff line change
@@ -1,15 +1,21 @@
package checkup
package dns

import (
"encoding/json"
"fmt"
"net"
"time"

"github.com/miekg/dns"

"github.com/sourcegraph/checkup/types"
)

// DNSChecker implements a Checker for TCP endpoints.
type DNSChecker struct {
// Type should match the package name
const Type = "dns"

// Checker implements a Checker for TCP endpoints.
type Checker struct {
// Name is the name of the endpoint.
Name string `json:"endpoint_name"`
// This is the name of the DNS server you are testing.
Expand All @@ -31,29 +37,43 @@ type DNSChecker struct {
Attempts int `json:"attempts,omitempty"`
}

// New creates a new Checker instance based on json config
func New(config json.RawMessage) (Checker, error) {
var checker Checker
err := json.Unmarshal(config, &checker)
return checker, err
}

// Type returns the checker package name
func (Checker) Type() string {
return Type
}

// Check performs checks using c according to its configuration.
// An error is only returned if there is a configuration error.
func (c DNSChecker) Check() (Result, error) {
func (c Checker) Check() (types.Result, error) {
if c.Attempts < 1 {
c.Attempts = 1
}

result := Result{Title: c.Name, Endpoint: c.URL, Timestamp: Timestamp()}
result := types.NewResult()
result.Title = c.Name
result.Endpoint = c.URL
result.Times = c.doChecks()

return c.conclude(result), nil
}

// doChecks executes and returns each attempt.
func (c DNSChecker) doChecks() Attempts {
func (c Checker) doChecks() types.Attempts {
var conn net.Conn

timeout := c.Timeout
if timeout == 0 {
timeout = 1 * time.Second
}

checks := make(Attempts, c.Attempts)
checks := make(types.Attempts, c.Attempts)
for i := 0; i < c.Attempts; i++ {
var err error
start := time.Now()
Expand Down Expand Up @@ -86,7 +106,7 @@ func (c DNSChecker) doChecks() Attempts {
// computes remaining values needed to fill out the result.
// It detects degraded (high-latency) responses and makes
// the conclusion about the result's status.
func (c DNSChecker) conclude(result Result) Result {
func (c Checker) conclude(result types.Result) types.Result {
result.ThresholdRTT = c.ThresholdRTT

// Check errors (down)
Expand Down
10 changes: 5 additions & 5 deletions dnschecker_test.go → check/dns/dns_test.go
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
package checkup
package dns

import (
"net"
"testing"
"time"
)

func TestDNSChecker(t *testing.T) {
func TestChecker(t *testing.T) {
// Listen on localhost, random port
srv, err := net.Listen("tcp", "localhost:8382")
if err != nil {
Expand All @@ -28,7 +28,7 @@ func TestDNSChecker(t *testing.T) {
// Should know the host:port by now
endpt := srv.Addr().String()
testName := "TestDNS"
hc := DNSChecker{Name: testName, URL: endpt, Attempts: 2}
hc := Checker{Name: testName, URL: endpt, Attempts: 2}

// Try an up server
result, err := hc.Check()
Expand Down Expand Up @@ -102,7 +102,7 @@ func TestDNSChecker(t *testing.T) {
}
}

func TestDNSCheckerWithAgressiveTimeout(t *testing.T) {
func TestCheckerWithAgressiveTimeout(t *testing.T) {
// Listen on localhost, random port
srv, err := net.Listen("tcp", "localhost:0")
if err != nil {
Expand All @@ -124,7 +124,7 @@ func TestDNSCheckerWithAgressiveTimeout(t *testing.T) {
// Should know the host:port by now
endpt := srv.Addr().String()
testName := "TestTCP"
hc := DNSChecker{Name: testName, URL: endpt, Attempts: 2, Timeout: 1 * time.Nanosecond}
hc := Checker{Name: testName, URL: endpt, Attempts: 2, Timeout: 1 * time.Nanosecond}

result, err := hc.Check()
if err != nil {
Expand Down
Loading