graft

An implementation of the Raft protocol in Go using a REST API for intra-node communication.

Usage

Cluster Setup

A cluster may be created using the graft command.

$ graft node run --id 1 --port 9001 --peers '[{"id":2,"address":"http://127.0.0.1:9002"},{"id":3,"address":"http://127.0.0.1:9003"}]'
$ graft node run --id 2 --port 9002 --peers '[{"id":1,"address":"http://127.0.0.1:9001"},{"id":3,"address":"http://127.0.0.1:9003"}]'
$ graft node run --id 3 --port 9003 --peers '[{"id":3,"address":"http://127.0.0.1:9003"},{"id":2,"address":"http://127.0.0.1:9002"}]'

Note that the command may be run once, without a peer list to create a single node cluster.

Key/value Operations

Key/value operations may be dispatched to any node in the cluster; requests will be redirected to the leader node automatically.

$ graft kv set http://127.0.0.1:9001 key '{"a":"b","c":{"d":"e"}}'
$ graft kv get http://127.0.0.1:9001 key
{"a":"b","c":{"d":"e"}}
$ graft kv del http://127.0.0.1:9001 key

Logging

By default graft logs at the info level. The log level may be configured using the GOAMT_LOG_LEVEL environment variable. Valid options are debug, info, warn, error and fatal.

Building

graft uses Go modules so may be imported and built using go; a makefile is provided for local building.

$ make build # Build all binaries contained in the ./cmd subdirectory

$ ./build/bin/graft
Run/manage 'graft' clusters/nodes

Usage:
   [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command
  node        Run/manage 'graft' cluster nodes

Flags:
  -h, --help   help for this command

Use " [command] --help" for more information about a command.

$ make clean # Remove an generated files (build/coverage.out)

Development

A convenience makefile is provided for common development tasks.

$ make coverage # Run unit testing an output a coverage report to 'coverage.out'
$ make generate # Generate mocks
$ make lint     # Run linting
$ make test     # Run unit testing

To Do

• Implement leadership election
• Implement log replication
• Add API to set/get/delete key/value data
• Add a binary/command line user interface
• Implement persistent storage
• Implement log compaction/snapshotting
• Implement node addition/removal
• Disallow protocol requests which originate from outside the cluster
• Add an RPC/gRPC backend
• Optimise bulk log replication

FAQ

Why REST and not RPC/gRPC?

This was an advanced Go task I set up while at Couchbase for a group new graduate software engineers to get to grips with Go. I am aware that generally some form of RPC library is used for intra-cluster communication however, given the graduates would be working on a backend REST API, I valued experience with REST over RPC/gRPC. The method of intra-cluster communication in graft is decoupled meaning additional backends can be added and switched out in the future.

License

Copyright 2022 James Lee [email protected]

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.