Bnewbold/readme rainbow

HACKING.md

@@ -14,6 +14,7 @@ Run with, eg, `go run ./cmd/bigsky`):
 - `cmd/supercollider`: event stream load generation tool
 - `cmd/sonar`: event stream monitoring tool
 - `cmd/hepa`: auto-moderation rule engine service
+- `cmd/rainbow`: firehose fanout service
 - `gen`: dev tool to run CBOR type codegen
 
 Packages:

cmd/rainbow/README.md

@@ -0,0 +1,32 @@
+
+`rainbow`: atproto Firehose Fanout Service
+==========================================
+
+This is an atproto service which consumes from a firehose (eg, from a relay or PDS) and fans out events to many subscribers.
+
+Features and design points:
+
+- retains "backfill window" on local disk (using [pebble](https://github.com/cockroachdb/pebble))
+- serves the `com.atproto.sync.subscribeRepos` endpoint (WebSocket)
+- retains upstream firehose "sequence numbers"
+- does not validate events (signatures, repo tree, hashes, etc), just passes through
+- does not archive or mirror individual records or entire repositories (or implement related API endpoints)
+- disk I/O intensive: fast NVMe disks are recommended, and RAM is helpful for caching
+- single golang binary for easy deployment
+- observability: logging, prometheus metrics, OTEL traces
+
+## Running 
+
+This is a simple, single-binary Go program. You can also build and run it as a docker container (see `./Dockerfile`).
+
+From the top level of this repo, you can build:
+
+```shell
+go build ./cmd/rainbow -o rainbow-bin
+```
+
+or just run it, and see configuration options:
+
+```shell
+go run ./cmd/rainbow --help
+```