Skip to content

Commit

Permalink
Improve codelab and record asciinema demo (#266)
Browse files Browse the repository at this point in the history
Also update test keys in mysql to have key names that better represent how they should look (with schemeless URLs).
  • Loading branch information
mhutchinson authored Sep 25, 2024
1 parent be6d7b2 commit a8c4779
Show file tree
Hide file tree
Showing 11 changed files with 810 additions and 18 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
- name: Start Tessera
run: docker compose -f ./cmd/conformance/mysql/docker/compose.yaml up --build --detach
- name: Run benchmark
run: go run ./hammer --log_public_key=Test-Betty+df84580a+AQQASqPUZoIHcJAF5mBOryctwFdTV1E0GRY4kEAtTzwB --log_url=http://localhost:2024 --max_read_ops=0 --num_writers=512 --max_write_ops=512 --max_runtime=1m --leaf_write_goal=2500 --show_ui=false
run: go run ./hammer --log_public_key=transparency.dev/tessera/example+ae330e15+ASf4/L1zE859VqlfQgGzKy34l91Gl8W6wfwp+vKP62DW --log_url=http://localhost:2024 --max_read_ops=0 --num_writers=512 --max_write_ops=512 --max_runtime=1m --leaf_write_goal=2500 --show_ui=false
- name: Stop Tessera
if: ${{ always() }}
run: docker compose -f ./cmd/conformance/mysql/docker/compose.yaml down
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/integration_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
- name: Start Docker services (tessera-conformance-mysql-db and tessera-conformance-mysql)
run: docker compose -f ./cmd/conformance/mysql/docker/compose.yaml up --build --detach
- name: Run integration test
run: go test -v -race ./integration/... --run_integration_test=true --log_url="http://localhost:2024" --write_log_url="http://localhost:2024" --log_public_key="Test-Betty+df84580a+AQQASqPUZoIHcJAF5mBOryctwFdTV1E0GRY4kEAtTzwB"
run: go test -v -race ./integration/... --run_integration_test=true --log_url="http://localhost:2024" --write_log_url="http://localhost:2024" --log_public_key="transparency.dev/tessera/example+ae330e15+ASf4/L1zE859VqlfQgGzKy34l91Gl8W6wfwp+vKP62DW"
- name: Stop Docker services (tessera-conformance-mysql-db and tessera-conformance-mysql)
if: ${{ always() }}
run: docker compose -f ./cmd/conformance/mysql/docker/compose.yaml down
Expand Down
18 changes: 12 additions & 6 deletions cmd/conformance/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,14 +25,16 @@ The commands below add entries to the log, and then show a few approaches to ins

```shell
# Add 3 entries in parallel, and wait for all requests to complete
curl -d 'foo' -H "Content-Type: application/data" -X POST ${WRITE_URL}/add &
curl -d 'bar' -H "Content-Type: application/data" -X POST ${WRITE_URL}/add &
curl -d 'baz' -H "Content-Type: application/data" -X POST ${WRITE_URL}/add &
curl -d 'one!' -H "Content-Type: application/data" -X POST ${WRITE_URL}add &
curl -d 'two!' -H "Content-Type: application/data" -X POST ${WRITE_URL}add &
curl -d 'three!' -H "Content-Type: application/data" -X POST ${WRITE_URL}add &
wait

# Check that the checkpoint is of the correct size and the leaves are present
curl ${READ_URL}/checkpoint
curl -output - ${READ_URL}/tile/entries/000.p/3
# Check that the checkpoint is of the correct size
curl -s ${READ_URL}checkpoint

# Look at the leaves. Piping into xxd to reveal the leaf sizes.
curl -s ${READ_URL}tile/entries/000.p/3 | xxd
```

The tiles format is plain-text, but it's better to inspect the log via tooling made for this purpose:
Expand All @@ -47,3 +49,7 @@ go run github.com/mhutchinson/woodpecker@main \
Use arrow keys left and right to go backwards and forwards through the entries in the log.
Use `q` to quit.

Here's a demo of the codelab being followed:

![Codelab demo](./demo.gif)

786 changes: 786 additions & 0 deletions cmd/conformance/demo.cast

Large diffs are not rendered by default.

Binary file added cmd/conformance/demo.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions cmd/conformance/mysql/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ Note that the writes are sent to the HTTP server we brought up in the previous s

```shell
go run ./hammer \
--log_public_key=Test-Betty+df84580a+AQQASqPUZoIHcJAF5mBOryctwFdTV1E0GRY4kEAtTzwB \
--log_public_key=transparency.dev/tessera/example+ae330e15+ASf4/L1zE859VqlfQgGzKy34l91Gl8W6wfwp+vKP62DW \
--log_url=http://localhost:2024/ \
--max_read_ops=0 \
--num_writers=256 \
Expand All @@ -69,5 +69,5 @@ go run ./hammer \
Optionally, inspect the log using the woodpecker tool to see the contents:

```shell
go run github.com/mhutchinson/woodpecker@main --custom_log_type=tiles --custom_log_url=http://localhost:2024/ --custom_log_origin=Test-Betty --custom_log_vkey=Test-Betty+df84580a+AQQASqPUZoIHcJAF5mBOryctwFdTV1E0GRY4kEAtTzwB
go run github.com/mhutchinson/woodpecker@main --custom_log_type=tiles --custom_log_url=http://localhost:2024/ --custom_log_vkey=transparency.dev/tessera/example+ae330e15+ASf4/L1zE859VqlfQgGzKy34l91Gl8W6wfwp+vKP62DW
```
2 changes: 1 addition & 1 deletion cmd/conformance/mysql/docker/testdata/key
Original file line number Diff line number Diff line change
@@ -1 +1 @@
PRIVATE+KEY+Test-Betty+df84580a+Afge8kCzBXU7jb3cV2Q363oNXCufJ6u9mjOY1BGRY9E2
PRIVATE+KEY+transparency.dev/tessera/example+ae330e15+AXEwZQ2L6Ga3NX70ITObzyfEIketMr2o9Kc+ed/rt/QR
2 changes: 1 addition & 1 deletion cmd/conformance/mysql/docker/testdata/key.pub
Original file line number Diff line number Diff line change
@@ -1 +1 @@
Test-Betty+df84580a+AQQASqPUZoIHcJAF5mBOryctwFdTV1E0GRY4kEAtTzwB
transparency.dev/tessera/example+ae330e15+ASf4/L1zE859VqlfQgGzKy34l91Gl8W6wfwp+vKP62DW
4 changes: 2 additions & 2 deletions docs/performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ MiB Swap: 0.0 total, 0.0 free, 0.0 used. 37.3 avail Mem
1. Run `hammer` and get performance metrics

```sh
hammer:~/trillian-tessera$ go run ./hammer --log_public_key=Test-Betty+df84580a+AQQASqPUZoIHcJAF5mBOryctwFdTV1E0GRY4kEAtTzwB --log_url=http://10.128.0.3:2024 --max_read_ops=0 --num_writers=512 --max_write_ops=512
hammer:~/trillian-tessera$ go run ./hammer --log_public_key=transparency.dev/tessera/example+ae330e15+ASf4/L1zE859VqlfQgGzKy34l91Gl8W6wfwp+vKP62DW --log_url=http://10.128.0.3:2024 --max_read_ops=0 --num_writers=512 --max_write_ops=512
```

### GCP Free Tier VM Instance + Cloud SQL (MySQL)
Expand Down Expand Up @@ -168,7 +168,7 @@ The bottleneck comes from CPU usage of the `cmd/conformance/mysql` binary on the
1. Run `hammer` and get performance metrics

```sh
hammer:~/trillian-tessera$ go run ./hammer --log_public_key=Test-Betty+df84580a+AQQASqPUZoIHcJAF5mBOryctwFdTV1E0GRY4kEAtTzwB --log_url=http://10.128.0.3:2024 --max_read_ops=0 --num_writers=512 --max_write_ops=512
hammer:~/trillian-tessera$ go run ./hammer --log_public_key=transparency.dev/tessera/example+ae330e15+ASf4/L1zE859VqlfQgGzKy34l91Gl8W6wfwp+vKP62DW --log_url=http://10.128.0.3:2024 --max_read_ops=0 --num_writers=512 --max_write_ops=512
```

## POSIX
Expand Down
4 changes: 2 additions & 2 deletions hammer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Example usage to test a deployment of `cmd/conformance/mysql`:

```shell
go run ./hammer \
--log_public_key=Test-Betty+df84580a+AQQASqPUZoIHcJAF5mBOryctwFdTV1E0GRY4kEAtTzwB \
--log_public_key=transparency.dev/tessera/example+ae330e15+ASf4/L1zE859VqlfQgGzKy34l91Gl8W6wfwp+vKP62DW \
--log_url=http://localhost:2024 \
--max_read_ops=1024 \
--num_readers_random=128 \
Expand All @@ -35,7 +35,7 @@ If the timeout of 1 minute is reached first, then it exits with an exit code of

```shell
go run ./hammer \
--log_public_key=Test-Betty+df84580a+AQQASqPUZoIHcJAF5mBOryctwFdTV1E0GRY4kEAtTzwB \
--log_public_key=transparency.dev/tessera/example+ae330e15+ASf4/L1zE859VqlfQgGzKy34l91Gl8W6wfwp+vKP62DW \
--log_url=http://localhost:2024 \
--max_read_ops=0 \
--num_writers=512 \
Expand Down
4 changes: 2 additions & 2 deletions storage/mysql/mysql_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,8 @@ var (
)

const (
testPrivateKey = "PRIVATE+KEY+Test-Betty+df84580a+Afge8kCzBXU7jb3cV2Q363oNXCufJ6u9mjOY1BGRY9E2"
testPublicKey = "Test-Betty+df84580a+AQQASqPUZoIHcJAF5mBOryctwFdTV1E0GRY4kEAtTzwB"
testPrivateKey = "PRIVATE+KEY+transparency.dev/tessera/example+ae330e15+AXEwZQ2L6Ga3NX70ITObzyfEIketMr2o9Kc+ed/rt/QR"
testPublicKey = "transparency.dev/tessera/example+ae330e15+ASf4/L1zE859VqlfQgGzKy34l91Gl8W6wfwp+vKP62DW"
)

// TestMain checks whether the test MySQL database is available and starts the tests including database schema initialization.
Expand Down

0 comments on commit a8c4779

Please sign in to comment.