Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get Identity DB ready for prod #394

Merged
merged 15 commits into from
May 31, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions .mockery.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
with-expecter: true
packages:
github.com/xmtp/xmtp-node-go/pkg/proto/mls/api/v1:
config:
dir: ./pkg/mocks
outpkg: mocks
interfaces:
MlsApi_SubscribeGroupMessagesServer:
config:
MlsApi_SubscribeWelcomeMessagesServer:
config:
github.com/xmtp/xmtp-node-go/pkg/mlsvalidate:
config:
dir: ./pkg/mocks
outpkg: mocks
interfaces:
MLSValidationService:
config:
4 changes: 2 additions & 2 deletions dev/generate
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ set -e

go generate ./...

mockgen -package api github.com/xmtp/xmtp-node-go/pkg/proto/mls/api/v1 MlsApi_SubscribeGroupMessagesServer,MlsApi_SubscribeWelcomeMessagesServer > pkg/mls/api/v1/mock.gen.go
mockgen -package mocks -source ./pkg/mlsvalidate/service.go MLSValidationService > pkg/mlsvalidate/mocks/mock.gen.go
# Generate mocks
mockery
Copy link
Collaborator Author

@neekolas neekolas May 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Steven was previously using mockgen to generate mocks. I decided to switch us back to the more standard testify/mock and mockery. It's more widely used and more actively maintained. Otherwise very similar, other than some syntax changes.

This probably should have been a separate PR, but a lot of the things that were being mocked were also changing.

rm -rf pkg/proto/**/*.pb.go pkg/proto/**/*.pb.gw.go pkg/proto/**/*.swagger.json
if ! buf generate https://github.com/xmtp/proto.git#branch=main,subdir=proto; then
echo "Failed to generate protobuf definitions"
Expand Down
2 changes: 1 addition & 1 deletion dev/up
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ if ! which golangci-lint &>/dev/null; then brew install golangci-lint; fi
if ! which shellcheck &>/dev/null; then brew install shellcheck; fi
if ! which protoc &>/dev/null; then brew install protobuf; fi
if ! which protoc-gen-go &>/dev/null; then go install google.golang.org/protobuf/cmd/protoc-gen-go@latest; fi
if ! which mockgen &>/dev/null || [ `mockgen --version` != "v0.4.0" ]; then go install go.uber.org/mock/[email protected]; fi
if ! which mockery &>/dev/null; then brew install mockery; fi
if ! which protolint &>/dev/null; then go install github.com/yoheimuta/protolint/cmd/protolint@latest; fi

dev/generate
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@ require (
github.com/waku-org/go-waku v0.8.0
github.com/xmtp/go-msgio v0.2.1-0.20220510223757-25a701b79cd3
github.com/yoheimuta/protolint v0.39.0
go.uber.org/mock v0.4.0
go.uber.org/zap v1.24.0
golang.org/x/sync v0.3.0
google.golang.org/genproto v0.0.0-20230223222841-637eb2293923
Expand All @@ -42,6 +41,7 @@ require (
)

require (
github.com/btcsuite/btcd/chaincfg/chainhash v1.1.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
github.com/jackc/puddle/v2 v2.2.1 // indirect
Expand Down
5 changes: 2 additions & 3 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,8 @@ github.com/bradfitz/gomemcache v0.0.0-20220106215444-fb4bf637b56d/go.mod h1:H0wQ
github.com/btcsuite/btcd v0.20.1-beta/go.mod h1:wVuoA8VJLEcwgqHBwHmzLRazpKxTv13Px/pDuV7OomQ=
github.com/btcsuite/btcd/btcec/v2 v2.2.1 h1:xP60mv8fvp+0khmrN0zTdPC3cNm24rfeE6lh2R/Yv3E=
github.com/btcsuite/btcd/btcec/v2 v2.2.1/go.mod h1:9/CSmJxmuvqzX9Wh2fXMWToLOHhPd11lSPuIupwTkI8=
github.com/btcsuite/btcd/chaincfg/chainhash v1.0.1 h1:q0rUy8C/TYNBQS1+CGKw68tLOFYSNEs0TFnxxnS9+4U=
github.com/btcsuite/btcd/chaincfg/chainhash v1.1.0 h1:59Kx4K6lzOW5w6nFlA0v5+lk/6sjybR934QNHSJZPTQ=
github.com/btcsuite/btcd/chaincfg/chainhash v1.1.0/go.mod h1:7SFka0XMvUgj3hfZtydOrQY2mwhPclbT2snogU7SQQc=
github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f/go.mod h1:TdznJufoqS23FtqVCzL0ZqgP5MqXbb4fg/WgDys70nA=
github.com/btcsuite/btcutil v0.0.0-20190425235716-9e5f4b9a998d/go.mod h1:+5NJ2+qvTyV9exUAL/rxXi3DcLg2Ts+ymUAY5y4NvMg=
github.com/btcsuite/go-socks v0.0.0-20170105172521-4720035b7bfd/go.mod h1:HHNXQzUsZCxOoE+CPiyCTO6x34Zs86zZUiwtpXoGdtg=
Expand Down Expand Up @@ -1191,8 +1192,6 @@ go.uber.org/fx v1.20.0 h1:ZMC/pnRvhsthOZh9MZjMq5U8Or3mA9zBSPaLnzs3ihQ=
go.uber.org/fx v1.20.0/go.mod h1:qCUj0btiR3/JnanEr1TYEePfSw6o/4qYJscgvzQ5Ub0=
go.uber.org/goleak v1.1.11-0.20210813005559-691160354723/go.mod h1:cwTWslyiVhfpKIDGSZEM2HlOvcqm+tG4zioyIeLoqMQ=
go.uber.org/goleak v1.1.12 h1:gZAh5/EyT/HQwlpkCy6wTpqfH9H8Lz8zbm3dZh+OyzA=
go.uber.org/mock v0.4.0 h1:VcM4ZOtdbR4f6VXfiOpwpVJDL6lCReaZ6mw31wqh7KU=
go.uber.org/mock v0.4.0/go.mod h1:a6FSlNadKUHUa9IP5Vyt1zh4fC7uAwxMutEAscFbkZc=
go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
go.uber.org/multierr v1.3.0/go.mod h1:VgVr7evmIr6uPjLBxg28wmKNXyqE9akIJ5XnfpiKl+4=
go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU=
Expand Down
14 changes: 7 additions & 7 deletions pkg/identity/api/v1/identity_service_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ func TestPublishedUpdatesCanBeRead(t *testing.T) {
svc, _, cleanup := newTestService(t, ctx)
defer cleanup()

inbox_id := "test_inbox"
inbox_id := test.RandomInboxId()
address := "test_address"

_, err := svc.PublishIdentityUpdate(ctx, publishIdentityUpdateRequest(inbox_id, makeCreateInbox(address)))
Expand All @@ -173,7 +173,7 @@ func TestPublishedUpdatesAreInOrder(t *testing.T) {
svc, _, cleanup := newTestService(t, ctx)
defer cleanup()

inbox_id := "test_inbox"
inbox_id := test.RandomInboxId()
address := "test_address"

_, err := svc.PublishIdentityUpdate(ctx, publishIdentityUpdateRequest(inbox_id, makeCreateInbox(address)))
Expand Down Expand Up @@ -210,10 +210,10 @@ func TestQueryMultipleInboxes(t *testing.T) {
svc, _, cleanup := newTestService(t, ctx)
defer cleanup()

first_inbox_id := "test_inbox"
second_inbox_id := "second_inbox"
first_address := "test_address"
second_address := "test_address"
first_inbox_id := test.RandomInboxId()
second_inbox_id := test.RandomInboxId()
first_address := test.RandomInboxId()
second_address := test.RandomInboxId()

_, err := svc.PublishIdentityUpdate(ctx, publishIdentityUpdateRequest(first_inbox_id, makeCreateInbox(first_address)))
require.NoError(t, err)
Expand All @@ -233,7 +233,7 @@ func TestInboxSizeLimit(t *testing.T) {
svc, _, cleanup := newTestService(t, ctx)
defer cleanup()

inbox_id := "test_inbox"
inbox_id := test.RandomInboxId()
address := "test_address"

_, err := svc.PublishIdentityUpdate(ctx, publishIdentityUpdateRequest(inbox_id, makeCreateInbox(address)))
Expand Down
5 changes: 0 additions & 5 deletions pkg/migrations/mls/20231023050806_init-schema.down.sql

This file was deleted.

23 changes: 0 additions & 23 deletions pkg/migrations/mls/20231023050806_init-schema.up.sql

This file was deleted.

32 changes: 0 additions & 32 deletions pkg/migrations/mls/20240109001927_add-messages.up.sql

This file was deleted.

5 changes: 0 additions & 5 deletions pkg/migrations/mls/20240122230601_add-hpke-key.down.sql

This file was deleted.

5 changes: 0 additions & 5 deletions pkg/migrations/mls/20240122230601_add-hpke-key.up.sql

This file was deleted.

8 changes: 0 additions & 8 deletions pkg/migrations/mls/20240411200242_init-identity.down.sql

This file was deleted.

24 changes: 0 additions & 24 deletions pkg/migrations/mls/20240411200242_init-identity.up.sql

This file was deleted.

8 changes: 0 additions & 8 deletions pkg/migrations/mls/20240425021053_add-inbox-filters.up.sql

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
SET statement_timeout = 0;

--bun:split
DROP TABLE IF EXISTS messages;

SELECT 1

--bun:split

SELECT 2
8 changes: 8 additions & 0 deletions pkg/migrations/mls/20240528181822_wipe-db.up.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
SET statement_timeout = 0;

--bun:split
DROP TABLE IF EXISTS installations, group_messages, welcome_messages, inbox_log, address_log;

--bun:split
DROP TYPE IF EXISTS inbox_filter;

Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
SET statement_timeout = 0;

--bun:split
DROP TYPE IF EXISTS inbox_filter;

SELECT 1

--bun:split

SELECT 2
71 changes: 71 additions & 0 deletions pkg/migrations/mls/20240528181851_init-schema.up.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
SET statement_timeout = 0;
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I consolidated all our MLS schema into one big migration after the drops. Just easier to read this way


--bun:split
CREATE TABLE installations(
id BYTEA PRIMARY KEY,
created_at BIGINT NOT NULL,
updated_at BIGINT NOT NULL,
inbox_id BYTEA NOT NULL,
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We now store inbox_id on installations instead of account_address

key_package BYTEA NOT NULL,
expiration BIGINT NOT NULL
);

--bun:split
CREATE TABLE group_messages(
id BIGSERIAL PRIMARY KEY,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
group_id BYTEA NOT NULL,
data BYTEA NOT NULL,
group_id_data_hash BYTEA NOT NULL
);

--bun:split
CREATE INDEX idx_group_messages_group_id_id ON group_messages(group_id, id);
Copy link
Collaborator Author

@neekolas neekolas May 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This index is different from before. We were previously indexing on (group_id, created_at) and it never got updated when we changed the sort/cursor on queries to be the id field.


--bun:split
CREATE UNIQUE INDEX idx_group_messages_group_id_data_hash ON group_messages(group_id_data_hash);

--bun:split
CREATE TABLE welcome_messages(
id BIGSERIAL PRIMARY KEY,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
installation_key BYTEA NOT NULL,
data BYTEA NOT NULL,
hpke_public_key BYTEA NOT NULL,
installation_key_data_hash BYTEA NOT NULL
);

--bun:split
CREATE INDEX idx_welcome_messages_installation_key_id ON welcome_messages(installation_key, id);
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also changed to match the sort/cursor field


--bun:split
CREATE UNIQUE INDEX idx_welcome_messages_group_key_data_hash ON welcome_messages(installation_key_data_hash);

--bun:split
CREATE TABLE inbox_log(
sequence_id BIGSERIAL PRIMARY KEY,
inbox_id BYTEA NOT NULL,
neekolas marked this conversation as resolved.
Show resolved Hide resolved
server_timestamp_ns BIGINT NOT NULL,
identity_update_proto BYTEA NOT NULL
);

--bun:split
CREATE INDEX idx_inbox_log_inbox_id_sequence_id ON inbox_log(inbox_id, sequence_id);

--bun:split
CREATE TABLE address_log(
address TEXT NOT NULL,
inbox_id BYTEA NOT NULL,
association_sequence_id BIGINT,
revocation_sequence_id BIGINT
);

--bun:split
CREATE INDEX idx_address_log_address_inbox_id ON address_log(address, inbox_id);

--bun:split
CREATE TYPE inbox_filter AS (
inbox_id TEXT, -- Because this is serialized as JSON, we can't use a BYTEA type
sequence_id BIGINT
);

Loading
Loading