Question: How to approach testing / mocks

[dbhoot]

I'm just starting to learn / work with river and I have a question about how to test. Here's my set up:

• pgx as core driver
• sqlc as access layer
• migrate as my migration tool
• currently using pgxmock as mock driver

On app start, I create the river client and pass it around to the modules which need to insert jobs. No concerns there, the app seems to work fine. However, I'm not sure how to (or if I can at all) get my tests to pass since I can't create a client from the mock pgx pool. I have been considering switching to dockertest for other reasons and that may solve the problem (but also may not) -- however, I'm hoping there's a way to reduce the initial scope.

Related question -- is it better to create the client and pass it as needed or create a client on the fly?
In my situation, I chose to do it on start up and pass it since the pool that's accessible deeper in the app is actually an interface since it can be a real pool or a mock pool. Since it's an interface, I likely can't create a client from the pool interface.

On the flip side, passing it forces you to pass it like riverClient *river.Client[pgx.Tx] -- so which ever mocking strategy you use has to yield the same instantiated type.

I tried peeking at how river itself is running its tests and I saw the internal/rivertest package. However, I'm not able to use that to create pools/clients. use of internal package github.com/riverqueue/river/internal/riverinternaltest not allowed

[brandur]

Okay, first of all, I'd very, very, VERY much recommend not mocking your database. It might seem like a good idea at first glance because it's marginally faster, but there is just far too many details on the interactions between your app and its data store for this to be an effective testing strategy. Not only will mocking everything out be painful, but you'll find before too long that you might as well not be writing tests at all because they're not exercising anywhere near enough of your stack to be be effective. You're basically setting a whole bunch of mocks, and then testing that you set your mocks in the way you expected to set them. Nothing is really being tested.

Just as a very basic example, a database mock will be perfectly happy to select values out of a column that doesn't exist. There are 10,000 more examples of that, all the way from a simple missing column/table, all the way up to more complex things like failing FKs, unique constraints, or checks.

Instead, just save to a real database. It might seem too slow, but it's not. Databases are fast and you can keep your tests very fast too. Use a technique like test transactions to isolate test cases from each other, but still have them running heavily in parallel.

If I can't convince you otherwise, what you can do is create your own mock interface for River's inserts:

type RiverInserter[TTx any] interface {
	Insert(ctx context.Context, args JobArgs, opts *InsertOpts) (*rivertype.JobInsertResult, error)
	InsertTx(ctx context.Context, tx TTx, args JobArgs, opts *InsertOpts) (*rivertype.JobInsertResult, error)
}

Pass a RiverInserter[pgx.Tx] into your submodules. pgx.Tx is an interface, and pgxmock will return a compliant mock object from its Begin that will work with River.

Related question -- is it better to create the client and pass it as needed or create a client on the fly?

It's pretty much always better to instantiate fewer times if you can manage it. River client is a fairly heavy object so try to only make one of them.

I tried peeking at how river itself is running its tests and I saw the internal/rivertest package. However, I'm not able to use that to create pools/clients. use of internal package github.com/riverqueue/river/internal/riverinternaltest not allowed

Yeah that's an internal package that sees regular API changes and isn't stable. There's a public river/rivertest package, although it contains mostly asseritons and won't scaffold a test database for you — you'll have to do that for yourself.

[dbhoot]

Thank you for the guidance!

Okay, first of all, I'd very, very, VERY much recommend not mocking your database. It might seem like a good idea at first glance because it's marginally faster, but there is just far too many details on the interactions between your app and its data store for this to be an effective testing strategy.

Yes, I stopped writing tests with the mock pretty quickly as it was tedious, fairly brittle, and I started to question how effective the strategy was. I recognized I needed an alternate strategy but wasn't sure what that looked like. You've sold me on using a real db.

My follow up questions are:

• Do you know of any public projects which test against a real db which I could use to model my own testing approach?
• How would local environments works vs ci? Should everyone who works on the project create a separate db in their existing instance to run the test suite?
• Any opinions on dockertest? I think it would work here and I like it because it keeps the test suite self contained. The down side though I have to create the instance/pool once, programmatically apply the migrations then run the suite. All do-able and manageable but a bunch of little decisions to make to get things working right.

Random aside -- I'm going to take some time and go through your writings. Seems like a lot of great nuggets of wisedom there that I don't quite grok yet. Thank you for making your knowledge and expertise available to the community. 🙏

[brandur]

Do you know of any public projects which test against a real db which I could use to model my own testing approach?

So I was I had a simpler one, but the best I can offer you right now is River's riverinternaltest.

River's internal test suite has two test strategies:

1. A "test DB manager" which spins up a pool of River databases and then has a manager that selects between them, truncates all tables, and then hands off a database to a particular test that needs one.

2. Use of a single shared test database along with TestTx. Tests run in a test transaction so changes are automatically rolled back after each one and so no state needs cleaning up.

I'd recommend avoiding (1): River makes use of it in some cases because it wants to test a database-wise listen/notify, but it comes with all kinds of problems: slower, brittle (if you add a table but then forget to add a truncate for it, the test strategy breaks), and it breaks Go's built-in test parallelism because test packages being run in parallel have no way to coordinate use of test databases with each other.

If you just copied River's implementation of (2) though, that'd take you a long way. See here:

river/internal/riverinternaltest/riverinternaltest.go

Lines 257 to 261 in 755295f

	// TestTx starts a test transaction that's rolled back automatically as the test
	// case is cleaning itself up. This can be used as a lighter weight alternative
	// to `testdb.Manager` in components where it's not necessary to have many
	// connections open simultaneously.
	func TestTx(ctx context.Context, tb testing.TB) pgx.Tx {

How would local environments works vs ci? Should everyone who works on the project create a separate db in their existing instance to run the test suite?

Yep, everyone just has a local database that they create and keep migrated. CI uses GitHub's built-in Postgres action and creates the database there. Use a common name for all the DBs like river_test_db.

Any opinions on dockertest? I think it would work here and I like it because it keeps the test suite self contained. The down side though I have to create the instance/pool once, programmatically apply the migrations then run the suite. All do-able and manageable but a bunch of little decisions to make to get things working right.

You'll get mixed opinions on this, but personally I avoided Docker like the plague. It adds heavy layers to everything and the abstraction advantage it buys you is oversold IMO. Its value add varies by language -- in some languages configuration and setting up a project is so absolutely arduous that it makes to try and abstract that away into something like a Dockerfile, but in Go, there's ~zero set up steps because so much is built-into the toolchain. Your test run instructions are:

1. createdb river_test_db
2. [Run migrations]
3. go test ./...

Random aside -- I'm going to take some time and go through your writings. Seems like a lot of great nuggets

Please do and thank you :)

[pmenglund]

@dbhoot I've got a dockertest setup that I use for one of projects (but it is in a private repo so I can't show you), and it is "decently fast". On my laptop I get the following:

1. setup takes about 1.3s, which creates a new Postgres docker container, runs the migration
2. a sample test runs in about 0.03s, which creates a worker, inserts a job and waits for it to execute
3. tearing down takes about 0.1s, which deletes the docker container

If there is interest, I can create a PR with my sample setup.

My take on dockertest is that it is useful when you have "random people" contributing, and you want to make it easy for them to run local testing with "zero setup" before they submit a PR, but as @brandur pointed out, Docker can add a lot of overhead and headache, so YMMV.

[dbhoot]

Thanks for the info. How long did it take you to set up and configure the way you wanted? Do you also run it in a CI?

[pmenglund]

Since I've been using dockertest for years, and I have other projects where I use it - I could reuse an old Postgres setup, so it took me about 15 minutes to get up and running. I use this setup in CI too, as it guarantees a clean start for all PRs, and we have other tests that run that take much longer than this, so it isn't a limiting factor.

[dbhoot]

@brandur I finally got around to implementing the testing strategy using your suggestions as a template. It's glorious! I'm so pleased with the result / outcome. Thanks again for the guidance. 🙏

[Debetux]

@dbhoot I have been on the same quest as you! Can I ask you what main resource helped you in the end?

It may not be of any help anymore, but I stumbled upon a package called pgtestdb that looks interesting for this problem.