After upgrade, getting missing table errors

[jceddy]

I deployed the latest keygen on 9/26 (previous deployment was in early May), and since then, I am seeing this error from the web container, and both the web and worker are continuously crashing and restarting:

/usr/local/bundle/ruby/3.3.0/gems/activerecord-7.1.3.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:19:in `exec': ERROR:  relation "license_users" does not exist (PG::UndefinedTable)
LINE 10:  WHERE a.attrelid = '"license_users"'::regclass

I've verified that the postgres database does NOT have the license_users table, and that it also didn't back in May (checked a schema dump I did at the time).

At the time of the deployment, I ran the "release" command, this was the output:

Running command: bundle exec rails db:migrate
I, [2024-09-26T14:18:39.242140 #1]  INFO -- : [RailsAutoscale] Preparing middleware

All of the containers are run with KEYGEN_MODE: singleplayer

I am stymied as to how to debug/fix this and was hoping that someone would be able to guide me at what to look at next.

[ezekg]

If you don't have the tables, then the new version's migrations didn't run against the correct database.

[jceddy]

Yes, I can see that the latest applied migration on the database is "20231013164654", the migration that adds the missing table is "20231106203122", and there are other later migrations that are missing.

All I'm missing now is why the release command is not applying the missing migrations.

The database connection is correct.

[ezekg]

Try running it with RAILS_LOG_LEVEL=debug. I'd also double check that its running on the correct database and isn't spinning up its own database for the release container.

[jceddy]

This is the output with RAILS_LOG_LEVEL=debug:

Running command: bundle exec rails db:migrate
I, [2024-10-04T20:31:02.125983 #1]  INFO -- : [RailsAutoscale] Preparing middleware
D, [2024-10-04T20:31:02.730838 #1] DEBUG -- :   \u001b[1m\u001b[35m (0.7ms)\u001b[0m  \u001b[1m\u001b[34mSELECT pg_try_advisory_lock(3687950948857684005)\u001b[0m
D, [2024-10-04T20:31:02.737691 #1] DEBUG -- :   \u001b[1m\u001b[36mActiveRecord::SchemaMigration Pluck (1.7ms)\u001b[0m  \u001b[1m\u001b[34mSELECT "schema_migrations"."version" FROM "schema_migrations" ORDER BY "schema_migrations"."version" ASC\u001b[0m
D, [2024-10-04T20:31:02.742945 #1] DEBUG -- :   \u001b[1m\u001b[36mActiveRecord::InternalMetadata Load (0.8ms)\u001b[0m  \u001b[1m\u001b[34mSELECT "ar_internal_metadata".* FROM "ar_internal_metadata" WHERE "ar_internal_metadata"."key" = 'environment' LIMIT 1\u001b[0m
D, [2024-10-04T20:31:02.751534 #1] DEBUG -- :   \u001b[1m\u001b[35m (0.5ms)\u001b[0m  \u001b[1m\u001b[34mSELECT pg_advisory_unlock(3687950948857684005)\u001b[0m

The last three values from the schema_migrations query should return (only including the last three records):

20230914172659
20230914184536
20231013164654

The ar_internal_metadata query should return (added some tabs to the headers for readability):

"key"		"value"		"created_at"			"updated_at"
"environment"	"production"	"2024-06-19 14:00:00.634235"	"2024-06-19 14:00:00.634235"

The podman command being run is:

podman run --name keygen_release --rm --detach=False --secret ******** --secret ******** --secret ******** --secret ******** --env DATABASE_URL=postgres://keygen:<redacted>@dbserver.dev.company.local:5432/keygen --env REDIS_URL=redis://:<redacted>@redis.release.environment.company.com:6379 --env KEYGEN_HOST=api.env.licensing.company.com --env KEYGEN_MODE=singleplayer --env KEYGEN_EDITION=CE --env KEYGEN_ACCOUNT_ID=<redacted> --env RAILS_LOG_LEVEL=debug docker.io/keygen/api release

Edit: I just double-checked to make sure that the release container is spun up from the same image as the web and worker containers. Just in case!

[ezekg]

Can you run the following inside the release container and share the output?

bundle exec rails runner 'puts version: Keygen.version, revision: Keygen.revision'

[jceddy]

When I ran the release command manually, it applied the migrations as expected.

There seems to be some issue with running the podman commands via Ansible that I need to look into.