[cmds][scd/store] Add 'db-manager scd-evict' subcommand enabling listing and deletion of expired SCD entities

[mickmis]

Stacked on top of #1114, please review only b8b2ff9
Addresses #1074

This PR adds a CLI tool db-evictor that enables deletion of expired entities. See enclosed README for more details on its usage.

Do note the approach taken when entities do not have end times: the last update times are used.

Testing

I've performed basic testing of this tool locally. I manually inserted in the database valid and expired for about a month subscriptions and operational intents. Then I validated that it identifies and deletes correctly entities by running it manually.

Testing data

scd_operations table

id	owner	version	url	altitude_lower	altitude_upper	starts_at	ends_at	subscription_id	updated_at	state	cells	uss_requested_ovn	past_ovns
00000185-e36d-40be-8d38-beca6ca30000	httpclient	1	https://dummy.uss	84	169	2024-09-14 15:48:36.190936 +00:00	2024-09-14 16:33:36.190936 +00:00	9f0d4575-b275-4a4c-a261-e1e04d324565	2024-09-13 12:36:29.881840 +00:00	Accepted	{-8768904281496485888,-8768904178417270784}		{}
00000185-e36d-40be-8d38-beca6ca30001	httpclient	1	https://dummy.uss	84	169	2024-08-15 15:48:36.190000 +00:00	2024-08-15 16:33:36.190000 +00:00	78f98cc5-94f3-4c04-8da9-a8398feba3f3	2024-08-13 12:38:30.041000 +00:00	Accepted	{-8768904281496485888,-8768904178417270784}		{}
00000185-e36d-40be-8d38-beca6ca30003	httpclient	1	https://dummy.uss	84	169			189ec22f-5e61-418a-940b-36de2d201fd5	2024-08-13 12:39:04.958561 +00:00	Accepted	{-8768904281496485888,-8768904178417270784}		{}

scd_subscriptions table

id	owner	version	url	notification_index	notify_for_operations	notify_for_constraints	implicit	starts_at	ends_at	updated_at	cells
189ec22f-5e61-418a-940b-36de2d201fd5	httpclient	0	https://dummy.uss	1	true	false	true	2024-09-16 15:48:36.190936 +00:00	2024-09-16 16:33:36.190936 +00:00	2024-09-13 12:39:04.958561 +00:00	{-8768904281496485888,-8768904178417270784}
78f98cc5-94f3-4c04-8da9-a8398feba3f3	httpclient	0	https://dummy.uss	1	true	false	true	2024-08-15 15:48:36.190000 +00:00	2024-08-15 16:33:36.190000 +00:00	2024-08-13 12:38:30.041000 +00:00	{-8768904281496485888,-8768904178417270784}
9f0d4575-b275-4a4c-a261-e1e04d324565	httpclient	0	https://dummy.uss	1	true	false	true			2024-08-13 12:36:29.881000 +00:00	{-8768904281496485888,-8768904178417270784}

Result of running the tool

$ docker compose -f docker-compose_dss.yaml -p dss_sandbox exec local-dss-core-service db-evictor -cockroach_host=local-dss-crdb 
2024/09/13 13:09:26 no entity older than 2024-05-24 13:09:26.563521603 +0000 UTC m=-9676799.999555160 found

$ docker compose -f docker-compose_dss.yaml -p dss_sandbox exec local-dss-core-service db-evictor -cockroach_host=local-dss-crdb -ttl=72h
2024/09/13 13:12:25 found operational intent 00000185-e36d-40be-8d38-beca6ca30001; expired due to end time before 2024-09-10 13:12:25.076411047 +0000 UTC m=-259199.999435947
2024/09/13 13:12:25 found operational intent 00000185-e36d-40be-8d38-beca6ca30003; expired due to last update before 2024-09-10 13:12:25.076411047 +0000 UTC m=-259199.999435947 (missing end time)
2024/09/13 13:12:25 found subscription 78f98cc5-94f3-4c04-8da9-a8398feba3f3; expired due to end time before 2024-09-10 13:12:25.076411047 +0000 UTC m=-259199.999435947
2024/09/13 13:12:25 found subscription 9f0d4575-b275-4a4c-a261-e1e04d324565; expired due to last update before 2024-09-10 13:12:25.076411047 +0000 UTC m=-259199.999435947 (missing end time)

$ docker compose -f docker-compose_dss.yaml -p dss_sandbox exec local-dss-core-service db-evictor -cockroach_host=local-dss-crdb -ttl=72h -scd_subs=false
2024/09/13 13:12:57 found operational intent 00000185-e36d-40be-8d38-beca6ca30001; expired due to end time before 2024-09-10 13:12:57.34400607 +0000 UTC m=-259199.999589390
2024/09/13 13:12:57 found operational intent 00000185-e36d-40be-8d38-beca6ca30003; expired due to last update before 2024-09-10 13:12:57.34400607 +0000 UTC m=-259199.999589390 (missing end time)

$ docker compose -f docker-compose_dss.yaml -p dss_sandbox exec local-dss-core-service db-evictor -cockroach_host=local-dss-crdb -ttl=72h -op_intents=false
2024/09/13 13:14:13 found subscription 78f98cc5-94f3-4c04-8da9-a8398feba3f3; expired due to end time before 2024-09-10 13:14:13.905220633 +0000 UTC m=-259199.999567171
2024/09/13 13:14:13 found subscription 9f0d4575-b275-4a4c-a261-e1e04d324565; expired due to last update before 2024-09-10 13:14:13.905220633 +0000 UTC m=-259199.999567171 (missing end time)

$ docker compose -f docker-compose_dss.yaml -p dss_sandbox exec local-dss-core-service db-evictor -cockroach_host=local-dss-crdb -ttl=72h -delete
2024/09/13 13:14:29 deleted operational intent 00000185-e36d-40be-8d38-beca6ca30001; expired due to end time before 2024-09-10 13:14:29.591041778 +0000 UTC m=-259199.999568721
2024/09/13 13:14:29 deleted operational intent 00000185-e36d-40be-8d38-beca6ca30003; expired due to last update before 2024-09-10 13:14:29.591041778 +0000 UTC m=-259199.999568721 (missing end time)
2024/09/13 13:14:29 deleted subscription 78f98cc5-94f3-4c04-8da9-a8398feba3f3; expired due to end time before 2024-09-10 13:14:29.591041778 +0000 UTC m=-259199.999568721
2024/09/13 13:14:29 deleted subscription 9f0d4575-b275-4a4c-a261-e1e04d324565; expired due to last update before 2024-09-10 13:14:29.591041778 +0000 UTC m=-259199.999568721 (missing end time)

$ docker compose -f docker-compose_dss.yaml -p dss_sandbox exec local-dss-core-service db-evictor -cockroach_host=local-dss-crdb -ttl=72h -delete
2024/09/13 13:14:36 no entity older than 2024-09-10 13:14:36.095393278 +0000 UTC m=-259199.999564308 found

[barroco]

You will find a first round of comments. In addition, I think we could leverage the infrastructure used for RID to automate the tests you have made manually without too much effort: https://github.com/interuss/dss/blob/master/Makefile#L113.

[mickmis]

FYI still need to test manually the CLI + add unit tests.