Issues encountered when helm upgrading or re-installing mission-control-agent

[junaid-ebrahim]

My scenario is a local minikube cluster using a cloud based mission-control. When upgrading from an older version of the chart to commit version #155 , I encounter the following issues

1. Postgres user password set in secret incident-commander-postgres for Postgres-0 pod did not match the secret in the database
   workaround: was to exec into pod and update postgres secret manually

2. After applying point 1. workaround it fixed canary checker and configDB pods but the mission-control-agent pod did not start because the migrations failed on sql statement below

   alter table agents drop column last_seen;
   workaround: manually log into the postgres pod and run the above sql command with CASCADE at the end.

3. After workaround above , the mission-control-agent pod ran but I noticed some old\deleted scrapers were still active and executing
   workaround: manually log into the postgres pod and run "truncate config_scrapers CASCADE;"

4. After all this the configDB Logs contain a 2 column missing errors , workaround is to add the columns mentioned
   2024/03/16 15:50:47 logger.go:165: /app/db/config.go:76 ERROR: column "last_scraped_time" of relation "config_items" does not exist (SQLSTATE 42703)

   2024-03-16T15:51:21.011 ERROR ReconcileConfigs ReconcileConfigs{https://mc.org-tdmh6m6mujgk.workload-prod-eu-01.flanksource.com} failed to reconcile table config_scrapers: failed to fetch unpushed items for table {00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 0001-01-01 00:00:00 +0000 UTC }: ERROR: column "is_pushed" does not exist (SQLSTATE 42703)