Map ingestion processing

[davenquinn]

@CannonLock @brianaydemir - just sketching how we could implement an initial "auto-processing" system for the map ingestion steps:

• Watch for inserts/updates on ingest_process or a dedicated queue table. Could be a polling implementation or something with a trigger / pg_notify event
• Either an always-on watcher container or kubernetes cron job listen/polls the queue and kicks off the processing.
• We'd have to dump errors/logs to a table so that we could keep track of processing and issues

This actually seems pretty straightforward given where we are now. Maybe we can try to implement in the next few weeks so we don't have to remember as many CLI commands at/leading up to the workshop?

[brianaydemir]

Getting something up and running time for the workshop ought to be doable.

Assuming that querying ingest_process is relatively cheap, I'd lean towards polling it as the initial implementation. The polling interval would determine whether I learn towards Kubernetes CronJob vs. a Pod that runs an infinite loop with periodic sleeps.