ETL enhancements v1

[vinayvenu]

This epic will make some necessary modifications that will improve operational and other capability of ETL.

Objectives

• Both implementers and users need to understand if the ETL for their organisation is healthy. Currently, the only way to find this out is by looking at the logs. In order to change that, we should be able to see status of ETL broken down by organisation.
• We also need to be able to run ETL for an organisation separately (with auth) so that any issues can be fixed separately without using up too much of system resources
• We also have to ensure that no two ETL job runs together to avoid issues of data corruption.
• Lastly, there are some parts of ETL that have not migrated out to the schema level tables. Some stories will enable this

Tasks

Preview Give feedback

1. Run ETL through API #49
   +0
   
2. Authentication for ETL Service #53
   bug +1
   
3. Secondary table for repeatable question groups #22
   +0
   
4. Support group subjects in ETL schema #50
   +0
   
5. Move job to run separately per organisation #48
   +0