Data import: maintenance mode during data update

[niconoe]

There's now a mechanism in place to automatically refresh data by downloading from GBIF.

During this process, the database spends a few minutes in a messed up state (occurrence duplicates because new data is added before the previous one is deleted, ...).

We need to show the website as temporarily unavailable during this time to avoid presenting incorrect data to users. I consider using django-maintenance-mode for that.

Additional considerations:

• the data imports will probably be run at night, to limit interference with the users
• infrastructure: we'll need an alert (mail? others) for urgent intervention if the website stays for too long in maintenance mode (error during the data import process: GBIF outage, bug, ...) (not needed thanks to transactions)
• we need to take precautions so user alerts (mail for new observations, ...) are not sent when the website is in maintenance mode

[damianooldoni]

I definitely agree with you.
If we run the data import at night (I would say midnight) as you said AND the early warning system alerts are sent at 2pm for example, then we have time to check what is going wrong.

[niconoe]

Yeah. But I'll also implement also a check in the "sending alert" code to make sure they just it just waits if maintenance mode is on (so we don't break everything because we decided to change hours or because the server time is incorrect)

[damianooldoni]

Yes, for sure! 👍

[niconoe]

The occurrence import is now atomic (#62). If all goes well, we should be able to tackle this simply and elegantly using the following logic:

1. new data is downloaded from GBIF
2. maintenance mode is set to ON (website inaccessible + no emails sent)
3. the occurrence import is done (atomically: all changes or nothing in the database)
4. if the transaction was aborted: send an email to administrators to check what happened
5. in all cases, the maintenance is set to OFF (in worse case - transaction aborted - no new data will be visible in the website but at least the data is still perfectly consistent and the website isn't stuck in maintenance mode)

[damianooldoni]

I find it a good workflow, it's defensive enough, I think. One small question, maybe trascurable problem: what to do with downloaded data in 1 if 4 happens for many consecutive days (e.g. you and your hpothetic back-up person are both on holidays)? Probably memory it's not a big issue, right?

[niconoe]

@damianooldoni: Indeed, I don't expect any disk space problem:

• The downloads are quite small (with the current species list, we're at 7 Mb / day)
• In addition to that, the download happens in a temporary directory: as soon as the processes finish, the system knows this disk space can be freed when necessary.

[niconoe]

Update: I'm considering using https://github.com/fabiocaccamo/django-maintenance-mode for the maintenance mode itself.

[niconoe]

This is working fine for years, IU think we can close this!