Retry updating conda packages

[soapy1]

Fixes #852

Description

Th issue described in #852 arises because part of the environment build process is to execute the action_add_conda_prefix_package. This action tries to insert all the packages from the environment into the db. If these packages overlap with any of the packages inserted by task_update_conda_channel (which is common on a first run of the task) then we get the integrity error.

This PR adds a retry to the part of the task_update_conda_channel task that updates the conda_package table. So, if an integrity error occurs due to a package getting inserted by another process while a bulk update is happening, then the whole process for inserting the conda packages will try again. Subsequent bulk inserts are expected to work, since part of the update process is to only insert packages that haven't already been added.

Other options I looked out to resolve this issue are:

• using an upsert statements instead of an insert statement
  • sqlalchemy does not provide a platform agnostic way to do upserts. Since conda-store is meant to be db platform agnostic, I think this is not an great option.
• briefly looked into locking the conda_package table to writes while the bulk insert is happening.
  • I don't think this is a great option since it's pretty heavy handed and may have unintentional consequences as the system evolves.

Pull request checklist

• Did you test this change locally?
• Did you update the documentation (if required)?
• Did you add/update relevant tests for this change (if required)?

[netlify]

✅ Deploy Preview for conda-store canceled.

Name	Link
🔨 Latest commit	1510115
🔍 Latest deploy log	https://app.netlify.com/sites/conda-store/deploys/67328ca67680630008d61efa

[netlify]

✅ Deploy Preview for conda-store canceled.

Name	Link
🔨 Latest commit	c16061d
🔍 Latest deploy log	https://app.netlify.com/sites/conda-store/deploys/67328fccf119570008a467c6

[peytondmurray]

The reason we're seeing this issue is a race condition that happens between

1. The task that inserts packages found from the repodata.json that gets downloaded when a channel gets updated (task_update_conda_channel), and
2. The preexisting conda packages in the the conda-store prefixes (action_add_conda_prefix_packages)

It seems like making this decorator instantly retry the decorated function upon failure is just delaying one of the two competing tasks rather than solving the underlying race condition. The bulk insert above that gets retried seems like an optimization (that's why it's using bulk insert rather than package by package insertion), so is it possible that action_add_conda_prefix_packages wouldn't yet be done by the time the retry is made?

[soapy1]

Yep, I think you're right. We could wait some time before retrying. But maybe it's worth thinking about it a little more to find a better solution 🤔

[soapy1]

I was thinking of a few other options:

1. don't run the action_add_conda_prefix_packages if the task_update_conda_channel task is running. This option isn't so good because if the task fails, the package won't get added and we'll have no way know about the missing data.
2. when updating channels, check and insert each package instead of doing a bulk insert. This would put a lot more load on the database. This will definitely make things less efficient. We are inserting on the order of 10000's packages per architecture. Probably not a good option