Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove Generator in _quarantine_media_txn() #17813

Open
wants to merge 2 commits into
base: develop
Choose a base branch
from

Conversation

realtyem
Copy link
Contributor

when updating the remote_media_cache table

Pull Request Checklist

@CLAassistant
Copy link

CLAassistant commented Oct 10, 2024

CLA assistant check
All committers have signed the CLA.

@realtyem realtyem marked this pull request as ready for review October 10, 2024 17:44
@realtyem realtyem requested a review from a team as a code owner October 10, 2024 17:44
@anoadragon453
Copy link
Member

@realtyem do you have a reference for why providing generators to txn.executemany or txn.execute_batch may be a bad idea?

@realtyem
Copy link
Contributor Author

@realtyem do you have a reference for why providing generators to txn.executemany or txn.execute_batch may be a bad idea?

matrix-org/synapse#15439

Is a lot of the basis. From that, I distill the TLDR as "Generators may cause inadvertent data loss during a db query retry by exhausting an iterable"

@realtyem
Copy link
Contributor Author

The comments at spot seem to be relevant. Although this set of PR's may not necessarily be (part of|used by) this infrastructure, still seems prudent to not use generators as a good habit/safety precaution. It may be they end up used in the future, so don't let our tomorrow selves find the foot gun. (And if they do get used in this infrastructure, then is the 'inspection' infrastructure in this function working?)

"""Start a new database transaction with the given connection.
Note: The given func may be called multiple times under certain
failure modes. This is normally fine when in a standard transaction,
but care must be taken if the connection is in `autocommit` mode that
the function will correctly handle being aborted and retried half way
through its execution.
Similarly, the arguments to `func` (`args`, `kwargs`) should not be generators,
since they could be evaluated multiple times (which would produce an empty
result on the second or subsequent evaluation). Likewise, the closure of `func`
must not reference any generators. This method attempts to detect such usage
and will log an error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants