[23.2] Anticipate PendingRollbackError in check_database_connection

[mvdbeek]

Should be a straightforward way to fix:

PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (psycopg2.OperationalError) server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

[SQL: UPDATE dataset SET update_time=%(update_time)s, object_store_id=%(object_store_id)s WHERE dataset.id = %(dataset_id)s]
[parameters: ({'update_time': datetime.datetime(2024, 3, 4, 12, 27, 29, 277355), 'object_store_id': 'corral4', 'dataset_id': 103938181}, {'update_time': datetime.datetime(2024, 3, 4, 12, 27, 29, 277359), 'object_store_id': 'corral4', 'dataset_id': 103938182}, {'update_time': datetime.datetime(2024, 3, 4, 12, 27, 29, 277361), 'object_store_id': 'corral4', 'dataset_id': 103938183})]
(Background on this error at: https://sqlalche.me/e/14/e3q8) (Background on this error at: https://sqlalche.me/e/14/7s2a)
  File "galaxy/jobs/handler.py", line 376, in __monitor
    self.__monitor_step()
  File "galaxy/jobs/handler.py", line 394, in __monitor_step
    self.__handle_waiting_jobs()
  File "galaxy/jobs/handler.py", line 406, in __handle_waiting_jobs
    check_database_connection(self.sa_session)
  File "galaxy/model/base.py", line 69, in check_database_connection
    if session and session.connection().invalidated:
  File "<string>", line 2, in connection
  File "sqlalchemy/orm/session.py", line 1545, in connection
    return self._connection_for_bind(
  File "sqlalchemy/orm/session.py", line 1555, in _connection_for_bind
    return self._transaction._connection_for_bind(
  File "sqlalchemy/orm/session.py", line 724, in _connection_for_bind
    self._assert_active()
  File "sqlalchemy/orm/session.py", line 604, in _assert_active
    raise sa_exc.PendingRollbackError(

from https://sentry.galaxyproject.org/share/issue/5c54d2c8fb474fe7b38614dac773550e/

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[jdavcs]

How about this:

def check_connection(session):
    assert session
    if not session.get_transaction().is_active or session.connection().invalidated:
        session.rollback()
        log.error("Database transaction rolled back due to invalid state.")

Here's the gist of it. There is a very specific state in which this particular error will be raised: the session's transaction's internal state should be DEACTIVE AND there should have been at least one error raised during a previous rollback (on this OR a nested transaction). Then this error will be raised as a result of one of several method calls inside the transaction class (that's all internal SA code), and one of them is Session.connection() - that's what triggers the error on line 69. However, instead of checking for the above conditions, we can simply check SessionTransaction.is_active. The code above handles both bases. The call to rollback will not re-raise the pre-existing rollback exception.

[mvdbeek]

This does seem a bit more reliant on SQLAlchemy internal state but that's fine either way. Can you push that to my PR?

[jdavcs]

whoops. I see it. Fixing.