[24.0] Fix deadlock that can occur when changing job state

[mvdbeek]

This is a deadlock that occurred on main:

DeadlockDetected: deadlock detected
DETAIL:  Process 116772 waits for ShareLock on transaction 5465347; blocked by process 3511756.
Process 3511756 waits for ShareLock on transaction 5465546; blocked by process 116772.
HINT:  See server log for query details.
CONTEXT:  while updating tuple (3575330,77) in relation "job"

  File "sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (psycopg2.errors.DeadlockDetected) deadlock detected
DETAIL:  Process 116772 waits for ShareLock on transaction 5465347; blocked by process 3511756.
Process 3511756 waits for ShareLock on transaction 5465546; blocked by process 116772.
HINT:  See server log for query details.
CONTEXT:  while updating tuple (3575330,77) in relation "job"

[SQL: UPDATE job SET update_time=%(update_time)s, state=%(state)s WHERE job.id = %(id_1)s AND (job.state NOT IN (%(state_1_1)s, %(state_1_2)s))]
[parameters: {'update_time': datetime.datetime(2024, 4, 2, 14, 58, 14, 475927), 'state': <JobState.PAUSED: 'paused'>, 'id_1': 56817470, 'state_1_1': <JobState.DELETING: 'deleting'>, 'state_1_2': <JobState.DELETED: 'deleted'>}]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
  File "galaxy/jobs/handler.py", line 563, in __handle_waiting_jobs
    job.set_state(model.Job.states.PAUSED)
  File "galaxy/model/__init__.py", line 1695, in set_state
    rval = session.execute(
  File "sqlalchemy/orm/session.py", line 1717, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "sqlalchemy/engine/base.py", line 1710, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
  File "sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
    ret = self._execute_context(
  File "sqlalchemy/engine/base.py", line 1953, in _execute_context
    self._handle_dbapi_exception(
  File "sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
    util.raise_(
  File "sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)

the other side also set the state to paused. We can reduce the probability of that happening by shortening the window during which the transaction is open. Also excludes more states in the where clause.

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[jdavcs]

2 minor edit suggestions for inline comments, the rest looks good. Thanks!