Deadlock on saving a task instance

[ppawlak]

Hello,

We have a production setup with lots of workers running in parallel and a deadlock error occur from time to time, I think when two workers are trying the update the same task at the same moment. Here is the traceback of the error:

DeadlockDetected: deadlock detected
DETAIL:  Process 3256555 waits for ShareLock on transaction 3220907091; blocked by process 3256561.
Process 3256561 waits for ShareLock on transaction 3220907090; blocked by process 3256555.
HINT:  See server log for query details.
CONTEXT:  while updating tuple (1239,37) in relation "huey_monitor_taskmodel"

  File "django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)

OperationalError: deadlock detected
DETAIL:  Process 3256555 waits for ShareLock on transaction 3220907091; blocked by process 3256561.
Process 3256561 waits for ShareLock on transaction 3220907090; blocked by process 3256555.
HINT:  See server log for query details.
CONTEXT:  while updating tuple (1239,37) in relation "huey_monitor_taskmodel"

  File "huey/consumer.py", line 94, in initialize
    startup_hook()
  File "huey_monitor/tasks.py", line 122, in startup_handler
    update_task_instance(
  File "huey_monitor/tasks.py", line 45, in update_task_instance
    instance.save(update_fields=update_fields)
  File "bx_django_utils/models/timetracking.py", line 59, in save
    super().save(**kwargs)
  File "django/db/models/base.py", line 814, in save
    self.save_base(
  File "django/db/models/base.py", line 877, in save_base
    updated = self._save_table(
  File "django/db/models/base.py", line 990, in _save_table
    updated = self._do_update(
  File "django/db/models/base.py", line 1054, in _do_update
    return filtered._update(values) > 0
  File "django/db/models/query.py", line 1231, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "django/db/models/sql/compiler.py", line 1982, in execute_sql
    cursor = super().execute_sql(result_type)
  File "django/db/models/sql/compiler.py", line 1560, in execute_sql
    cursor.execute(sql, params)
  File "django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)

Any idea on how to prevent this?
We are using postgreSQL if it matters.