When migrating migration 0003 I get a UnicodeDecodeError error...

[hvdklauw]

Describe the problem

Traceback (most recent call last):
  File "/code/api/manage.py", line 29, in <module>
    execute_from_command_line(sys.argv)
  File "/pypoetry/venv/zupr-MATOk_fk-py3.12/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/pypoetry/venv/zupr-MATOk_fk-py3.12/lib/python3.12/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/pypoetry/venv/zupr-MATOk_fk-py3.12/lib/python3.12/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/pypoetry/venv/zupr-MATOk_fk-py3.12/lib/python3.12/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pypoetry/venv/zupr-MATOk_fk-py3.12/lib/python3.12/site-packages/django/core/management/base.py", line 107, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pypoetry/venv/zupr-MATOk_fk-py3.12/lib/python3.12/site-packages/django/core/management/commands/migrate.py", line 357, in handle
    post_migrate_state = executor.migrate(
                         ^^^^^^^^^^^^^^^^^
  File "/pypoetry/venv/zupr-MATOk_fk-py3.12/lib/python3.12/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pypoetry/venv/zupr-MATOk_fk-py3.12/lib/python3.12/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/pypoetry/venv/zupr-MATOk_fk-py3.12/lib/python3.12/site-packages/django/db/migrations/executor.py", line 255, in apply_migration
    state = migration.apply(state, schema_editor)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pypoetry/venv/zupr-MATOk_fk-py3.12/lib/python3.12/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/pypoetry/venv/zupr-MATOk_fk-py3.12/lib/python3.12/site-packages/django/db/migrations/operations/special.py", line 196, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/pypoetry/venv/zupr-MATOk_fk-py3.12/lib/python3.12/site-packages/constance/migrations/0003_drop_pickle.py", line 54, in migrate_pickled_data
    if value is not None and not is_already_migrated(value):
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pypoetry/venv/zupr-MATOk_fk-py3.12/lib/python3.12/site-packages/constance/migrations/0003_drop_pickle.py", line 17, in is_already_migrated
    data = json.loads(value)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/json/__init__.py", line 341, in loads
    s = s.decode(detect_encoding(s), 'surrogatepass')
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

Steps to reproduce

Have some pickled data in the database that's not decodable

tried clearing redis right before dpeloyment but it's getting filled up again by the currently running proces, can probably get around it by bringing everything down and then deploying, but ideally the decode errors or for that matter any exception get's caught and handled.

I think handling this in a migration is a bad call, it should be handled in constance when loading from redis. Because this redis migration is not transactional like the database migrations are.

System configuration

• Django version: 5.1.1
• Python version: 3.12.2
• Django-Constance version: 4.1.2