Migration error upgrading from v2.9.1 to v4.1.2

[glennmatthews]

Describe the problem

We're seeing an issue upgrading an existing deployment from Constance 2.9.1 to 4.1.2.

Steps to reproduce

• Have a project with database records from 2.9.1 (Constance configs in constance_config database table, migrations run include database 0001_initial and database 0002_auto_20190129_2304
• Have apps in the project that attempt to read from the Constance config at startup time (e.g. to determine which app features are administratively enabled)
• Upgrade to Constance 4.1.2 and run migrations

04:22:08.517 WARNING nautobot.apps.config :
  "PLUGINS_CONFIG['nautobot_chatops']['enable_grafana']" is not in settings, and could not read from the Constance datab
ase table (perhaps not initialized yet?)
...
04:22:17.282 WARNING nautobot.core.utils.config :
  Configuration "NETWORK_DRIVERS" is not in settings, and could not read from the Constance database table (perhaps not 
initialized yet?)
Performing database migrations...
...
Applying constance.0001_initial... OK
  Applying constance.0002_migrate_from_old_table...Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "constance_constance_pkey"
DETAIL:  Key (id)=(1) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/nautobot-server", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/nautobot/core/cli/__init__.py", line 293, in main
    execute_from_command_line([sys.argv[0], *unparsed_args])
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_l
ine
    utility.execute()
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/nautobot/core/management/commands/post_upgrade.py", line 91, in handle
    call_command(
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 194, in call_command
    return command.execute(*args, **defaults)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
                         ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwa
rds
    self.code(from_state.apps, schema_editor)
  File "/usr/local/lib/python3.11/site-packages/constance/migrations/0002_migrate_from_old_table.py", line 21, in _migra
te_from_old_table
    cursor.execute(
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.IntegrityError: duplicate key value violates unique constraint "constance_constance_pkey"
DETAIL:  Key (id)=(1) already exists.

Sure enough, somehow the attempt to read the records from constance_constance during app startup has resulted in records being created in this newly initialized table, which are now conflicting with the records that Constance is trying to auto-migrate from constance_config:

nautobot=# select id, key from constance_config;
 id |                             key                              
----+--------------------------------------------------------------
  1 | constance:nautobot:DEPLOYMENT_ID
  2 | constance:nautobot:DYNAMIC_GROUPS_MEMBER_CACHE_TIMEOUT
  3 | constance:nautobot:nautobot_chatops__enable_grafana
  4 | constance:nautobot:nautobot_chatops__enable_slack
  5 | constance:nautobot:nautobot_chatops__enable_ms_teams
  6 | constance:nautobot:nautobot_chatops__enable_webex
  7 | constance:nautobot:nautobot_chatops__enable_mattermost
  8 | constance:nautobot:SUPPORT_MESSAGE
  9 | constance:nautobot:CHANGELOG_RETENTION
 10 | constance:nautobot:RELEASE_CHECK_URL
 11 | constance:nautobot:BANNER_TOP
 12 | constance:nautobot:BANNER_BOTTOM
 13 | constance:nautobot:PAGINATE_COUNT
 14 | constance:nautobot:MAX_PAGE_SIZE
 15 | constance:nautobot:PER_PAGE_DEFAULTS
 16 | constance:nautobot:nautobot_chatops__enable_ansible
 17 | constance:nautobot:nautobot_chatops__enable_aci
 18 | constance:nautobot:nautobot_chatops__enable_ipfabric
 19 | constance:nautobot:nautobot_chatops__enable_aristacv
 20 | constance:nautobot:nautobot_chatops__enable_meraki
 21 | constance:nautobot:nautobot_chatops__enable_panorama
 22 | constance:nautobot:NETWORK_DRIVERS
 23 | constance:nautobot:PREFER_IPV4
 24 | constance:nautobot:DEVICE_NAME_AS_NATURAL_KEY
 25 | constance:nautobot:LOCATION_NAME_AS_NATURAL_KEY
 26 | constance:nautobot:nautobot_golden_config__DEFAULT_FRAMEWORK
 27 | constance:nautobot:BANNER_LOGIN
 28 | constance:nautobot:RACK_ELEVATION_DEFAULT_UNIT_WIDTH
 29 | constance:nautobot:RACK_ELEVATION_DEFAULT_UNIT_HEIGHT
(29 rows)

nautobot=# select id, key from constance_constance;
 id |                          key                           
----+--------------------------------------------------------
  1 | constance:nautobot:nautobot_chatops__enable_grafana
  2 | constance:nautobot:NETWORK_DRIVERS
  3 | constance:nautobot:nautobot_chatops__enable_slack
  4 | constance:nautobot:nautobot_chatops__enable_ms_teams
  5 | constance:nautobot:nautobot_chatops__enable_webex
  6 | constance:nautobot:nautobot_chatops__enable_mattermost
(6 rows)

All of the records currently showing in constance_constance are settings that apps are attempting to read during application startup, so something (presumably django-constance itself?) resulted in these being written to the database in the middle of running migrations.

System configuration

• Django version: 4.2
• Python version: 3.11
• Django-Constance version: 4.1.2