Deps: Update mongoengine to 0.29.1 and pymongo 4.6.3 #6252

cognifloyd · 2024-09-25T02:48:42Z

This updates mongoengine and pymongo because they do not officially support mongo 7 (CI will be switching to 7: See #6236 and #6246) until mongoengine 0.29 (the most recent release) and pymongo 4.4 (where 4.9 is the latest release).

Updating from pymongo 3 to pymongo 4 is a major change that requires following the massive pymongo 4 upgrade guide. This included:

I updated to mongoengine 0.25 and pymongo 3.13
- mongoengine 0.24 is required for pymongo 4 support: changelog
- mongonengine 0.25 fixes some pymongo 4 issues: changelog
- pymongo 3.13.0 backports many pymongo 4 changes adding a bunch of DeprecationWarnings to facilitate migration: changelog
To figure out which deprecations affect us, I temporarily turned all pymongo and mongoengine DeprecationWarnings into errors.
UUID format handling is changing to be more universal across languages. Eventually we should provide a migration guide and/or script with an st2.conf option to allow people to configure that and migrate their existing databases to the new format. There is no hard deadline for doing this however, so I just hard-coded the legacyPython behavior (ie the behavior of pymongo 3.x).
The biggest deprecation we had to deal with is the ssl_* connection vars are no longer supported in pymongo 4. This change was merged in Rename [database].ssl* options to [database].tls* to support pymongo 4 #6250 in preparation for this PR.
I also stepped through the pymongo 4 migration guide using git grep to look for code that used each method or var highlighted by the pymongo 4 upgrade guide. This revealed a few other minor updates. Luckily mongoengine isolated us from most of the pymongo api changes.
Then I updated to mongoengine 0.29 and pymongo 4.9.1, however pymongo 4.7+ will require additional effort as they now support python standard logging. So, if we enable debug logs, we get a huge flood of pymongo logs obscuring everything st2 is doing. Note that debug logs are enabled for local development and for integration tests, so this was a significant issue.
Then I downgraded to pymongo 4.6.3 which is much more recent than 3.12.3 and still supports mongo 7 without the additional refactoring that pymongo 4.7+ will require.
- pymongo 4.6 changelog

Effective `lockfiles/st2.lock` diff

The other requirements files (and the Makefile) were updated based on lockfiles/st2.lock.

Lockfile diff: lockfiles/st2.lock [st2]

==                    Upgraded dependencies                     ==

  eventlet                       0.36.1       -->   0.37.0
  filelock                       3.16.0       -->   3.16.1
  greenlet                       3.0.3        -->   3.1.0
  idna                           3.8          -->   3.10
  importlib-metadata             8.4.0        -->   8.5.0
  kombu                          5.4.0        -->   5.4.2
  mongoengine                    0.23.1       -->   0.29.1
  msgpack                        1.0.8        -->   1.1.0
  paramiko                       3.4.1        -->   3.5.0
  platformdirs                   4.3.2        -->   4.3.6
  pyasn1                         0.6.0        -->   0.6.1
  pyasn1-modules                 0.4.0        -->   0.4.1
  pymongo                        3.12.3       -->   4.6.3
  pytest                         8.3.2        -->   8.3.3
  pytz                           2024.1       -->   2024.2
  setuptools                     74.1.2       -->   75.1.0
  urllib3                        2.2.2        -->   2.2.3
  virtualenv                     20.26.4      -->   20.26.5
  zipp                           3.20.1       -->   3.20.2

to prepare for further upgrades

Lockfile diff: lockfiles/st2.lock [st2] == Upgraded dependencies == eventlet 0.36.1 --> 0.37.0 filelock 3.16.0 --> 3.16.1 greenlet 3.0.3 --> 3.1.0 idna 3.8 --> 3.10 importlib-metadata 8.4.0 --> 8.5.0 kombu 5.4.0 --> 5.4.2 mongoengine 0.23.1 --> 0.25.0 msgpack 1.0.8 --> 1.1.0 paramiko 3.4.1 --> 3.5.0 platformdirs 4.3.2 --> 4.3.6 pyasn1 0.6.0 --> 0.6.1 pyasn1-modules 0.4.0 --> 0.4.1 pymongo 3.12.3 --> 3.13.0 pytest 8.3.2 --> 8.3.3 pytz 2024.1 --> 2024.2 setuptools 74.1.2 --> 75.1.0 urllib3 2.2.2 --> 2.2.3 virtualenv 20.26.4 --> 20.26.5 zipp 3.20.1 --> 3.20.2

…aster This matches the method used in st2common.models.db

Existing databases will have the old pymongo 3.x format, so we need to use PYTHON_LEGACY until there is some kind of migration mechanism. STANDARD would be better (more compatible with Java, C#, etc), but we only use python any way, so that should not be a significant issue. Plus, we use orjson encoded fields, so the database is already only accessible via st2 python code.

The UuidRepresentation enum has integer values. pymongo converts the string into the enum instance.

Lockfile diff: lockfiles/st2.lock [st2] == Upgraded dependencies == mongoengine 0.25.0 --> 0.29.1 pymongo 3.13.0 --> 4.9.1

I don't want to wade through all of our logging code right now. The pymongo 4.7.0 changelog entry says: > Added support for Python’s native logging library, enabling developers > to customize the verbosity of log messages for their applications. > Please see Logging for more information. And the Logging doc says: > Starting in 4.8, PyMongo supports Python’s native logging library, > enabling developers to customize the verbosity of log messages for > their applications. https://pymongo.readthedocs.io/en/stable/examples/logging.html But, we currently have logic in these places that enable DEBUG logs for integration tests and local development, and the now-enabled DEBUG pymongo logs are too overwhelming to deal with right now. - conf/st2.dev.conf [system].debug = True - st2common/st2common/service_setup.py - st2common/st2common/logging/misc.py - st2common/st2common/util/debugging.py Here's the lockfile diff: Lockfile diff: lockfiles/st2.lock [st2] == !! Downgraded dependencies !! == pymongo 4.9.1 --> 4.6.3

amanda11

Just a typo on a comment

amanda11 · 2024-09-25T20:43:56Z

st2common/tests/unit/test_db.py

+    @classmethod
+    def tearDownClass(cls):
+        # since tearDown discconnects, dropping the database in tearDownClass
+        # fails withotu establishing a new connection.


Typo here on withotu

I'll fix that in the next PR

cognifloyd added 8 commits September 24, 2024 21:24

update to mongoengine 0.25 and pymongo 3.13

ac33734

to prepare for further upgrades

update requirements files to match lockfiles/st2.lock

e3681cd

make mongoengine/pymongo deprecations fail tests (hopefully)

69f4da1

pants-plugins/uses_services: use mongo ping instead of deprecated ism…

aa57b53

…aster This matches the method used in st2common.models.db

MongoClient(uuid_representation=) has to be a string

5dc882a

The UuidRepresentation enum has integer values. pymongo converts the string into the enum instance.

add uuidRepresentation to expected args in test_db

3948d60

cognifloyd added enhancement external dependency database mongodb labels Sep 25, 2024

cognifloyd added this to the 3.9.0 milestone Sep 25, 2024

cognifloyd self-assigned this Sep 25, 2024

pull-request-size bot added the size/XL PR that changes 500-999 lines. Consider splitting work into several ones that easier to review. label Sep 25, 2024

cognifloyd added 10 commits September 24, 2024 22:31

pymongo: database_names replaced with list_database_names

85491f2

pymongo: collection_names replaced with list_collection_names

62403ca

bump mongoengine and pymongo to the latest

31a99ec

regenerate lockfiles/st2.lock to bump pymongo+mongoengine

cf28188

Lockfile diff: lockfiles/st2.lock [st2] == Upgraded dependencies == mongoengine 0.25.0 --> 0.29.1 pymongo 3.13.0 --> 4.9.1

update requirements files to match lockfiles/st2.lock

72f07e2

fix "InvalidOperation: Cannot use MongoClient after close" in test_db

d473f42

drop warnings->errors for mongo update

51c70c0

add changelog entry

407bc3e

pants-plugins/uses_services/mongo: Add uuidRepresentation

4c0a316

cognifloyd force-pushed the update-mongo-libs branch from f391142 to 4c0a316 Compare September 25, 2024 03:32

update changelog entry

b213645

cognifloyd marked this pull request as ready for review September 25, 2024 03:47

cognifloyd enabled auto-merge September 25, 2024 03:48

cognifloyd requested a review from nzlosh September 25, 2024 03:48

cognifloyd requested review from mamercad, guzzijones, amanda11 and a team September 25, 2024 03:48

cognifloyd mentioned this pull request Sep 25, 2024

Update from Mongo 4.4 to Mongo 7 in CI #6246

Merged

nzlosh approved these changes Sep 25, 2024

View reviewed changes

cognifloyd requested a review from a team September 25, 2024 15:14

guzzijones approved these changes Sep 25, 2024

View reviewed changes

cognifloyd merged commit a5f4d1a into master Sep 25, 2024
29 checks passed

cognifloyd deleted the update-mongo-libs branch September 25, 2024 19:45

amanda11 reviewed Sep 25, 2024

View reviewed changes

cognifloyd mentioned this pull request Oct 8, 2024

TSC Meeting (8 Oct 2024) StackStorm/community#143

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deps: Update mongoengine to 0.29.1 and pymongo 4.6.3 #6252

Deps: Update mongoengine to 0.29.1 and pymongo 4.6.3 #6252

cognifloyd commented Sep 25, 2024 •

edited

Loading

amanda11 left a comment

amanda11 Sep 25, 2024

cognifloyd Sep 25, 2024

Deps: Update mongoengine to 0.29.1 and pymongo 4.6.3 #6252

Deps: Update mongoengine to 0.29.1 and pymongo 4.6.3 #6252

Conversation

cognifloyd commented Sep 25, 2024 • edited Loading

Effective lockfiles/st2.lock diff

amanda11 left a comment

Choose a reason for hiding this comment

amanda11 Sep 25, 2024

Choose a reason for hiding this comment

cognifloyd Sep 25, 2024

Choose a reason for hiding this comment

cognifloyd commented Sep 25, 2024 •

edited

Loading

Effective `lockfiles/st2.lock` diff