Improve the “update constraints” mechanism #1129
Conversation
To compute the set of Python constraints, we install all the Python packages we need in a bare Ubuntu image (same version as the one used to build the ODK itself) in which we only install Python3 and PIP. This ensure that `pip-constraints.txt`, the list of Python packages that are installed by the system and that PIP should not touch no matter what, always reflect the actual list of system-managed Python packages found in the ODK. This has the side-effect of allowing to use the `update-constraints.sh` script for the specific case of bootstrapping the constraints when the base image is upgraded to a newer version of Ubuntu, something that previously necessitated an undocumented manual operation.
When building the ODK builder image, before attempting to install the Python packages, we obtain the list of packages that have already been installed by the system (i.e. by APT) and compare it with the pre-computed list in `pip-constraints.txt`. The two lists should normally always be the same. They MAY differ if, between the time the constraints were last updated and the current build, Ubuntu updated some of its Python packages (e.g. if they bump `python3-setuptools` to a more recent minor version because of a security vulnerability). If that happens, we allow the build to continue (it may not necessarily cause a fatal error), but we log a warning so that, should the build fail, we will know that we likely need to refresh the constraints to reflect the updated Ubuntu packages.
|
And another run of the test suite, yeah! |
As a general rule, I wait for the test suite to have passed before asking for a review. :) (Officially to avoid wasting people’s time, by not asking them to review something that may need to be modified because the tests failed; in reality, to preserve the chance that, if I make a stupid mistake – the kind of mistake that the test suite will catch immediately –, I can fix it before anyone else can see it. :D ) |
I am mildly curious of what in the |
|
To be fair I think that Pip’s documentation is quite lacking here. |
This PR builds upon #1126 to make the updating of Python constraints more robust.
The key change is that, when computing the Python constraints, we install all the required Python packages in a minimal image that is just
ubuntu:24.04(same image as the base used to build the ODK) in which we install nothing else thanpython3-devandpython3-pip.Previously, we were installing the Python packages in an image that was the ODK itself (
obolibrary/odkfull:latest), which was problematic for at least two reasons:setuptoolsandwheels, and possibly others) this is not enough. Using a minimal image ensures that we are always computing the Python constraints from a clean state.obolibrary/odkfull:latest(based on the updated Ubuntu image) to compute the new Python constraints, but we need the new Python constraints to build the newobolibrary/odkfull:latest. Previously, we were working around that issue by having a separate, undocumented process specifically to update the Python constraints after a base image upgrade. Now we will be able to use exactly the same process (sameupdate-constraints.shscript) both for the regular updates of the Python constraints, and for the “big updates” after a migration to a new base image.In addition, when building the ODK itself (more precisely, the
odkbuilderimage), we check that the list of system-managed Python packages matches what we are expecting (what was determined by theupdate-constraints.shscript, and stored inpip-constraints.txt), and we log a warning if there is a discrepancy (which may happen if a Ubuntu-provided Python package has been updated, e.g. for security reasons, since the last refresh of the constraints -- something that should be unlikely to happen often, but just in case).