-
Notifications
You must be signed in to change notification settings - Fork 53
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Upgrade base system to Ubuntu 24.04 #1076
Conversation
This should have been done right after the 1.5 release, so let's do it now.
The NodeJs package manager (NPM) is no longer automatically installed when we install NodeJS, so we ask for it explicitly. We need it to install Obographviz. Ideally it should be possible to install both NPM and Obographviz in the builder image and then to transfer only Obographviz to the final ODKFull image, thereby reducing the clutter in ODKFull, but this will need further investigation.
This is a runtime dependency of Konclude (even the statically compiled one that we use on x86_64). It was probably automatically pulled by another package on Ubuntu 22.04, but it is not on 22.04, so we need to ask for it explicitly.
Pip is now by default refusing to install anything in the system-wide Python library path, which is considered the "private garden" of the underlying operating system (e.g., on Ubuntu, only APT tools should add Python packages to the system-wide path). This may be fine in general, in a user-facing scenario, but not for the ODK which is in effect a "read-only" system overall. The OBO Dashboard is part of the tools/libraries we provide with the ODK and there is no reason for us to package it separately inside a virtual environment (the recommended way of installing non-system Python packages) -- the entire ODK is already a "virtual environment". Ultimately the right thing to do here would be for the OBO Dashboard upstream to make proper releases, which we could then install at the same time as any other Python packages.
The odk.py script has some issues when we run it under Python 3.12, we fix them here.
Installing virtualenv in the builder image has nafarious consequences when we try to later install Python packages. That's because the Ubuntu package for virtualenv automatically installs platformdirs version 2.5.1, which then prevents us from installing the platformdirs 4.x that we need as a dependency for some of our packages. The only reason we had virtualenv in the builder image was that we use it to run the update-constraints workflow (in which we try to install all our Python packages in a virtualenv). So here, we 1) remove virtualenv from the builder image; 2) amend the update-constraints.sh script to make it install virtualenv itself.
PIP emits a warning when we attempt to install executable scripts in a location that is not in the system PATH. In this instance the warning is not warranted. The scripts are installed in a staging location ON PURPOSE -- they are later copied over to the final ODK images, where the scripts will end up in the system PATH. So we shut that warning down.
Get freshly frozen Python constraints for the newer version of Python provided in Ubuntu 24.04.
(For those keeping track, this the same PR as #1067 modulo the wording of some commit messages and freshly updated Python constraints.) |
OLD COMMENT PERTAINING TO setuptools INSTALL ERRORI was almost done testing, when I encountered some issues with some incompatible python packages. Can you instruct me again how to update constraints.txt locally? Just running the update script locally causes errors like
And I am not sure if this is due to my local setup or something else. I also tried to do this from within the freshly build ODK container, but the error seems the same.. |
We don’t have a standard procedure for that when we update the base system. The normal procedure ( The idea is to create a Ubuntu-based Docker images with the same packages (Ubuntu packages, not Python packages) as in the builder image, then within that image try to install all the Python packages without the current constraints. Then freeze the resulting Python environment to produce a whole new
I reproduce the error here. From what I can tell, this is due to |
Where to begin? It's just another Python screw-up. Developers of setuptools have announced, back in 2017 (!) their intention to remove the `test` command (found in setuptools.command.test) [1]. Today, they thought that 7 years of advance deprecation warnings were enough, and they published setuptools v72.0.0 which no longer provides that command. [2] But the Python ecosystem being what it is (a joke that has lasted long enough), it turns out that 7 years was not actually enough, and there are still many Python packages out there that didn't get the memo and that are still dependent on setuptools.command.test (including some packages that are used in the ODK). There are several ways of fixing this: a) Ditch Python altogether. This would be by far the best solution, but one that is, alas, unlikely to happen any time soon. b) Fix _all_ the packages used by the ODK (and their dependencies) so that they do what they should have done in the past 7 years. This is the correct solution, but one that will take time (7 years were already not enough, so who knows how long it is going to take?). c) Torture the setuptools developers until they agree to revert the removal of the `test` command. I believe this may be illegal in many jurisdictions. d) Force the ODK build process to use a version of `setuptools` older than 72.0.0. This is what we are doing here. Yes, this is a local workaround and we should favour "upstream fixing" instead (solution b), but if we want to be be able to build the ODK before 2031 (optimistic estimation of the time it will take to get all the Python packages fixed), we don't have much choice. [1] pypa/setuptools#931 [2] https://setuptools.pypa.io/en/stable/history.html
The j2cli project (which provides the j2 tool) is no longer maintained and is broken under Python 3.12, so we replace it with Jinjanator (providing the similar command jinjanate).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After 10 rebuilds and dozens of tests and runs and pipeline upgrades, I am personally convinced everything is ok with this change. I want to note a tiny wrinkle which is a bit un-Damien like, which is coupling the python migration with the image-base migration.
As it is likely that we need 1.5.3 before 1.6, there is a small but not non-existent risk that some interactions between the old base image and the new python packages could cause some problems (unless of course, you would personally judge the base image migration as a "minor" change, since it does not affect the repo setup).
I know you have thought all of this through! Good to go, great work and thank you!
Not sure what is so “un-Damien like” in that. Python 3.12 is the version of Python that is provided with Ubuntu 24.04, while Ubuntu 22.04 (our previous base image) provided Python 3.10. The Python migration from 3.10 to 3.12 is merely a consequence of the base image migration, so of course it belongs to the same PR. In other words, the base image and the Python version are already coupled by default. To change that, we’d need to actively uncouple them, which would be a completely different matter and would warrant a distinct PR just for that. Uncoupling could be done by:
Overall, I believe uncoupling Python from the base system would be a needless complication. Having to migrate Python (which can include having to tweak the
I’d rather have a 1.6 before we need to make a 1.5.3 release, so as to avoid the question altogether, but it we do need to make a 1.5.3 release, yes, I would consider the base image migration a minor change. It should not impact existing instances that uses the standard workflows. Instances that use custom workflows MAY be impacted (e.g. when a custom workflow is using |
I did not realise this, probably because I didnt read something carefully enough!
All good then! Thanks for the clarifications! |
This PR updates the base system of all our images to the latest LTR version of Ubuntu, 24.04.
libpcre3
, needed for Konclude, ornpm
, needed to install Obographviz).--break-system-packages
, otherwisepip
will by default refuse to install it system-wide (the idea is that only APT tools are allowed to install anything system-wide).virtualenv
in the builder image, as it messes with the installation of our Python packages.odk.py
script has some issues with Python 3.12 that need fixing.closes #1008