Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade base system to Ubuntu 24.04 #1076

Merged
merged 16 commits into from
Jul 31, 2024
Merged

Upgrade base system to Ubuntu 24.04 #1076

merged 16 commits into from
Jul 31, 2024

Conversation

gouttegd
Copy link
Contributor

This PR updates the base system of all our images to the latest LTR version of Ubuntu, 24.04.

  • Some system packages that were previously implicitly installed now need to be explicitly asked for (e.g. libpcre3, needed for Konclude, or npm, needed to install Obographviz).
  • The OBO Dashboard must now be installed with --break-system-packages, otherwise pip will by default refuse to install it system-wide (the idea is that only APT tools are allowed to install anything system-wide).
  • We can no longer have virtualenv in the builder image, as it messes with the installation of our Python packages.
  • The odk.py script has some issues with Python 3.12 that need fixing.

closes #1008

This should have been done right after the 1.5 release, so let's do it
now.
The NodeJs package manager (NPM) is no longer automatically installed
when we install NodeJS, so we ask for it explicitly.

We need it to install Obographviz.

Ideally it should be possible to install both NPM and Obographviz in
the builder image and then to transfer only Obographviz to the final
ODKFull image, thereby reducing the clutter in ODKFull, but this will
need further investigation.
This is a runtime dependency of Konclude (even the statically compiled
one that we use on x86_64). It was probably automatically pulled by
another package on Ubuntu 22.04, but it is not on 22.04, so we need to
ask for it explicitly.
Pip is now by default refusing to install anything in the system-wide
Python library path, which is considered the "private garden" of the
underlying operating system (e.g., on Ubuntu, only APT tools should add
Python packages to the system-wide path).

This may be fine in general, in a user-facing scenario, but not for the
ODK which is in effect a "read-only" system overall. The OBO Dashboard
is part of the tools/libraries we provide with the ODK and there is no
reason for us to package it separately inside a virtual environment (the
recommended way of installing non-system Python packages) -- the entire
ODK is already a "virtual environment".

Ultimately the right thing to do here would be for the OBO Dashboard
upstream to make proper releases, which we could then install at the
same time as any other Python packages.
The odk.py script has some issues when we run it under Python 3.12, we
fix them here.
Installing virtualenv in the builder image has nafarious consequences
when we try to later install Python packages.

That's because the Ubuntu package for virtualenv automatically installs
platformdirs version 2.5.1, which then prevents us from installing the
platformdirs 4.x that we need as a dependency for some of our packages.

The only reason we had virtualenv in the builder image was that we use
it to run the update-constraints workflow (in which we try to install
all our Python packages in a virtualenv). So here, we

1) remove virtualenv from the builder image;

2) amend the update-constraints.sh script to make it install virtualenv
   itself.
PIP emits a warning when we attempt to install executable scripts in a
location that is not in the system PATH.

In this instance the warning is not warranted. The scripts are installed
in a staging location ON PURPOSE -- they are later copied over to the
final ODK images, where the scripts will end up in the system PATH.

So we shut that warning down.
Get freshly frozen Python constraints for the newer version of Python
provided in Ubuntu 24.04.
@gouttegd gouttegd self-assigned this Jun 24, 2024
@gouttegd
Copy link
Contributor Author

(For those keeping track, this the same PR as #1067 modulo the wording of some commit messages and freshly updated Python constraints.)

@gouttegd gouttegd requested a review from matentzn June 24, 2024 16:46
@matentzn
Copy link
Contributor

matentzn commented Jul 29, 2024

OLD COMMENT PERTAINING TO setuptools INSTALL ERROR

I was almost done testing, when I encountered some issues with some incompatible python packages. Can you instruct me again how to update constraints.txt locally? Just running the update script locally causes errors like

Collecting matplotlib_venn (from -r requirements.txt.full (line 20))
  Using cached matplotlib-venn-1.0.0.tar.gz (34 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [18 lines of output]
      Traceback (most recent call last):
        File "/Users/matentzn/ws/ontology-development-kit/tmpdir/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/matentzn/ws/ontology-development-kit/tmpdir/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/matentzn/ws/ontology-development-kit/tmpdir/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/vj/ks1_0k8x3t9ftrwcr0t9vjwr0000gn/T/pip-build-env-1rlkx9mk/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 327, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/vj/ks1_0k8x3t9ftrwcr0t9vjwr0000gn/T/pip-build-env-1rlkx9mk/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 297, in _get_build_requires
          self.run_setup()
        File "/private/var/folders/vj/ks1_0k8x3t9ftrwcr0t9vjwr0000gn/T/pip-build-env-1rlkx9mk/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 313, in run_setup
          exec(code, locals())
        File "<string>", line 14, in <module>
      ModuleNotFoundError: No module named 'setuptools.command.test'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

And I am not sure if this is due to my local setup or something else. I also tried to do this from within the freshly build ODK container, but the error seems the same..

Dockerfile Show resolved Hide resolved
@gouttegd
Copy link
Contributor Author

Can you instruct me again how to update constraints.txt locally?

We don’t have a standard procedure for that when we update the base system. The normal procedure (update-constraints.sh) is not appropriate in this situation.

The idea is to create a Ubuntu-based Docker images with the same packages (Ubuntu packages, not Python packages) as in the builder image, then within that image try to install all the Python packages without the current constraints. Then freeze the resulting Python environment to produce a whole new constraints.txt file. Working on that now.

And I am not sure if this is due to my local setup or something else.

I reproduce the error here. From what I can tell, this is due to setuptools 72.0.0 (https://setuptools.pypa.io/en/stable/history.html#v72-0-0pip) which has removed a command that has been deprecated for something like 5 years, but that somehow Python package developers were (and are) still using. That’s the Python ecosystem for you.

Where to begin? It's just another Python screw-up.

Developers of setuptools have announced, back in 2017 (!) their
intention to remove the `test` command (found in
setuptools.command.test) [1].

Today, they thought that 7 years of advance deprecation warnings were
enough, and they published setuptools v72.0.0 which no longer provides
that command. [2]

But the Python ecosystem being what it is (a joke that has lasted long
enough), it turns out that 7 years was not actually enough, and there
are still many Python packages out there that didn't get the memo and
that are still dependent on setuptools.command.test (including some
packages that are used in the ODK).

There are several ways of fixing this:

a) Ditch Python altogether. This would be by far the best solution, but
one that is, alas, unlikely to happen any time soon.

b) Fix _all_ the packages used by the ODK (and their dependencies) so
that they do what they should have done in the past 7 years. This is the
correct solution, but one that will take time (7 years were already not
enough, so who knows how long it is going to take?).

c) Torture the setuptools developers until they agree to revert the
removal of the `test` command. I believe this may be illegal in many
jurisdictions.

d) Force the ODK build process to use a version of `setuptools` older
than 72.0.0. This is what we are doing here. Yes, this is a local
workaround and we should favour "upstream fixing" instead (solution b),
but if we want to be be able to build the ODK before 2031 (optimistic
estimation of the time it will take to get all the Python packages
fixed), we don't have much choice.

[1] pypa/setuptools#931
[2] https://setuptools.pypa.io/en/stable/history.html
The j2cli project (which provides the j2 tool) is no longer maintained
and is broken under Python 3.12, so we replace it with Jinjanator
(providing the similar command jinjanate).
Copy link
Contributor

@matentzn matentzn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After 10 rebuilds and dozens of tests and runs and pipeline upgrades, I am personally convinced everything is ok with this change. I want to note a tiny wrinkle which is a bit un-Damien like, which is coupling the python migration with the image-base migration.

As it is likely that we need 1.5.3 before 1.6, there is a small but not non-existent risk that some interactions between the old base image and the new python packages could cause some problems (unless of course, you would personally judge the base image migration as a "minor" change, since it does not affect the repo setup).

I know you have thought all of this through! Good to go, great work and thank you!

@gouttegd
Copy link
Contributor Author

I want to note a tiny wrinkle which is a bit un-Damien like, which is coupling the python migration with the image-base migration.

Not sure what is so “un-Damien like” in that. Python 3.12 is the version of Python that is provided with Ubuntu 24.04, while Ubuntu 22.04 (our previous base image) provided Python 3.10. The Python migration from 3.10 to 3.12 is merely a consequence of the base image migration, so of course it belongs to the same PR.

In other words, the base image and the Python version are already coupled by default. To change that, we’d need to actively uncouple them, which would be a completely different matter and would warrant a distinct PR just for that.

Uncoupling could be done by:

  • Installing Python 3.10 through third-party package repositories such as ppa:deadsnakes. Not keen on that (not keen on using any third-party repository as a general rule).
  • Compiling Python 3.10 from source. Not keen on that either. I’ve seen people compiling Python themselves and ending up exposing some subtle bugs in various packages because their Python was built in an environment that didn’t match the expectations of the core Python developers (for example, a bug in the subprocess module because Python was built on a system where /dev/shm was not mounted at compile time). I’d rather avoid doing that unless there is a very compelling reason to do so.
  • Keeping the Python 3.12 from the base system, but create a venv or virtualenv (or similar) environment where we install Python 3.10. AKA “why going through the hassle of managing ONE version of Python when we could be managing TWO versions instead?”

Overall, I believe uncoupling Python from the base system would be a needless complication. Having to migrate Python (which can include having to tweak the odk.py script or replacing some now obsolete packages such as j2cli) at the same time than we migrate the base system is but a minor hassle in comparison.

unless of course, you would personally judge the base image migration as a "minor" change, since it does not affect the repo setup

I’d rather have a 1.6 before we need to make a 1.5.3 release, so as to avoid the question altogether, but it we do need to make a 1.5.3 release, yes, I would consider the base image migration a minor change. It should not impact existing instances that uses the standard workflows. Instances that use custom workflows MAY be impacted (e.g. when a custom workflow is using j2), but custom workflows are not in the scope of the “minor releases don’t require updating or tweaking your repository” promise.

@matentzn
Copy link
Contributor

In other words, the base image and the Python version are already coupled by default.

I did not realise this, probably because I didnt read something carefully enough!

I’d rather have a 1.6 before we need to make a 1.5.3 release, so as to avoid the question altogether, but it we do need to make a 1.5.3 release, yes, I would consider the base image migration a minor change.

All good then!

Thanks for the clarifications!

@gouttegd gouttegd merged commit 94d7315 into master Jul 31, 2024
1 check passed
@gouttegd gouttegd deleted the update-base-system branch July 31, 2024 08:58
@gouttegd gouttegd mentioned this pull request Oct 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Upgrade the base system to 24.04
2 participants