docker: Add ability to build shared library flavours of CPython

[rcoup]

Add a build option to enable building shared/libpython-enabled Python. Helps a number of use-cases (eg: #1149 #793 #255 #30). In mine, using PyInstaller for creating a works-on-many-linux distribution.

Obviously as described in some of the above tickets, it doesn't fit the manylinux requirements and shouldn't be on by default. But adding the option helps people reuse the huge work put into manylinux images without maintaining forks or redoing it all.

Enable via:

MANYLINUX_BUILD_EXTRA="--build-arg PY_SHARED=1" \
PLATFORM=$(uname -m) POLICY=manylinux2014 COMMIT_SHA=latest \
./build.sh

If set, builds both static & shared versions of CPython interpreters into the images.

Shared versions end up in (eg) /opt/python/cp37-cp37m-shared/ alongside the existing static /opt/python/cp37-cp37m/, with a Python binary as /usr/local/bin/python3.7-shared.

[rcoup]

If people are happy enough I will update the README/etc too. And could even expose it as a slightly less hidden top-level PY_SHARED=1 option.

[auvipy]

looks good to me

[rcoup]

If people are happy enough I will update the README/etc too. And could even expose it as a slightly less hidden top-level PY_SHARED=1 option.

I've done that in the latest push.

[mayeut]

Thanks for taking the time to do this.
Before this can be merged, please do explain why you'd need a manylinux like image. The issues you mentioned provides another way to handle the PyInstaller/cx-freeze use-case with minimal effort. What use-case is not solved with a minimal "CentOS" (or other base image) image with python ?
I'd rather understand the added value for downstream repos before including bits of code that will never be tested here.

[mayeut]

Suggested change

	ln -s ${PREFIX} /opt/python/${ABI_TAG}${SUFFIX:-}
	ln -s ${PREFIX} /opt/python/${ABI_TAG}${SUFFIX}

[termim]

Before this can be merged, please do explain why you'd need a manylinux like image.

To build and test PyInstaller/cx-freeze based executables for every supported python version.

The issues you mentioned provides another way to handle the PyInstaller/cx-freeze use-case with minimal effort.

No, the issues mentioned basically say - use whatever python provided by the base image or build your own. And manylinux project itself proves that it is not a minimal effort.

What use-case is not solved with a minimal "CentOS" (or other base image) image with python ?

The use case is pretty obvious - base your work on top of well done manylinux project and avoid reinventing this wheel (properly build all supported python versions) over and over again.

I'd rather understand the added value for downstream repos before including bits of code that will never be tested here.

The added value for downstream repos is ability to use properly built all supported python versions with up to date supporting libraries like openssl etc. for building/testing executables from python scripts.

[mayeut]

To build and test PyInstaller/cx-freeze based executables for every supported python version.

@termim, can you please explain the use-case (or better, provide a real reference for the use-case) of building an application/executable with every supported python version ? This is where I'm lost. If you're building an application, only one python version should be enough ?

Not sure if it's been mentioned before but there's also https://github.com/indygreg/python-build-standalone/ python builds which are meant to be highly portable builds of python (manylinux just build without any care that the python builds themselves should be portable, only the built wheels are taken care of).

[termim]

To build and test PyInstaller/cx-freeze based executables for every supported python version.

@termim, can you please explain the use-case (or better, provide a real reference for the use-case) of building an application/executable with every supported python version ? This is where I'm lost. If you're building an application, only one python version should be enough ?

Example: my application uses Python-3.8 and I build an executable with PyInstaller-4.8 currently. Obviously I plan to update to more recent Python, PyInstaller and a bunch of other packages my application uses. Obviously I have to test it as an executable on different Linux distros as well. Obviously I need to properly build Python for that. Currently I have to maintain myself these builds. Manylinux project does exactly that except one little option - build shared libpython. This option does not need to be on by default or there could be 'shared' flavor of x86_64 images - in any case that would help a lot to those who use PyInstaller/cx-Freeze. That would be very natural extension to manylinux project IMHO.

Not sure if it's been mentioned before but there's also https://github.com/indygreg/python-build-standalone/ python builds which are meant to be highly portable builds of python (manylinux just build without any care that the python builds themselves should be portable, only the built wheels are taken care of).

There is a lot of duplication in this project and manylinux. IMHO manylinux does better job by explicitly addressing different glibc versions and providing well defined policies and I do not see why python built with it should not be as portable as the wheels.

[black-sliver]

👍 for /opt/python/cp39-cp39-shared.
My use-case would be that the project I want to bundle in an AppImage uses (and kind of depends on) cx_Freeze and is going to move to py3.9+ soon. I chose the AppImage path specifically because it would allow users to run it on older OS with older glibc that do not ship the latest and greatest python and building that that on a manylinux build would give me a defined glibc dependency that is equal to the wheel's.

[pkit]

Python community is notoriously sloppy with how glibc dependencies are handled.
This project is the only place where some sanity is preserved.
Not having libpython.so though is a major dealbreaker for people who need a stable python environment, with clearly defined glibc dependency.
"Do it yourself" is not a good idea, because in the end it will be yet another manylinux just less used/tested.
You can look into aws-cli for amount of hoops people need to jump.

[pkit]

@mayeut

This is where I'm lost. If you're building an application, only one python version should be enough ?

Mwahahaha. We are not in the fantasy world with rainbows and unicorns. We live in the harsh reality where some library update may require a new python version. Now what?