Add support for Python 3.10.

[icemac]

To actually support Python 3.10 we need to fix and release zopefoundation/RestrictedPython#214. The current version of RestrictedPython can only be installed on Python 3.10 because zc.buildout ignores python_requires.

There are some deprecation warnings (e. g. zopefoundation/ZODB#353) which could be fixed, too.
I did not yet create issues for all of them.

Open issues:

• Support Python 3.10 RestrictedPython#214
• Fix deprecation warnings
• Update to new releases

The only fixable deprecation warning is zopefoundation/ZODB#353 which is not critical, yet.
There are deprecation warnings in some C code, which are not critical yet:

• Uses deprecated API PyEval_CallObjectWithKeywords, raises warnings under 3.9 and 3.10 zope.proxy#43
• Uses deprecated API PyEval_CallObjectWithKeywords, raises warnings under 3.10 Acquisition#55
• Uses deprecated API, raises warnings under 3.10 AccessControl#118

[dataflake]

For the Zope package itself those changes are totally fine, I'm just concerned about RestrictedPython and possible security issues.

[loechel]

@dataflake RestrictedPython has a very security aware release process:

• python_requires are limited to supported / checked versions
• The implemented Node Transformer used in RestricedPython uses a allow-list approach, all new Language features are forbidden by default unless a specific visit_ method is added that handles or allow the feature.
• a regression test suite that checks all defined language features and shows if there are any changes / Problems with a new version

So there should not be any real security issues upfront. But maybe confusion by releases that only add partial support for a Python version. So it is more about communication than security.

[icemac]

I'd like to keep this PR unmerged until a final release of RestrictedPython supporting Python 3.10 as this is one of the requirements to actually support Python 3.10 for Zope. We should not add a halfhearted Python 3.10 support in a bugfix release of Zope but in a feature release with a bit amount of preparation especially for the dependencies.

[dataflake]

@icemac Do we really need code coverage checking on Windows? I suggest removing that, it seems a waste of GitHub Actions minutes.

[dataflake]

@icemac Do we really need code coverage checking on Windows? I suggest removing that, it seems a waste of GitHub Actions minutes.

And along with it I don't see any need to test building the docs on Windows or linting there. Both those will just duplicate the tests already run on Linux.

[dataflake]

The more I think about this PR the more I am bothered by a specific aspect, especially after the discussions about roadmaps and backwards compatibility: Please do not remove Python 3.6 support here.

Removing that support "feels" good because Python 3.6 is EOL soon, that is true. But that is throwing applications that use Zope 5 and Python 3.6 under the bus - for no good technical reason. Keep in mind that many Linux distributions still use Python 3.6.

With the recent roadmap discussions in mind I propose a new policy: When you start a release series with a given set of supported Python versions you're free to add to that set. That's a feature and won't break anyone's setup. But you must not take away from that set of supported versions. That is only allowed for the following major release.

@icemac @mauritsvanrees what do you think?

[d-maurer]

Jens Vagelpohl wrote at 2021-10-31 01:29 -0700:

Please do not remove Python 3.6 support here.

+1 In my opinion, you could stop testing for it. But you should not prevent installation with a corresponding `python_requires >= 3.7`.

[dataflake]

In my opinion, you could stop testing for it.

Please no half measures. All versions that we claim support for must be tested.

[d-maurer]

Jens Vagelpohl wrote at 2021-11-1 01:03 -0700:

> In my opinion, you could stop testing for it. Please no half measures. All versions that we claim support for must be tested

It is one thing not to claim support for a specific Python version and another thing to prevent installation for this version. I propose to handle the Python version analogous to the versions of other packages: document for which versions tests have been run successfully and let users use other versions at their own risk. To find out which Python (or Zope or Plone) versions are officially supported, I always look at the platform classifiers on PyPI. This might be a good way to document the officially supported Python versions.

[mauritsvanrees]

Zope 5 is used by Plone 6, which we only test on Python 3.7-3.9. So from a Plone standpoint it is no problem to drop support for 3.6.
But for other users, it would be nice to only drop it if it is really needed. And I agree, for clarity it would then be better to make this incompatibility clear by doing a new major Zope release.

Let me tell how we handled this in Plone 4:

• In Plone 4.0 (long time ago) we started with support for only Python 2.6.
• In Plone 4.2 we supported 2.6 and 2.7.
• Plone 4.3 started the same
• Around Plone 4.3.17 we stopped officially supporting Python 2.6. Biggest reason was that it was getting harder and harder to run all tests on Python 2.6.
• We tried to not actively break 2.6 support, as would for example happen when you use {} instead of {0} in string substitutions. If such breakage accidentally would have landed in a Plone 4.3 version, I would have considered this a bug, and released a new version with a fix.

Danger for Zope 5 is that more and more Python versions will need to be supported because we have no other reason yet to create a breaking Zope release, until one day we say it is enough.
Is that reason enough for creating a Zope 6? You can say yes. That is what Zope 5 did: drop support for Py 2.7, 3.5 and ZServer, so we have a new version.

I just checked Django:

• Django 3.0.0 and 3.1.0 have classifiers for 3.6-3.8
• Django 3.2.0 has 3.6-3.9
• Django 3.2.9 has 3.6-3.10
• Django 4.0b1 has 3.8-3.10

O, I could have checked the docs for that...

For Plone 6, I can see us removing support for older Python versions in a minor release. We want to do a minor release once a year (which is faster than in the past) so this means we support 3 or 4 Python releases at the same time.

I wonder if this idea might help: in tox.ini add a checker for both the oldest and newest Python version that does basic syntax checking and import checking, without yet running tests. Preferably the newest Python version would be picked automatically when a new one is released, so there is no need to explicitly run the meta package update on all repos.
The idea is then that if both checkers pass, there is at least a good chance that the package works with those versions, even if tests may need to be fixed for the newest version, which can then be tricky to keep running for the oldest version.

I am probably not very clear, you can go multiple ways with this answer...
Let me pick a side:

• Keep support for as long as we can, otherwise we quickly have three major Zope versions that we would need to support, at least when we want to keep current Plone versions supported.
• If a newer Python version requires breaking changes that cannot be solved with a try/except ImportError or similar easy fix: yes, create a new major release.

[dataflake]

This all is getting a bit out of hand and is getting way too complicated. All I am asking for is reverting those changes that drop Python 3.6, including the dropped tests, and that in the future dropping Python versions should only be done in a new major release. And that we stay consistent and test all Python versions we claim to support. We don't need any version recognition magic if we just continue running the tests.

[icemac]

I agree with @dataflake: If we support a version we should test it. Currently we use check-python-versions which expects that the versions allowed by python_requires align with the trove classifiers. So we either have to drop using check-python-versions or keep python_requires and classifiers in sync.

I have a problem with keeping Python 3.6 support until the release of Zope 6: There is no Zope 6 on the horizon and the Python core developers will make relatively sure that supporting newer Python versions will not require to drop support for older versions. (Currently we are even able to run Python 2.7 and 3.10 in the same code base (not for Zope but for all of its dependencies).)
I think we need to have a timeframe when to drop support for older Python versions: I could imagine to say 2 years after a Python version is no longer officially supported by the Python developers we start dropping support for it in the Zope ecosystem. This gives users enough time to migrate, especially because updating the Python version is usually no pain.
Using this approach we would not have to make a major release for dropping Python 3.6 support by the end of 2023 because it is really outdated by that date.

What do you think about this idea?

[dataflake]

@icemac That is a good compromise. So for the moment we'll return everything to the way it was before (trove classifiers, python_requires, test configurations) and I can add a note to the roadmap with a specific date.

IMHO it's perfectly fine to schedule a major release without big features in order to clean up the supported Python versions. We just need to signal very clearly how long the older Zope are supported so we have light at the end of the (support) tunnel for ourselves.

[dataflake]

I have changed this branch to omit any changes related to Python 3.6 and updated the wording at https://www.zope.org/developer/roadmap.html to reflect Michael's suggestion.

[icemac]

@dataflake I think this PR is ready now, do you agree in merging it?

[dataflake]

I agree, thanks for doing the work!

[icemac]

Thank you for reviewing this PR. 😃