Sphinx does not work with docutils 0.21.1

[rikardn]

Describe the bug

When trying to use the latest version of docutils together with Sphinx 7.2.6 I get the following error:

The conflict is caused by:
    The user requested docutils==0.21.1
    sphinx 7.2.6 depends on docutils<0.21 and >=0.18.1

This is a repetition of #11414

The sdist on PyPI for docutils 0.21 was incomplete so the 0.21.1 is an important fix.

Perhaps again you could consider not capping the docutils version since it hinders downstream users to handle dependency issues like this. This stance is further explained in the following blog post (linking to the TL;DR): https://iscinumpy.dev/post/bound-version-constraints/#tldr

How to Reproduce

In requirements.txt:

docutils==0.21.1
sphinx==7.2.6

Environment Information

Not needed

Sphinx extensions

No response

Additional context

No response

[jayaddison]

Hi @rikardn - thanks for the suggestion; from my perspective I tend to think that Sphinx is following a reasonably good practice here.

I do agree that there are sometimes good reasons not to range-limit dependency versions, but the choice of whether to do so (either for setup-based dependency specs, and/or for package metadata requirements) can vary contextually by source project, and over time.

In the case of Sphinx, a fairly well-deployed project that wants to cautiously upgrade the core reStructuredText-processing docutils component, I think it's OK to have a version restriction that indicates more-or-less what the maintainers believe is well-supported and should work for users. None of us have infinite time, and constraining version ranges also has the effect of reducing the number of combinatorial possibilities for bug interactions, and from unexpected interactions from relatively-new code.

Even so: I do believe that early testing of dependencies is useful, especially from users who are familiar with projects. So I would encourage you to upgrade, but I would not encourage Sphinx to upgrade everyone's defaults until the project itself is comfortable to do so.

(security upgrades can be something of an exception to this rule, although even then I think there can be reasons to be careful, and sometimes to backport changes to prior versions)

To achieve your upgrade, I'd note that removing the docutils line entry from the requirements.txt file, and running both pip install -r requirements.txt and then pip -U docutils<0.22' would be one way to do that, albeit with an extra step. It is true that pip` will continue to warn you about the conflict, but the ecosystem doesn't prevent you from doing it - ultimately it is a set of source code files that are placed into a filesystem directory structure.

My understanding from reading a relevant pip issue thread at pypa/pip#8076 is that allowing relaxation of some dependency specifications is a feature-request that is under consideration, and that it would theoretically be possible in future for an additional file format to allow overrides of certain package constraints - but I'd tend to sympathise with the thinking that maintenance, bugreport and debugging overhead could arise if those override files became used (even if rarely, given sufficiently-complex cases).

Those kinds of overheads tend to apply ecosystem-wide - during learning about systems, during packaging and release, during maintenance, during development, bugreporting, testing, investigations, auditing, continuous integration, and so on - so even apparently-small improvements that seem to address a reasonable-sounding use-case can in fact cause problems that drag on an entire community. That's a frustration that I frequently find when trying to get small changes for things I care about introduced in other systems, and it can be annoying, but simplicity is extremely beneficial in software engineering. Dependency versioning and compatibility for code is, I think, a fairly intrinsically hard problem - perhaps the API-versioning side of it is mostly-tractable given strongly-enough typed languages, but runtime behaviour is also a consideration, and, to my knowledge, that's difficult to express or constrain using version numbers.

On a theoretical-possibilities note, I'd also theorize that if a future docutils upgrade did break Sphinx in some way -- maybe causing a bunch of frustration and time lost by people who install that, their userbase, and then Sphinx/docutils maintainers to investigate under some sense of pressure -- then a hypothetically-likely (in my mind) short-term remediation would be to reintroduce a version range constrant, and then we'd be back at the current status-quo.

So: please do upgrade if you're ready, but I'm fairly skeptical about whether Sphinx should remove the version range.

[chrisjsewell]

Very much agree with everything well outlined by @jayaddison, so its a hard no from me.

[rikardn]

Thanks @jayaddison for your thorough reply. I don't agree to your conclusions, but I respect that this is what the Sphinx project has decided, so I won't press the issue. Instead I would like to ask how downstream users could be helpful in these scenarios.

Can I assume that you would ideally like to bump the cap of the dependency as soon as possible after a new version of the dependency has been released? In that case what criteria needs to be fulfilled so that you would increase the cap (passed tests of a pull request for example)? Do you see any way a downstream maintainer, to scratch their itch, could help sphinx in the process of bumping the cap? I mean increase the cap in the development version so it gets into the next released version.

[jayaddison]

(quoting part 1-of-2)

Can I assume that you would ideally like to bump the cap of the dependency as soon as possible after a new version of the dependency has been released? In that case what criteria needs to be fulfilled so that you would increase the cap (passed tests of a pull request for example)?

A version increment for a dependency doesn't always make sense as-soon-as-possible, but it can be useful to stay up-to-date even when problems are encountered. The reason for that is that it's generally easiest to review small changes -- so it's worthwhile to update to keep the upgrade-diff / changelog small.

In an ideal situation I think an upgrade would occur after:

• At least one maintainer has inspected the source code changes between the existing dependency version and the upgraded version in detail, and is comfortable that they're acceptable. If they're concerned about something, that should be raised as a discussion/bugreport, and resolved first if it's considered serious enough.
• As you mention, unit tests have been run with the new version and confirm that all known/intended functionality remains correct (although note: in rare cases, an upgrade might uncover bugs in the testcases themselves).
• Non-functional testing (performance, energy consumption, user experience, developer experience) has taken place and the results indicate improvement, or at least continued acceptance, as determined by the project maintainers.
• The combined application (Sphinx in this case) including the updated dependency works correctly when it is deployed as an upgrade over an existing installation of the previous version -- so, even if we are only updating docutils, we shouldn't break existing deployments. This is tricky because we can't replicate every deployment out there (we don't even know of many of them), but testing some common known and anticipated deployment methods is feasible.

However, in practice a lot of that is wasteful on a per-change basis (especially per-commit, but potentially even per-dependency-update, per-minor-version or per-major version).

I think that good ecosystems try hard to determine when adjusting one item cannot possibly affect the relevant properties (test results, performance, ...) negatively, and initially rule those out. In practice computing is unfortunately full of edge cases that unexpectedly break people's assumptions about those "this shouldn't affect that" cases - and also, even ignoring the shouldn't-happen cases, there are a wide enough variety of legitimate change-x-affects-y cases that it's difficult for any individual to learn/remember them all.

So the eventual answer I'd respond with is that the goal is for the team of contributors (and that includes users, folks on mailing lists, etc) to collectively build enough confidence from inspection of the code and changes that generally things are OK -- without having to exhaustively test everything. And to report potential problems when they become aware of them, but preferably after validating them.

Eventually perhaps there's a programming language / environment where all of the ripple effects of an individual diff can unambiguously be extrapolated from it (within the codebase, to affected dependencies, to transitive dependencies, etc), at which point all of this hopefully becomes a lot easier - but that isn't Python (yet?) :)

(quoting part 2-of-2)

Do you see any way a downstream maintainer, to scratch their itch, could help sphinx in the process of bumping the cap? I mean increase the cap in the development version so it gets into the next released version.

Good question, and it makes me think that my advice to upgrade in your own project may have been too hasty. Since Sphinx is open source, what I'd probably recommend -- after deciding whether to do this online or offline and looking at #12095 -- would be to get a copy of the latest Sphinx code, upgrade docutils in that, and then convince yourself about whether it works OK for you (run the tests, build a project similar to your, test any other things you care about). And if that all looks ok, then upgrade your own project, and check the results of that. It would be nice if you could let us know, especially about any problems, but if it's open source then again in theory we could check that ourselves (not sure we have time to currently, but the idea is there).

That's a time cost on your side, and I'd understand if it's not an acceptable one to pay. Similarly, I don't inspect the diff of every upgrade each time I run apt upgrade on my computers. But I know that I could, and sometimes I do, and I expect that a few people tend to care about those individual packages, and certainly lots of people care about the resulting systems. In any case, I think projects can make themselves more maintainable by trying to make the upgrade process safe and seamless, if possible easier to review, and then those help to create a (hopefully good) reputation - but scrutiny of the changesets continues to be useful in any case (further opportunities for improvements, suggestions, etc).

Sidenote

Not exactly on-topic, but on the subject of transparency and review: it can have a tendency to make maintainers/contributors feel particularly bad when they make legitimate mistakes, and I don't think that's something that's useful for open source generally. It's important to catch the mistakes, but also important to help people to fix them (if they don't notice them themselves), and for that process not to be offputting. There's a large emotional effort cost to that, though, and so increasing the community size and/or moving automatable checks into continuous integration seem like some of the best practices I'm aware of to load-balance that.

[rikardn]

Thanks for elaborating. Your explanation is very much appreciated and useful, not only in a Sphinx perspective, but also for how one could think about dependencies in general. What I take away from this is how we could test new versions of dependencies and help Sphinx by reporting issues. As a note the project I am working on is apart from using Sphinx as a documentation building tool also using it as a direct dependency of the software for the dynamic generation of reports. So in this case Sphinx is not only a dependency for the developers of the package, but also for the users, which might be an uncommon case.

[AA-Turner]

Docutils 0.21 is now supported.