Q: how to boostrap/contribute to hatch?

[kloczek]

Looks like to build hatch it is necessary to have hatch.
Q: Q: how to boostrap hatch?

[ofek]

You need hatchling, not hatch. What are you trying to do?

[julie777]

I think my question is the same. I cloned the hatch repo. I want to be able to build hatch, run tests and contribute to development. As was previously mentioned, it looks like hatch is used to build hatch. Assuming that I have cloned the hatch git repo, what are the steps to build, test, and run the version of hatch that I cloned?

[ofek]

Ah, I understand! All you'd have to do is pip install -e . (with a recent version of pip) then hatch run full to run the tests. If that works for you I'll add docs.

[kloczek]

Issue is broader.
Using pep517 it is possible to build hatch. Tha
however on install it four other modules are missing: dom-toml>= 0.4, domdf-python-tools >= 2.8, consolekit >= 1.2.2 and shippinglabel >= 0.16.
All those moduels needs hatch on build.

[kloczek]

install time dependencies present in module metadata. From whey-0.0.17.dist-info:

Metadata-Version: 2.1
Name: whey
Version: 0.0.17
Summary: A simple Python wheel builder for simple projects.
Home-page: https://github.com/repo-helper/whey
Author: Dominic Davis-Foster
Author-email: [email protected]
License: MIT License
Project-URL: Documentation, https://whey.readthedocs.io/en/latest
Project-URL: Issue Tracker, https://github.com/repo-helper/whey/issues
Project-URL: Source Code, https://github.com/repo-helper/whey
Keywords: pep517,pep621,build,sdist,wheel,packaging,distribution
Platform: Windows
Platform: macOS
Platform: Linux
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Topic :: Software Development :: Build Tools
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Archiving :: Packaging
Classifier: Typing :: Typed
Requires-Python: >=3.6.1
Description-Content-Type: text/x-rst
License-File: LICENSE
Requires-Dist: click (>=7.1.2)
Requires-Dist: consolekit (>=1.2.2)
Requires-Dist: dom-toml (>=0.4.0)
Requires-Dist: domdf-python-tools (>=2.8.0)
Requires-Dist: first (>=2.0.2)
Requires-Dist: natsort (>=7.1.1)
Requires-Dist: packaging (>=20.9)
Requires-Dist: pyproject-parser (>=0.3.0)
Requires-Dist: shippinglabel (>=0.16.0)
Provides-Extra: all
Requires-Dist: docutils (==0.16) ; extra == 'all'
Requires-Dist: pyproject-parser[readme] (>=0.3.0) ; extra == 'all'
Provides-Extra: readme
Requires-Dist: docutils (==0.16) ; extra == 'readme'
Requires-Dist: pyproject-parser[readme] (>=0.3.0) ; extra == 'readme'

[ofek]

I'm sorry, what exactly is the issue?

[kloczek]

Circular dependencies which does not allow build whey and its runtime or build dependencies from scratch.

[ofek]

It doesn't look like they depend on hatchling for builds nor does it use any of those. Are you sure?

[ofek]

Do you look into this further?

[kloczek]

It doesn't look like they depend on hatchling for builds nor does it use any of those. Are you sure?

Again:

• hatchling is OK
• grnerated module depends on:

Requires-Dist: click (>=7.1.2)
Requires-Dist: consolekit (>=1.2.2)
Requires-Dist: dom-toml (>=0.4.0)
Requires-Dist: domdf-python-tools (>=2.8.0)
Requires-Dist: first (>=2.0.2)
Requires-Dist: natsort (>=7.1.1)
Requires-Dist: packaging (>=20.9)
Requires-Dist: pyproject-parser (>=0.3.0)
Requires-Dist: shippinglabel (>=0.16.0)
Provides-Extra: all
Requires-Dist: docutils (==0.16) ; extra == 'all'
Requires-Dist: pyproject-parser[readme] (>=0.3.0) ; extra == 'all'
Provides-Extra: readme
Requires-Dist: docutils (==0.16) ; extra == 'readme'
Requires-Dist: pyproject-parser[readme] (>=0.3.0) ; extra == 'readme

• to build dom-toml, domdf-python-tools, onsolekit and shippinglabel you need to have installed hatchling which is not possible to install because those modules cannot be build because hatchling needs them on install/run time.

[ofek]

Please show repro case

[kloczek]

This issue has nothong to do with any repo.
This issue is about module BUILD proceses order.

Did you at all one time tried to build hatchling without any external resources by generate and hatchling` .whl archive and install it using pip without using publicy available pypi repo?
Please try do that so you will see where is the problem,

[ofek]

Where did that generated metadata come from? None of those packages have anything to do with hatchling. Please show me how to reproduce this

[kloczek]

Where did that generated metadata come from? None of those packages have anything to do with hatchling. Please show me how to reproduce this

How many times do I need to repeat tat those dependencies are not install time buut BUILD TIME?

[ofek]

@musicinmybrain Can you please take a look at this? I've no idea what the issue is here nor how to reproduce.

[musicinmybrain]

I’m not sure exactly what’s going on here, either.

In Fedora Linux, I build hatchling and run the tests using the dependencies specified in:

https://github.com/ofek/hatch/blob/dee98e4a759cac41f0101d681f9025dda1b214da/backend/src/hatchling/ouroboros.py#L34-L44

The resulting spec file is: https://src.fedoraproject.org/rpms/python-hatchling/blob/3e6dd0ab17f3a2aa9fd2542de71a9d762dbe5205/f/python-hatchling.spec

If any of editables, packaging, pathspec, pluggy, or tomli, or any of their recursive build dependencies, required hatchling to build, then we would have a bootstrapping loop that we would have to find a way to break. That’s not currently the case, at least for the versions packaged in Fedora. I’m guessing that most of these dependencies are required only for hatchling’s tests (i.e., at runtime rather than build time), so skipping the tests in the first build would probably suffice to break any bootstrapping loop that did appear in the future. I haven’t managed to run the hatchling tests in the offline build environment anyway.

While hatch 1.0.0 isn’t released yet, I didn’t encounter any bootstrapping loops when packaging it, either.

---

Which of whey’s recursive dependencies actually uses hatch or hatchling? I spot-checked and didn’t see any mention of either in the pyproject.toml files for dom-toml, domdf-python-tools, consolekit, or shippinglabel.

[kloczek]

In Fedora Linux, I build hatchling and run the tests using the dependencies specified in:

Since first release in Fedora pep517 builds have been introduced so at that time it was no such loopy dependencies, I'm talking exactly about building hatchling module using that procesure,
Additionally Fedora %pyproject_wheel macro does not builds modules with isiolation which means that during builds missing .whl archives are downloaded from public pypi repository https://src.fedoraproject.org/rpms/pyproject-rpm-macros/blob/rawhide/f/macros.pyproject#_28, and in that macto is used --no-dep switch.

In other words in Fedora there is no those loops in builds however I'm talking about using only locally avbalaible rwsources.

I'm talking about build order of the procedure few modules wgci hava now loopy dependencies (those loops are streaching across install and build time).

[musicinmybrain]

Additionally Fedora %pyproject_wheel macro does not builds modules with isiolation which means that during builds missing .whl archives are downloaded from public pypi repository https://src.fedoraproject.org/rpms/pyproject-rpm-macros/blob/rawhide/f/macros.pyproject#_28, and in that macto is used --no-dep switch.

Fedora RPM builds are done in an offline sandbox with no network access. Dependencies come from other RPM packages, and everything is built from source by both policy and practice. If the BuildRequires are inadequate then pip may try to download wheels, but it will never succeed; the RPM build will fail instead.

Any bootstrapping loops have to be dealt with at least every time Fedora introduces a new CPython interpreter version. Often this just means skipping tests or not building documentation in certain packages the first time around. It’s possible that there is a bootstrapping loop involving hatchling that I haven’t had to deal with yet, either because some packages in Fedora aren’t updated to the latest versions or because we haven’t been through a Python version update since I packaged python-hatchling. I don’t know how to guess what packages might be involved, though.

I think it would help explain what you are talking about if you could write down a dependency loop in the following form:

hatchling 0.22.0 → foo 1.2.3 → bar 3.4.5 → hatchling

Or, post a series of shell commands that demonstrate the dependency loop.

That way someone could take a look at exactly what is happening and why, and perhaps suggest a way to bootstrap successfully.

[kloczek]

Fedora RPM builds are done in an offline sandbox with no network access. Dependencies come from other RPM packages, and everything is built from source by both policy and practice. If the BuildRequires are inadequate then pip may try to download wheels, but it will never succeed; the RPM build will fail instead.

Just checed one more time. In case fedora modules all those modules are noparch and do not depend on python ABI versiom.
In otrher words even in case of moving to new major python version are downlaoded alradery build modules from prev major python version.

[musicinmybrain]

No, the Python version with which the RPM was built becomes part of its (RPM) dependency metadata, including for noarch packages. Introducing a new Python 3.x version means bootstrapping the whole ecosystem of Python packages. For example:

$ mock -r fedora-rawhide-x86_64 --dnf-cmd -- repoquery --requires python3-hatchling-0.22.0-1.fc37
[…]
/usr/bin/python3
python(abi) = 3.10
python3.10dist(editables) >= 0.2
python3.10dist(packaging) >= 21.3
python3.10dist(pathspec) >= 0.9
python3.10dist(pluggy) >= 1
python3.10dist(tomli) >= 1.2.2

Sometimes this does reveal previously-unnoticed dependency loops, especially in recently-introduced packages. They can usually be broken by introducing a build conditional somewhere.

[kloczek]

So how all those packages has beem build if all above packages have python3-hatchling in BuildRequires?
If you will look closer for example python-editables.spec and python-hatchling.spec have no any bconds.

[musicinmybrain]

The only package in Fedora that currently requires hatchling is python-userpath. Soon hatch will require it too. Everything else uses a different build backend: setuptools, flit_core, poetry-core, etc.

[musicinmybrain]

For example, building the python-editables source RPM requires:

pyproject-rpm-macros
python3-devel
python3dist(packaging)
python3dist(pip) >= 19
python3dist(pytest)
python3dist(setuptools)
python3dist(toml)
python3dist(wheel)