Integrate with pip-compile for dependency pinning

[oprypin]

I know that there's no supported solution for lock files. But there are external tools for that -- I really like pip-compile.

https://github.com/jazzband/pip-tools#requirements-from-pyprojecttoml already mentions Hatch
-- they say you can already run pip-compile pyproject.toml, even including extras.

But my case isn't for the project itself, it's for one of the environments.
It'd be invalid to replace this with an extra, it's really just a separate thing that doesn't need to install the project itself:

[tool.hatch.envs.docs]
detached = true
dependencies = [
    "mkdocs >=1.1.2",
    # ...
]

Real example

But what I still can at least do is ask Hatch for the requirements that it wants for that environment:

hatch env show docs

Then I just need to pass that to pip-compile, great!

hatch env show --json | jq -r '.docs.dependencies | .[]' | pip-compile -U - -o docs/requirements.txt

Put that into a script

And it gives us a pinned requirements file!
Real example

And we can just install it with pip install --no-deps -r requirements.txt
Real example

---

For people passing by: maybe you'll find these snippets useful.

For developers of Hatch: could something be done to make this integration more straightforward?

[vikigenius]

Very interesting, I would be interested in something like this as well. I know it's a big deal and we should get the locking mechanism right instead of just rushing into it. But if this is simple enough, we can always deprecate this when we do find a better solution long term for dependency pinning

[maresb]

As an alternative to hatch env show --json | jq -r '.docs.dependencies | .[]', it is now possible to use hatch-requirements-txt to store the abstract dependencies in docs/requirements.in so that you can run pip-compile directly. That would look roughly (untested) like

[build-system]
requires = ["hatchling", "hatch-requirements-txt"]

[project]
dynamic = ["optional-dependencies"]

[tool.hatch.metadata.hooks.requirements_txt.optional-dependencies]
docs = ["docs/requirements.in"]

[maresb]

Actually pip-compile can read directly from pyproject.toml, so using hatch-requirements-txt is not necessary.

[oprypin]

Yes... my post says that and expands on it

[maresb]

Ah, of course, thanks. I got confused. While in general pip-compile can read project dependencies from pyproject.toml, it cannot currently read environment dependencies directly, so hatch-requirements-txt is useful.

And depending on the desired behavior, one could specify the file to Hatch as either requirements.txt to install locked dependencies with Hatch, or requirements.in to install unlocked dependencies with Hatch and locked dependencies with pip.

[bigjools]

Then I just need to pass that to pip-compile, great!

hatch env show --json | jq -r '.docs.dependencies | .[]' | pip-compile -U - -o docs/requirements.txt

Not sure if I am missing something, but you can do pip-compile --extra docs pyproject.toml -o docs/requirements.txt

[oprypin]

Your answer is for

[project.optional-dependencies]
docs = [

My question is about

[tool.hatch.envs.docs]
detached = true
dependencies = [

but yes, as a compromise, this can work, thanks

[antonymilne]

I'm also interested in this. Do you think there's scope for a plugin hatch-pip-tools like there is one for hatch-pip-deepfreeze?

[juftin]

I've got a working hatch plugin with pip-compile going over at https://github.com/juftin/hatch-pip-compile.

It's still early days so I'm very open to suggestions if anyone has any workflow recommendations. Hopefully it's helpful!

pip install hatch-pip-compile

[tool.hatch.envs.default]
type = "pip-compile"

[oprypin]

This is really really great stuff! It's such a non-intrusive solution too - if some project uses this, external contributors don't even need to install Hatch or pip-compile if they want to just get by and make a quick contribution.

And actually a lot of other cool plugins (which might be too dynamic on their own) can now be made and optionally rely on this plugin to make things more grounded and still reproducible.

• I wrote a post here - really inspired by this work: Environment collector plugin + environment plugin = ❤️ #1052