rez-pip with custom package.py

[beatreichenbach]

Hello,

I recently started transitioning a lot of our internal packages to pure python packages to be installed with rez-pip instead of rez-release. This allows us to automatically install all the pip dependencies using rez-pip and we don't have to double up the requirements in the pyproject.toml and the package.py file. Great!

The Issue

Some features of the package definition are now missing and I'm curious how others solve this.
For example, packages that rely on environment variables or even use some of the more advanced features in the package definition file now need another way to define those settings. How can environment variables be set?

def commands():
    env.MY_CUSTOM_VAR.set('{root}/path')
    if 'nuke' in resolve:
        env.NUKE_PATH.prepend('{root}/python/nuke')

Possible Workarounds

• Create a utility rez package and specify the python package as a dependency and specify the environment variables there.
• Add custom rez configuration to pyproject.toml and update rez-pip to read that information from there, and insert in

  rez/src/rez/pip.py

  Line 404 in c039654

  with make_package(name, packages_path, make_root=make_root) as pkg:

• Add package.py and update rez-pip to merge pyproject.toml with the custom package.py into a new package.py.
  I'd be happy to implement something like this if this makes any sense.

How are other people solving this? I would love to hear other workflows!

[vanridal]

I believe rez-pip is to be replaced with rez-pip2 https://github.com/JeanChristopheMorinPerso/rez-pip
It might have what you are looking for, though I think its not ready yet.

That being said, my preference would be not to use rez-pip for releasing your internal packages. If keeping your dependencies list in one location, you could try what I did and use a requirements.txt, which can be sourced by a pyproject.toml (if need be, as there is a mechanism for that) and then in your build package.py you can do this.

@early()
def requires():
    import os
    with open(os.path.join(os.getcwd(), 'requirements.txt'), 'r') as f:
        requires = [
            _to_rez_pkg(str(line.rstrip()))
            for line in f.readlines()
            if not line.startswith('#')
        ]
    return requires

def _to_rez_pkg(string):
    from rez.utils.pip import packaging_req_to_rez_req, normalize_requirement
    return packaging_req_to_rez_req(normalize_requirement(string)[0]).safe_str()

For installing the deps, you can create a pre build command with rez-pip to read in the requirments.txt

This idea would work only for building, as it relies on os.getcwd(), it wont work if you had some dev workflow where you used the packages git repo directly as a override to a released package.
Hence why I want this issues PR to be merged as it would solve the issue of accurately getting the package.py location #1842

[instinct-vfx]

We release pure-python packages through rez-pip, everything else as rez packages. For DCC integrations we usually split packages that are pure python for functionality and packages that need special treatment (e.g. to boostrap/integrate with DCCs or contain non-python parts like plugins.

[beatreichenbach]

@vanridal

That being said, my preference would be not to use rez-pip for releasing your internal packages. If keeping your dependencies list in one location, you could try what I did and use a requirements.txt, which can be sourced by a pyproject.toml (if need be, as there is a mechanism for that) and then in your build package.py you can do this.

I agree 100%. I would also love a configuration option to default python packages to a different location. In my case, any pip packages go to separate location, making organization just a bit easier. Knowing that any package in that directory is a package from pypi is quite nice.

I do like the general idea of your approach. Instead of using rez-pip use rez-release with the traditional package.py configuration. What if there was a configuration option for package.py like this:

# package.py

python_package = True

def commands() -> None:
    ....

If python_package is present, any other setting is optional and could override what rez-pip creates.

@JeanChristopheMorinPerso
This brings up a whole other issue that I have with package.py files, It doesn't make sense that package definition files can't be proper python. Is this a topic that is being discussed? Meaning something like:

#  package.py

from rez.configuration import BaseConfiguration

class Configuration(BaseConfiguration):
    name = 'nuke'
    version = '14.0.0'
    tools = ('nuke', 'nukex')
    ...
    
    def commands(self) -> None:
        super().commands()
        self.env.PATH.prepend(f'{self.root}/nuke/bin')

Or like this for a package using pyproject.toml:

#  package.py

from rez.configuration import PythonConfiguration

class Configuration(PythonConfiguration):
    def commands(self) -> None:
        super().commands()
        self.env.PATH.prepend(f'{self.root}/my_custom_bin')

[instinct-vfx]

I agree 100%. I would also love a configuration option to default python packages to a different location. In my case, any pip packages go to separate location, making organization just a bit easier. Knowing that any package in that directory is a package from pypi is quite nice.

Could you share a bit of info why that helps (i personally stopped caring altogeher where packages came from. We split by life cycle not source or package type. With packages being immutable by design i don't spend a lot of time in repositories directly)

@JeanChristopheMorinPerso This brings up a whole other issue that I have with package.py files, It doesn't make sense that package definition files can't be proper python. Is this a topic that is being discussed? Meaning something like:

package.py files are declarative in nature and their contents are translated to package attributes and for the commands() function are converted to shell code using rex. I have not dabbled with this part of rez a lot myself, maybe someone else from the TSC can share some more insight? I am not even sure this would work for many of the things rez currently does under the hood (e.g. caching, making sure packages are both self-contained and not rez-version dependent whenever possible etc.

[beatreichenbach]

Could you share a bit of info why that helps.

I'd say this is purely for organization/clean up purposes. When I took over the pipeline department at the current place, I needed a way to clean up the hundreds of installed packages because there was just too make space being used up to efficiently sync things. The nature of rez is that it is very easy to accumulate lots of data in the packages directory. Having that separation made it easier to run custom clean scripts that would remove unused packages.
If there was an easier way to clean up things I don't think there would be a reason to worry about that though.

package.py files are declarative in nature and their contents are translated to package attributes and for the commands() function are converted to shell code using rex.

I wasn't aware of rex. I guess the point still stands, if we want to use python files for the configuration, we should use classes or pydantic models so we get auto-complete and validation for the different fields. If we don't want to use python files, we should use something like yaml or json that can do schema validation in most ides (not sure about toml). But the current way the package.py is structured, it isn't very user friendly at all. For reference just see docker compose.yml or spack package.py files.

[JeanChristopheMorinPerso]

I think the way to solve your problems for rez-pip to add an attribute in the released package to mark it as "created by rez-pip". This is what rez-pip2 does. See https://rez-pip.readthedocs.io/en/stable/metadata.html#extra-metadata-added-by-rez-pip.

[beatreichenbach]

I think the way to solve your problems for rez-pip to add an attribute in the released package to mark it as "created by rez-pip".

Thank you, this would help separate pip isntall packages and rez installed packages. But this wouldn't allow me to set custom environment variables. For that I guess I will follow #1871 (comment).

I created a separate issue for the package definition topic I brought up: #1880

[JeanChristopheMorinPerso]

Indeed, it doesn't answer the initial question. Custom definitions could be handled via plugins maybe. There is a PR in rez-pip2 that adds a plugin system. It has a hook to modify the metadata (the package definition). See JeanChristopheMorinPerso/rez-pip#91. Would that work for you? I didn't put a lot of thought into your use case and would love to get feedback on the implementation!

[JeanChristopheMorinPerso]

FYI, you can see the plugins docs in a rendered state at https://rez-pip--91.org.readthedocs.build/en/91/plugins.html#metadata.

[instinct-vfx]

The current rez-pip also marks packages from pip. See

rez/src/rez/pip.py

Line 437 in c039654

pkg.from_pip = True

You also get an attribute to tell if a package is pure python.

Just for the sake of completeness.

In my mind rez-pip serves a clearly defined purpose: Turning pypi packages into rez packages with as little change and loss as possible. Injecting custom logic feels out of scope for me.