Adding unionml to conda-forge channel

[sugatoray]

Premise

Anaconda (or conda) is used for package and environment management in Python. One of the popular public channels on conda is conda-forge. The packages maintained on this channel can be installed as:

conda install -c conda-forge <package-name>

With unionml on conda-forge, installation of unionml becomes very easy with a package manager, inside an isolated python environment.

Objective

I am the maintainer of unionml on conda-forge. So far, I have added close to over 40 python libraries to conda-forge, and it has been a humbling experience. In total, I maintain over 50 PyPI libraries on conda-forge channel currently.

I will try and explain a few things about conda-forge packaging here, to make the process as clear as possible to any current or future contributor or user of this repository.

Relevant Information

UnionML was added to conda-forge in the following PR.

• Adding unionml from PyPI conda-forge/staged-recipes#19349

And here is the corresponding issue:

• [FEATURE] Add conda installation option for unionml #128

I am also one of the co-maintainers of the following Flyte ecosystem packages on conda-forge channel.

• flytekit
• flyteidl

[sugatoray]

A common byproduct of conda-forge packaging

Often the activity of adding a PyPI package to conda-forge reveals common packaging errors/mistakes. For instance, the following PR was necessary in unionml to fix not including requirements.txt among others.

• Update source packaging #125

[sugatoray]

FAQs

I will present various concepts as frequently asked questions (FAQs) below.

[sugatoray]

FAQ-01: What is a feedstock and where is the conda-forge package maintained at?

The recipe for each conda-forge package is maintained in a GitHub repository, typically referred to as a feedstock. This is the general structure of a feedstock URL for a package on conda-forge.

• https://github.com/conda-forge/<package>-feedstock

For example,

• unionml: https://github.com/conda-forge/unionml-feedstock
• flytekit: https://github.com/conda-forge/flytekit-feedstock
• flyteidl: https://github.com/conda-forge/flyteidl-feedstock

[sugatoray]

FAQ-02: What was necessary for adding unionml to conda-forge channel?

Any dependency of unionml either has to be already present on conda-forge, or one must add them to conda-forge. Some of the dependencies of unionml were either not available on conda-forge (mangum) or they were serving old pypi versions (flytekit, flyteidl and grpcio-status).

💡 grpcio-status and flyteidl are dependencies of flytekit

Thus, several prerequisites were necessary to take care of before this PR (conda-forge/staged-recipes#19349) could be possible.

• Add dependency mangum to conda-forge: Adding mangum from PyPI conda-forge/staged-recipes#19362

• Update dependency versions on conda-forge for a few unionml dependencies:

  • Update flytekit from v0.26.1 to v1.0.5: Update to v1.0.5 conda-forge/flytekit-feedstock#12
  • Update flyteidl from v0.22.0 to v1.1.6: Update to v1.1.6 conda-forge/flyteidl-feedstock#12
  • Update grpcio-status from v1.41.1 to v1.46.3: Update to v1.46.3 conda-forge/grpcio-status-feedstock#7

---

🔥 Fixing unionml packaging problem

The PyPI source of unionml does not have any requirements.txt file at the moment. This causes problems with conda-forge packaging while installing from the source (.tar.gz file). I have temporarily circumvented this by dynamically creating the requirements.txt file. But this should be fixed at the source here.

• Issue: [BUG] Include requirements.txt in PyPI source #126
• PR (to fix this issue): Update source packaging #125

📄 Source: #128 (comment)

[sugatoray]

FAQ-03: How do I know which PyPI packages are available on conda-forge?

👉 Go to https://conda-forge.org/feedstock-outputs/ and search for PyPI package name.

Keep in mind that conda-forge package names could be slightly different than their PyPI package names. Mostly, in my experience, you could see variations in hyphens (-) or underscores (_).

[sugatoray]

FAQ-04: What if a PyPI package is not available on conda-forge?

👉 Someone will have to add the PyPI package as a conda-forge recipe by sending a pull request (PR) at: https://github.com/conda-forge/staged-recipes/.

[sugatoray]

FAQ-05: How do you update a PyPI package on conda-forge, typically after a new PyPI release?

You will need to send a PR to the respective feedstock repository to update the recipe for that package.

For instance, say, you want to update unionml to version 1.2.3, here are the steps:

1. Go to unionml-feedstock: https://github.com/conda-forge/unionml-feedstock
2. Fork the repository.
3. Create a branch with a meaningful name: update_v1.2.3.
4. Update the recipe in that branch according conda-forge recipe making guidelines.
5. Send a PR to the feedstock.

   🔥 Follow the instructions carefully while raising the PR.

6. Conda-Forge CI's will run and if no build error is thrown, let the maintainer(s) know of the PR.
7. Finally, after review process, if everything looks good, the maintainer(s) will merge the PR.