Skip to content

Commit

Permalink
fix(batch-predictor, pyfunc-server): Pin setuptools max version (#608)
Browse files Browse the repository at this point in the history
# Description
It seems like from the 20th of September 2024, CICD tasks that require
building a python environment started
[failing](https://github.com/caraml-dev/merlin/actions/runs/10938668959/job/30411359538#step:7:2657)
with the following error when the ancient `mlflow` package tries to log
artifacts:
```
ModuleNotFoundError: No module named 'distutils._modified'
```

This seems to be an error
[reported](pyinstaller/pyinstaller#8793)
recently by other people in the `pyinstaller` repo, with some users
mentioning that `distutils` is used by `setuptools`, which just had a
new version released recently. Since it doesn't seem like we're
importing `distutils` directly, I've decided to simply pin down the max
version of `setuptools` to `<75`. Note that the version of `setuputils`
somehow needs to be specified in the variable passed to this argument
`install_requires` here


https://github.com/caraml-dev/merlin/blob/a9184ede2126d71c84f6458bcc44515dd78b0d03/python/pyfunc-server/setup.py#L49

As such, I've added `setuptools<75` in both the batch predictor and
pyfunc server's requirements.txt file.

Note that this PR technically does not fix the root cause of the
problem; it's only a band-aid solution for the moment until we migrate
to newer versions of Mlflow **_very shortly_**. When that happens I'll
revert the changes in this PR.

# Modifications
- `.github/workflows/release.yml` - Updated the versions of `setuptools`
in the build jobs for the python-related components
- `python/batch-predictor/requirements.txt` - Added `setuptools<75` to
the requirements of the batch predictor
- `python/pyfunc-server/requirements.txt` - Added `setuptools<75` to the
requirements of the pyfunc server

# Tests
<!-- Besides the existing / updated automated tests, what specific
scenarios should be tested? Consider the backward compatibility of the
changes, whether corner cases are covered, etc. Please describe the
tests and check the ones that have been completed. Eg:
- [x] Deploying new and existing standard models
- [ ] Deploying PyFunc models
-->

# Checklist
- [x] Added PR label
- [ ] Added unit test, integration, and/or e2e tests
- [ ] Tested locally
- [ ] Updated documentation
- [ ] Update Swagger spec if the PR introduce API changes
- [ ] Regenerated Golang and Python client if the PR introduces API
changes

# Release Notes
<!--
Does this PR introduce a user-facing change?
If no, just write "NONE" in the release-note block below.
If yes, a release note is required. Enter your extended release note in
the block below.
If the PR requires additional action from users switching to the new
release, include the string "action required".

For more information about release notes, see kubernetes' guide here:
http://git.k8s.io/community/contributors/guide/release-notes.md
-->

```release-note
NONE
```
  • Loading branch information
deadlycoconuts authored Sep 23, 2024
1 parent a9184ed commit 2cb5e3b
Show file tree
Hide file tree
Showing 3 changed files with 5 additions and 3 deletions.
6 changes: 3 additions & 3 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ jobs:
cache: pip
- name: Install dependencies
working-directory: ./python/sdk
run: pip install setuptools>=64 setuptools_scm>=8 twine wheel
run: pip install setuptools>=64,<75 setuptools_scm>=8 twine wheel
- name: Build and publish
env:
TWINE_USERNAME: ${{ secrets.pypi_username }}
Expand All @@ -53,7 +53,7 @@ jobs:
cache: pip
- name: Install dependencies
working-directory: ./python/pyfunc-server
run: pip install setuptools>=64 setuptools_scm>=8 twine wheel
run: pip install setuptools>=64,<75 setuptools_scm>=8 twine wheel
- name: Build and publish
env:
TWINE_USERNAME: ${{ secrets.pypi_username }}
Expand All @@ -78,7 +78,7 @@ jobs:
cache: pip
- name: Install dependencies
working-directory: ./python/batch-predictor
run: pip install setuptools>=64 setuptools_scm>=8 twine wheel
run: pip install setuptools>=64,<75 setuptools_scm>=8 twine wheel
- name: Build and publish
env:
TWINE_USERNAME: ${{ secrets.pypi_username }}
Expand Down
1 change: 1 addition & 0 deletions python/batch-predictor/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@ merlin-sdk==0.44.0
mlflow==1.26.1
pyarrow>=0.14.1,<11.0.0
pyspark==3.0.1
setuptools<75
1 change: 1 addition & 0 deletions python/pyfunc-server/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,6 @@ merlin-sdk==0.44.0
numpy>=1.8.2
orjson>=2.6.8
prometheus-client
setuptools<75
tornado
uvloop

0 comments on commit 2cb5e3b

Please sign in to comment.