Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rename and update tox environments #1168

Merged
merged 4 commits into from
Jul 1, 2023
Merged

Rename and update tox environments #1168

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f9fb4ce
Remove python prefix from tox environment names
oliverholworthy Jun 30, 2023
4aa4cfd
Move Merlin dependencies to deps configuation of tox environment
oliverholworthy Jun 30, 2023
71f96c7
Use posargs for tests path in tox environments
oliverholworthy Jun 30, 2023
f41eae6
Merge branch 'main' into tox-env-rename-py
oliverholworthy Jun 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/cpu-horovod.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,4 +72,4 @@ jobs:
if [[ "${{ github.ref }}" != 'refs/heads/main' ]]; then
extra_pytest_markers="and changed"
fi
EXTRA_PYTEST_MARKERS="$extra_pytest_markers" MERLIN_BRANCH="$merlin_branch" COMPARE_BRANCH=${{ github.base_ref }} tox -e py38-horovod-cpu
EXTRA_PYTEST_MARKERS="$extra_pytest_markers" MERLIN_BRANCH="$merlin_branch" COMPARE_BRANCH=${{ github.base_ref }} tox -e horovod-cpu
2 changes: 1 addition & 1 deletion .github/workflows/cpu-nvtabular.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,4 +64,4 @@ jobs:
- name: Run tests
run: |
merlin_branch="${{ steps.get-branch-name.outputs.branch }}"
MERLIN_BRANCH="$merlin_branch" GIT_COMMIT=$(git rev-parse HEAD) tox -e py38-nvtabular-cpu
MERLIN_BRANCH="$merlin_branch" GIT_COMMIT=$(git rev-parse HEAD) tox -e nvtabular-cpu
2 changes: 1 addition & 1 deletion .github/workflows/cpu-systems.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,4 +64,4 @@ jobs:
- name: Run tests
run: |
merlin_branch="${{ steps.get-branch-name.outputs.branch }}"
MERLIN_BRANCH="$merlin_branch" GIT_COMMIT=$(git rev-parse HEAD) tox -e py38-systems-cpu
MERLIN_BRANCH="$merlin_branch" GIT_COMMIT=$(git rev-parse HEAD) tox -e systems-cpu
2 changes: 1 addition & 1 deletion .github/workflows/cpu-t4r.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,4 +60,4 @@ jobs:
- name: Run tests
run: |
merlin_branch="${{ steps.get-branch-name.outputs.branch }}"
MERLIN_BRANCH="$merlin_branch" GIT_COMMIT=$(git rev-parse HEAD) tox -e py38-transformers4rec-cpu
MERLIN_BRANCH="$merlin_branch" GIT_COMMIT=$(git rev-parse HEAD) tox -e transformers4rec-cpu
2 changes: 1 addition & 1 deletion .github/workflows/gpu-multi.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,4 +56,4 @@ jobs:
if [[ "${{ github.ref }}" != 'refs/heads/main' ]]; then
extra_pytest_markers="and changed"
fi
cd ${{ github.workspace }}; EXTRA_PYTEST_MARKERS=$extra_pytest_markers MERLIN_BRANCH=$branch COMPARE_BRANCH=${{ github.base_ref }} tox -e py38-multi-gpu
cd ${{ github.workspace }}; EXTRA_PYTEST_MARKERS=$extra_pytest_markers MERLIN_BRANCH=$branch COMPARE_BRANCH=${{ github.base_ref }} tox -e multi-gpu
4 changes: 2 additions & 2 deletions .github/workflows/gpu.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ jobs:
if [[ "${{ github.ref }}" != 'refs/heads/main' ]]; then
extra_pytest_markers="and changed"
fi
cd ${{ github.workspace }}; PYTEST_MARKERS="unit and not (examples or integration or notebook) $extra_pytest_markers" MERLIN_BRANCH=$branch COMPARE_BRANCH=${{ github.base_ref }} tox -e py310-gpu
cd ${{ github.workspace }}; PYTEST_MARKERS="unit and not (examples or integration or notebook) $extra_pytest_markers" MERLIN_BRANCH=$branch COMPARE_BRANCH=${{ github.base_ref }} tox -e gpu
tests-examples:
runs-on: 1GPU
Expand All @@ -55,4 +55,4 @@ jobs:
if [[ "${{ github.ref }}" != 'refs/heads/main' ]]; then
extra_pytest_markers="and changed"
fi
cd ${{ github.workspace }}; PYTEST_MARKERS="(examples or notebook) $extra_pytest_markers" MERLIN_BRANCH=$branch COMPARE_BRANCH=${{ github.base_ref }} tox -e py310-gpu
cd ${{ github.workspace }}; PYTEST_MARKERS="(examples or notebook) $extra_pytest_markers" MERLIN_BRANCH=$branch COMPARE_BRANCH=${{ github.base_ref }} tox -e gpu
57 changes: 30 additions & 27 deletions tox.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,33 +2,34 @@
; .github/workflows/cpu-ci.yml for the workflow definition.

[tox]
envlist = py310-gpu,py310-multi-gpu
envlist = gpu,multi-gpu,horovod-cpu,nvtabular-cpu,systems-cpu,transformers4rec-cpu,docs,docs-multi

[testenv]
commands =
pip install --upgrade pip
pip install -e .[all]

[testenv:py310-gpu]
[testenv:gpu]
; Runs in: Github Actions
; Runs GPU-based tests.
allowlist_externals =
bash
deps =
--no-deps -rrequirements/test.txt
-rrequirements/test.txt
git+https://github.com/NVIDIA-Merlin/core.git@{env:MERLIN_BRANCH}
git+https://github.com/NVIDIA-Merlin/dataloader.git@{env:MERLIN_BRANCH}
git+https://github.com/NVIDIA-Merlin/NVTabular.git@{env:MERLIN_BRANCH}
git+https://github.com/NVIDIA-Merlin/systems.git@{env:MERLIN_BRANCH}
passenv =
OPAL_PREFIX
setenv =
TF_GPU_ALLOCATOR=cuda_malloc_async
sitepackages=true
commands =
python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git@{env:MERLIN_BRANCH:main}
python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/dataloader.git@{env:MERLIN_BRANCH:main}
python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/nvtabular.git@{env:MERLIN_BRANCH:main}
python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git@{env:MERLIN_BRANCH:main}
bash -c 'python -m pytest --cov-report term --cov merlin -m "{env:PYTEST_MARKERS}" -rxs tests/ || ([ $? = 5 ] && exit 0 || exit $?)'
bash -c 'python -m pytest --cov-report term --cov merlin -m "{env:PYTEST_MARKERS}" -rxs {posargs:tests} || ([ $? = 5 ] && exit 0 || exit $?)'


[testenv:py310-multi-gpu]
[testenv:multi-gpu]
; Runs in: Github Actions
; Runs GPU-based tests.
allowlist_externals =
Expand All @@ -43,30 +44,32 @@ setenv =
LD_LIBRARY_PATH=${envdir}/hugectr/include/lib{:}/usr/local/lib/python3.10/dist-packages/tensorflow{:}{env:LD_LIBRARY_PATH}
LIBRARY_PATH=${envdir}/hugectr/lib{:}{env:LIBRARY_PATH}
sitepackages=true
deps =
git+https://github.com/NVIDIA-Merlin/core.git@{env:MERLIN_BRANCH}
git+https://github.com/NVIDIA-Merlin/dataloader.git@{env:MERLIN_BRANCH}
git+https://github.com/NVIDIA-Merlin/NVTabular.git@{env:MERLIN_BRANCH}
commands =
python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git@{env:MERLIN_BRANCH:main}
python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/dataloader.git@{env:MERLIN_BRANCH:main}
python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/nvtabular.git@{env:MERLIN_BRANCH:main}
sh examples/usecases/multi-gpu/install_sparse_operation_kit.sh {envdir}
bash -c 'horovodrun -np 2 sh examples/usecases/multi-gpu/hvd_wrapper.sh python -m pytest -m "horovod {env:EXTRA_PYTEST_MARKERS}" -rxs tests/unit || ([ $? = 5 ] && exit 0 || exit $?)'
bash -c 'horovodrun -np 2 sh examples/usecases/multi-gpu/hvd_wrapper.sh python -m pytest -m "unit and horovod {env:EXTRA_PYTEST_MARKERS}" -rxs {posargs:tests} || ([ $? = 5 ] && exit 0 || exit $?)'

[testenv:py310-horovod-cpu]
[testenv:horovod-cpu]
setenv =
HOROVOD_WITH_MPI=1
HOROVOD_WITH_TENSORFLOW=1
PATH={env:PATH}{:}{envdir}/env/bin
LD_LIBRARY_PATH={env:LD_LIBRARY_PATH}{:}{envdir}/env/lib
deps =
git+https://github.com/NVIDIA-Merlin/core.git@{env:MERLIN_BRANCH}
git+https://github.com/NVIDIA-Merlin/dataloader.git@{env:MERLIN_BRANCH}
git+https://github.com/NVIDIA-Merlin/NVTabular.git@{env:MERLIN_BRANCH}
commands =
conda update --yes --name base --channel defaults conda
conda env create --prefix {envdir}/env --file requirements/horovod-cpu-environment.yml --force
{envdir}/env/bin/python -m pip install 'horovod==0.27.0' --no-cache-dir
{envdir}/env/bin/horovodrun --check-build
{envdir}/env/bin/python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git@{env:MERLIN_BRANCH:main}
{envdir}/env/bin/python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/dataloader.git@{env:MERLIN_BRANCH:main}
{envdir}/env/bin/python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/nvtabular.git@{env:MERLIN_BRANCH:main}
{envdir}/env/bin/horovodrun -np 2 sh examples/usecases/multi-gpu/hvd_wrapper.sh pytest -m "horovod {env:EXTRA_PYTEST_MARKERS}" -rxs tests/unit
{envdir}/env/bin/horovodrun -np 2 sh examples/usecases/multi-gpu/hvd_wrapper.sh pytest -m "unit and horovod {env:EXTRA_PYTEST_MARKERS}" -rxs {posargs:tests}

[testenv:py310-nvtabular-cpu]
[testenv:nvtabular-cpu]
passenv=GIT_COMMIT
allowlist_externals = git
deps =
Expand All @@ -82,7 +85,7 @@ commands =
python -m pip install .
python -m pytest nvtabular-{env:GIT_COMMIT}/tests/unit

[testenv:py310-systems-cpu]
[testenv:systems-cpu]
passenv=GIT_COMMIT
allowlist_externals = git
deps =
Expand All @@ -99,7 +102,7 @@ commands =
python -m pip install .
python -m pytest -m "not notebook" systems-{env:GIT_COMMIT}/tests/unit

[testenv:py310-transformers4rec-cpu]
[testenv:transformers4rec-cpu]
passenv=GIT_COMMIT
allowlist_externals = git
commands =
Expand All @@ -120,10 +123,10 @@ changedir = {toxinidir}
deps =
-rrequirements/docs.txt
-rrequirements/test.txt
git+https://github.com/NVIDIA-Merlin/core.git
git+https://github.com/NVIDIA-Merlin/dataloader.git
git+https://github.com/NVIDIA-Merlin/NVTabular.git
commands =
python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/dataloader.git
python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/nvtabular.git
python -m sphinx.cmd.build -E -P -b html docs/source docs/build/html

[testenv:docs-multi]
Expand All @@ -132,9 +135,9 @@ changedir = {toxinidir}
deps =
-rrequirements/docs.txt
-rrequirements/test.txt
git+https://github.com/NVIDIA-Merlin/core.git
git+https://github.com/NVIDIA-Merlin/dataloader.git
git+https://github.com/NVIDIA-Merlin/NVTabular.git
commands =
python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/dataloader.git
python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/nvtabular.git
sphinx-multiversion --dump-metadata docs/source docs/build/html | jq "keys"
sphinx-multiversion docs/source docs/build/html