Set up reproducible Python environments with conda-lock #2968
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,7 +4,7 @@ name: bot-auto-merge | |
| on: | ||
| workflow_run: | ||
| types: [completed] | ||
| workflows: ["pytest"] | ||
|
|
||
| jobs: | ||
| bot-auto-merge: | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| --- | ||
| name: update-conda-lockfile | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| schedule: | ||
| - cron: "0 9 * * 1-5" # Weekdays at 9AM UTC | ||
| push: | ||
| paths: | ||
| - "pyproject.toml" | ||
| - "environments/*" | ||
| - ".github/workflows/update-conda-lockfile.yml" | ||
|
|
||
| # What branch does this action run on? | ||
| # - workflow_dispatch: Whatever branch it was run against. | ||
| # - schedule: Always the same branch (will be dev or main) | ||
| # - push: Base branch of the PR. | ||
|
|
||
| jobs: | ||
| update-conda-lockfile: | ||
| runs-on: ubuntu-latest | ||
| if: ${{ (github.event_name == 'push' && github.actor != 'pudlbot') || (github.event_name == 'schedule' && github.repository == 'catalyst-cooperative/pudl') || (github.event_name == 'workflow_dispatch') }} | ||
| defaults: | ||
| run: | ||
| shell: bash -l {0} | ||
| steps: | ||
| - name: Set GITHUB_REF for use with workflow_dispatch | ||
| if: ${{ (github.event_name == 'workflow_dispatch') }} | ||
| run: | | ||
| echo "GITHUB_REF="${{ github.ref_name }} >> $GITHUB_ENV | ||
| - name: Set GITHUB_REF for use with schedule | ||
| if: ${{ (github.event_name == 'schedule') }} | ||
| run: | | ||
| echo "GITHUB_REF=dev" >> $GITHUB_ENV | ||
| - name: Set GITHUB_REF for use with push | ||
| if: ${{ (github.event_name == 'push') }} | ||
| run: | | ||
| echo "GITHUB_REF="${{ github.ref_name }} >> $GITHUB_ENV | ||
|
Comment on lines
+36
to
+38
There was a problem hiding this comment. It looks like you're setting There was a problem hiding this comment. They were different at some point while I was developing the setup, and it felt more legible to enumerate the 3 cases we're trying to cover. But I could go either way. There was a problem hiding this comment. Ok! I'm down to keep it as is if it feels more legible. |
||
| - name: Log final value of GITHUB_REF | ||
| run: | | ||
| echo "Final GITHUB_REF:" ${{ env.GITHUB_REF }} | ||
| - uses: actions/checkout@v4 | ||
| with: | ||
| token: ${{ secrets.PUDL_BOT_PAT }} | ||
| ref: ${{ env.GITHUB_REF }} | ||
| - name: Install Micromamba | ||
| uses: mamba-org/setup-micromamba@v1 | ||
| with: | ||
| environment-name: conda-lock | ||
| create-args: >- | ||
| python=3.11 | ||
| conda-lock | ||
| prettier | ||
| - name: Run conda-lock to recreate lockfile from scratch | ||
| run: | | ||
| make conda-clean | ||
| make conda-lock.yml | ||
| - name: Commit updated conda lockfiles to branch | ||
| # If running on push due to dependency changes, commit directly to the base | ||
| # branch of the existing PR. Don't trigger the workflow again if we're already | ||
| # running it as pudlbot (to avoid infinite recursion). | ||
| if: ${{ (github.event_name == 'push' && github.actor != 'pudlbot') }} | ||
| uses: stefanzweifel/git-auto-commit-action@v5 | ||
| with: | ||
| file_pattern: "environments/*" | ||
| commit_message: "Update conda-lock.yml and rendered conda environment files." | ||
|
Comment on lines
+59
to
+66
There was a problem hiding this comment. What's stopping the action from running again on a push? There was a problem hiding this comment. The second time, the action is being run by There was a problem hiding this comment. I looked at the docs for the Would it make sense test this condition and abort earlier in the workflow? |
||
| - name: Make a PR to merge updated conda lockfiles | ||
| # If we are relocking dependencies on a schedule or workflow_dispatch, we need | ||
| # to make our own PR to check whether the updated environment actually solves | ||
| # and the tests pass. | ||
| if: ${{ (github.event_name == 'schedule' && github.repository == 'catalyst-cooperative/pudl') || (github.event_name == 'workflow_dispatch') }} | ||
| uses: peter-evans/create-pull-request@v5 | ||
| with: | ||
| commit-message: "Update conda-lock.yml and rendered conda environment files." | ||
| title: Update Lockfile | ||
| body: > | ||
| This pull request relocks the dependencies with conda-lock. | ||
| It is triggered by [update-conda-lockfile](https://github.com/catalyst-cooperative/pudl/blob/main/.github/workflows/update-conda-lockfile.yml). | ||
| labels: dependencies, conda-lock | ||
| reviewers: zaneselvans | ||
zaneselvans marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| branch: update-conda-lockfile | ||
| base: ${{ env.GITHUB_REF }} | ||
| draft: true | ||
| delete-branch: true | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Conda swallows all of the logs if you don't include
--no-capture-outputbut it sounds like--attach ""should work:Maybe we run part of a full build to make sure the logs are coming through before we merge into
dev?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did run one full build, I think it should be in the build outputs if you want to take a look at the logs and see if they look like you'd expect them to. I think using
pytest-xdistdoes obscure some of the test logging, but also shaves a couple of hours off the nightly builds.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the output from the parallelized tests if you want to take a look:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These logs look good to me!