-
Notifications
You must be signed in to change notification settings - Fork 80
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Setup GitHub Actions and Travis for automated submissions #428
Merged
Merged
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
b61ffe1
import script from core
3bffb8a
add action workflows
a8d14f1
trigger automerge for plugin-only web submissions
b3baebc
model_type=Brain_Model
268a031
python -> 3.7
0cf0c05
update repo to vision
2355a2b
cleanup
d2f2a87
cleanup
ed8ba1c
clarify imported scripts
dd80029
Merge branch 'integrate_core' into kvf/automated_submissions
kvfairchild File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,114 @@ | ||
| name: Automatically merge plugin-only PRs | ||
|
|
||
|
|
||
| # Triggered on all PRs either by | ||
| # - completion of CI checks, OR | ||
| # - tagging with label | ||
| # 1) If PR is labeled "automerge" or "automerge-web" | ||
| # (all website submissions are tagged "automerge-web"), | ||
| # checks if Travis tests pass. If yes, THEN | ||
| # 2) Checks if PR modifies any code outside plugin dirs. | ||
| # If no changes are made beyond new or revised plugins | ||
| # (subdirs of /benchmarks, /data, /models, or /metrics) | ||
| # the PR is automatically approved and merged. | ||
|
|
||
|
|
||
| on: | ||
| pull_request: | ||
| types: [labeled] | ||
| status: | ||
|
|
||
| permissions: write-all | ||
|
|
||
| jobs: | ||
|
|
||
| isautomerge: | ||
| name: Set as 'automerge' if PR is labeled with 'automerge' or 'automerge-web' | ||
| runs-on: ubuntu-latest | ||
| if: | | ||
| contains( github.event.pull_request.labels.*.name, 'automerge') || | ||
| contains( github.event.pull_request.labels.*.name, 'automerge-web') | ||
| outputs: | ||
| AUTOMERGE: ${{ steps.setautomerge.outputs.AUTOMERGE }} | ||
| steps: | ||
| - name: Set 'automerge' to 'True' # job only runs if True | ||
| id: setautomerge | ||
| run: | | ||
| echo "::set-output name=AUTOMERGE::True" | ||
|
|
||
|
|
||
| travis_success: | ||
| name: Check if Travis build is successful | ||
| runs-on: ubuntu-latest | ||
| needs: [isautomerge] | ||
| if: ${{ needs.isautomerge.outputs.AUTOMERGE == 'True' }} | ||
| outputs: | ||
| TRAVIS_OK: ${{ steps.istravisok.outputs.TRAVIS_OK }} | ||
| steps: | ||
| - name: Get Travis build status | ||
| id: gettravisstatus | ||
| run: | | ||
| echo ${{ github.event.pull_request.head.sha }} | ||
| echo "TRAVIS_CONCLUSION=$(python -c "import requests; r = requests.get(\"https://api.github.com/repos/brain-score/vision/commits/${{ github.event.pull_request.head.sha }}/check-runs\"); print(next(run['conclusion'] for run in r.json()['check_runs'] if run['name'] == 'Travis CI - Pull Request'))")" >> $GITHUB_ENV | ||
| - name: Check if Travis was successful | ||
| id: istravisok | ||
| run: | | ||
| if [ "$TRAVIS_CONCLUSION" == "success" ] | ||
| then | ||
| travisok=True | ||
| elif [ "$TRAVIS_CONCLUSION" == "None" ] | ||
| then | ||
| travisok=Wait | ||
| else | ||
| travisok=False | ||
| fi | ||
| echo "::set-output name=TRAVIS_OK::$travisok" | ||
|
|
||
|
|
||
| plugin_only: | ||
| name: Ensure PR ONLY changes plugin files | ||
| runs-on: ubuntu-latest | ||
| needs: travis_success | ||
| if: ${{ needs.travis_success.outputs.TRAVIS_OK == 'True' }} | ||
| outputs: | ||
| PLUGIN_ONLY: ${{ steps.ispluginonly.outputs.PLUGIN_ONLY }} | ||
| steps: | ||
| - name: Parse plugin_only confirmation from Travis status update | ||
| id: getpluginonlyvalue | ||
| run: echo "PLUGIN_ONLY=$(python -c "import requests; r = requests.get(\"https://api.github.com/repos/brain-score/vision/statuses/$github.event.pull_request.head.sha\"); print(next(status['description'].split('- ')[1] for status in r.json() if status['description'].startswith('Run automerge workflow')))")" >> $GITHUB_ENV | ||
| - name: Check if PR is plugin only | ||
| id: ispluginonly | ||
| run: | | ||
| if [ "$PLUGIN_ONLY" == "True" ] | ||
| then | ||
| pluginonly=True | ||
| else | ||
| pluginonly=False | ||
| fi | ||
| echo "::set-output name=PLUGIN_ONLY::$pluginonly" | ||
|
|
||
|
|
||
| automerge: | ||
| name: If plugin-only, approve and merge | ||
| runs-on: ubuntu-latest | ||
| needs: plugin_only | ||
| if: ${{ needs.plugin_only.outputs.PLUGIN_ONLY == 'True' }} | ||
| steps: | ||
| - name: Auto Approve | ||
| uses: hmarr/[email protected] | ||
|
|
||
| - name: Auto Merge (GitHub submissions) | ||
| uses: plm9606/[email protected] | ||
| with: | ||
| github-token: ${{ secrets.WORKFLOW_TOKEN }} | ||
| label-name: "automerge" | ||
| merge-method: "squash" | ||
| auto-delete: "true" | ||
|
|
||
| - name: Auto Merge (brain-score.org submissions) | ||
| uses: plm9606/[email protected] | ||
| with: | ||
| github-token: ${{ secrets.WORKFLOW_TOKEN }} | ||
| label-name: "automerge-web" | ||
| merge-method: "squash" | ||
| auto-delete: "true" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| name: Trigger scoring run | ||
|
|
||
|
|
||
| # Triggered on all PRs on merge to main | ||
| # If changes are made to a subdir of /benchmarks or /models, | ||
| # a Jenkins scoring run is triggered for the corresponding plugin | ||
|
|
||
|
|
||
| on: | ||
| pull_request: | ||
| branches: | ||
| - main | ||
| types: | ||
| - closed | ||
|
|
||
| env: | ||
| BSC_DATABASESECRET: secrets.BSC_DATABASESECRET | ||
|
|
||
| permissions: write-all | ||
|
|
||
| jobs: | ||
|
|
||
| changes_models_or_benchmarks: | ||
| name: Check if PR makes changes to /models or /benchmarks | ||
| runs-on: ubuntu-latest | ||
| outputs: | ||
| PLUGIN_INFO: ${{ steps.getpluginfo.outputs.PLUGIN_INFO }} | ||
| RUN_SCORE: ${{ steps.runscore.outputs.RUN_SCORE }} | ||
| steps: | ||
| - name: Check out repository code | ||
| uses: actions/checkout@v2 | ||
| with: | ||
| fetch-depth: 0 | ||
|
|
||
| - name: Set up Python 3.7 | ||
| uses: actions/setup-python@v4 | ||
| with: | ||
| python-version: 3.7 | ||
|
|
||
| - name: Save changed files to env var | ||
| run: echo "CHANGED_FILES=$(git diff --name-only origin/main~1 origin/$GITHUB_HEAD_REF | tr '\n' ' ')" >> $GITHUB_ENV | ||
|
|
||
| - name: Installing package dependencies | ||
| run: | | ||
| python -m pip install --upgrade pip setuptools | ||
| python -m pip install ".[test]" | ||
|
|
||
| - name: Get plugin info | ||
| id: getpluginfo | ||
| run: | | ||
| echo "PLUGIN_INFO='$(python -c 'from brainscore_core.plugin_management.parse_plugin_changes import get_scoring_info; get_scoring_info("${{ env.CHANGED_FILES }}", "brainscore_vision")')'" >> $GITHUB_OUTPUT | ||
|
|
||
| - name: Run scoring | ||
| id: runscore | ||
| run: | | ||
| echo "RUN_SCORE=$(jq -r '.run_score' <<< ${{ steps.getpluginfo.outputs.PLUGIN_INFO }})" >> $GITHUB_OUTPUT | ||
|
|
||
| get_submitter_info: | ||
| name: Get PR author email and (if web submission) Brain-Score user ID | ||
| runs-on: ubuntu-latest | ||
| needs: [setup, changes_models_or_benchmarks] | ||
| if: ${{ needs.changes_models_or_benchmarks.outputs.RUN_SCORE == 'True' }} | ||
| env: | ||
| PLUGIN_INFO: ${{ needs.changes_models_or_benchmarks.outputs.PLUGIN_INFO }} | ||
| outputs: | ||
| PLUGIN_INFO: ${{ steps.add_email_to_pluginfo.outputs.PLUGIN_INFO }} | ||
| steps: | ||
| - name: Parse user ID from PR title (WEB ONLY where we don't have access to the GitHub user) | ||
| id: getuid | ||
| if: ${{ github.event.pull_request.labels.*.name == 'automerge-web' }} | ||
| run: | | ||
| echo "BS_UID="$(<<<${{ github.event.pull_request.title }} | sed -E 's/.*\(user:([^)]+)\).*/\1/'"" >> $GITHUB_ENV | ||
| - name: Add user ID to PLUGIN_INFO (WEB ONLY) | ||
| id: add_uid_to_pluginfo | ||
| if: ${{ github.event.pull_request.labels.*.name == 'automerge-web' }} | ||
| run: | | ||
| echo "The Brain-Score user ID is ${{ steps.getuid.outputs.BS_UID }}" | ||
| echo "PLUGIN_INFO="$(<<<$PLUGIN_INFO jq '. + {user_id: ${{ steps.getuid.outputs.UID }} }')"" >> $GITHUB_ENV | ||
|
|
||
| - name: Get PR author email from GitHub username | ||
| id: getemail | ||
| uses: evvanErb/[email protected] | ||
| with: | ||
| github-username: ${{github.event.pull_request.user.login}} # PR author's username | ||
| token: ${{ secrets.GITHUB_TOKEN }} # Including token enables most reliable way to get a user's email | ||
| - name: Add PR author email to PLUGIN_INFO | ||
| id: add_email_to_pluginfo | ||
| run: | | ||
| echo "The PR author email is ${{ steps.getemail.outputs.email }}" | ||
| echo "PLUGIN_INFO=$(<<<$PLUGIN_INFO tr -d "'" | jq -c '. + {author_email: "${{ steps.getemail.outputs.email }}"}')" >> $GITHUB_OUTPUT | ||
|
|
||
|
|
||
| runscore: | ||
| name: Score plugins | ||
| runs-on: ubuntu-latest | ||
| needs: [changes_models_or_benchmarks, get_submitter_info] | ||
| if: ${{ needs.changes_models_or_benchmarks.outputs.RUN_SCORE == 'True' }} | ||
| env: | ||
| PLUGIN_INFO: ${{ needs.get_submitter_info.outputs.PLUGIN_INFO }} | ||
| JENKINS_USER: ${{ secrets.JENKINS_USER }} | ||
| JENKINS_TOKEN: ${{ secrets.JENKINS_TOKEN }} | ||
| JENKINS_TRIGGER: ${{ secrets.JENKINS_TRIGGER }} | ||
| steps: | ||
| - name: Add domain, public, competition, and model_type to PLUGIN_INFO | ||
| run: | | ||
| echo "PLUGIN_INFO=$(<<<$PLUGIN_INFO tr -d "'" | jq -c '. + {domain: "vision", public: true, competition: "None", model_type: "Brain_Model"}')" >> $GITHUB_ENV | ||
|
|
||
| - name: Check out repository code | ||
| uses: actions/checkout@v2 | ||
|
|
||
| - name: Build project and run scoring | ||
| run: | | ||
| python -m pip install --upgrade pip setuptools | ||
| python -m pip install ".[test]" | ||
| python -c 'from brainscore_core.submission.endpoints import call_jenkins; call_jenkins('\''${{ env.PLUGIN_INFO }}'\'')' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| #!/bin/bash | ||
|
|
||
| GH_WORKFLOW_TRIGGER=$1 | ||
| TRAVIS_PULL_REQUEST_SHA=$2 | ||
|
|
||
| curl -L -X POST \ | ||
| -H "Authorization: token $GH_WORKFLOW_TRIGGER" \ | ||
| -d '{"state": "success", "description": "Run automerge workflow for plugin-only PR", | ||
| "context": "continuous-integration/travis"}' \ | ||
| "https://api.github.com/repos/brain-score/brain-score/statuses/$TRAVIS_PULL_REQUEST_SHA" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought all
scriptnow come from the import?