-
Notifications
You must be signed in to change notification settings - Fork 23
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Cloud Deployment IIIb]: AWS batch deployment helper (#384)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: CodyCBakerPhD <[email protected]> Co-authored-by: Heberto Mayorquin <[email protected]>
- Loading branch information
4 people
authored
Aug 8, 2024
1 parent
ef656a0
commit b7ae085
Showing
9 changed files
with
820 additions
and
7 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| name: AWS Tests | ||
| on: | ||
| schedule: | ||
| - cron: "0 16 * * 1" # Weekly at noon on Monday | ||
| workflow_dispatch: | ||
|
|
||
| concurrency: # Cancel previous workflows on the same pull request | ||
| group: ${{ github.workflow }}-${{ github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| env: | ||
| AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} | ||
| AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} | ||
| DANDI_API_KEY: ${{ secrets.DANDI_API_KEY }} | ||
|
|
||
| jobs: | ||
| run: | ||
| name: ${{ matrix.os }} Python ${{ matrix.python-version }} | ||
| runs-on: ${{ matrix.os }} | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| python-version: ["3.12"] | ||
| os: [ubuntu-latest] | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - run: git fetch --prune --unshallow --tags | ||
| - name: Setup Python ${{ matrix.python-version }} | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: ${{ matrix.python-version }} | ||
|
|
||
| - name: Global Setup | ||
| run: | | ||
| python -m pip install -U pip # Official recommended way | ||
| git config --global user.email "[email protected]" | ||
| git config --global user.name "CI Almighty" | ||
| - name: Install full requirements | ||
| run: pip install .[aws,test] | ||
|
|
||
| - name: Run subset of tests that use S3 live services | ||
| run: pytest -rsx -n auto tests/test_minimal/test_tools/aws_tools.py |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,3 +19,4 @@ ignore: | |
| - "./setup.py" | ||
| - "./docs/" | ||
| - "./documentation/" | ||
| - "./src/neuroconv/tools/aws/" | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| from ._submit_aws_batch_job import submit_aws_batch_job | ||
|
|
||
| __all__ = ["submit_aws_batch_job"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| """Helper functions for operations on DynamoDB tables.""" | ||
|
|
||
| import os | ||
|
|
||
|
|
||
| def update_table_status( | ||
| *, | ||
| submission_id: str, | ||
| status: str, | ||
| status_tracker_table_name: str = "neuroconv_batch_status_tracker", | ||
| region: str = "us-east-2", | ||
| ) -> None: | ||
| """ | ||
| Helper function for updating a status value on a DynamoDB table tracking the status of EC2 jobs. | ||
| Intended for use by the running job to indicate its completion. | ||
| Parameters | ||
| ---------- | ||
| submission_id : str | ||
| The random hash that was assigned on submission of this job to the status tracker table. | ||
| status : str | ||
| The new status value to update. | ||
| status_tracker_table_name : str, default: "neuroconv_batch_status_tracker" | ||
| The name of the DynamoDB table to use for tracking job status. | ||
| region : str, default: "us-east-2" | ||
| The AWS region to use for the job. | ||
| """ | ||
| import boto3 | ||
|
|
||
| aws_access_key_id = os.environ["AWS_ACCESS_KEY_ID"] | ||
| aws_secret_access_key = os.environ["AWS_SECRET_ACCESS_KEY"] | ||
|
|
||
| dynamodb_resource = boto3.resource( | ||
| service_name="dynamodb", | ||
| region_name=region, | ||
| aws_access_key_id=aws_access_key_id, | ||
| aws_secret_access_key=aws_secret_access_key, | ||
| ) | ||
| table = dynamodb_resource.Table(name=status_tracker_table_name) | ||
|
|
||
| table.update_item(Key={"id": submission_id}, AttributeUpdates={"status": {"Action": "PUT", "Value": status}}) | ||
|
|
||
| return None |
Oops, something went wrong.