Copy resource contents from Prod to Beta #5697
Conversation
| MINIO_SECRET_KEY: ${{ secrets.MINIO_SECRET_KEY }} | ||
| run: | | ||
| mc alias set prod $MINIO_PROD_URL $MINIO_ACCESS_KEY $MINIO_SECRET_KEY | ||
| mc alias set beta $MINIO_BETA_URL $MINIO_ACCESS_KEY $MINIO_SECRET_KEY |
There was a problem hiding this comment.
I'm pretty sure these keys will be printed to console. I wonder if you can write the keys to a configuration file instead.
Also, I think it would be best to not share key/secret between environments.
There was a problem hiding this comment.
Hi Scott,
I moved the keys to a config file and seperated beta and prod keys.
| IFS=',' read -ra RESOURCES <<< "$RESOURCE_IDS" | ||
|
|
||
| for RESOURCE_ID in "${RESOURCES[@]}"; do | ||
| echo "Checking resource: $RESOURCE_ID" |
There was a problem hiding this comment.
You can use mc mirror for the sync. By including the --remove --overwrite flags, it'll sync for the source to the target completely (including removing files that are on the target that aren't in the source).
You will also need to look up the bucket the resource is in, in order to sync the resource. There is an endpoint for this, GET https://beta.hydroshare.org/hsapi/resource/$resource_id/quota_holder_bucket_name/. An admin basic credentials are necessary to hit the endpoint.
You will also need to make sure the bucket exists on the target minio server. That can be done with mc mb --ignore-existing
| sudo mv mc /usr/local/bin/mc | ||
|
|
||
| - name: Create MinIO Config File | ||
| env: |
There was a problem hiding this comment.
You can collapse this run and the next run into just setting the environment variables with https://min.io/docs/minio/linux/reference/minio-mc/minio-client-settings.html#envvar.MC_HOST_-ALIAS
There was a problem hiding this comment.
The rest of it looks good and can should run successfully, given correct secret values.
GH Action to copy resource contents from prod to beta, if bucket exists
Pull Request Checklist:
Positive Test Case