[GEN-1653] Add integration tests to GH actions workflow #582
Conversation
|
| docker run -d --name genie-container \ | ||
| -e SYNAPSE_AUTH_TOKEN="${{ secrets.SYNAPSE_AUTH_TOKEN }}" \ | ||
| ghcr.io/sage-bionetworks/genie:${{ env.BRANCH_NAME }} \ | ||
| sh -c "while true; do sleep 1; done" |
There was a problem hiding this comment.
whats the motivation of this while true statement?
Whats your rationale for starting the container instead of just having 5 docker run commands?
There was a problem hiding this comment.
Hmm from my understanding I thought that we had to keep the container "open" while running the code to keep it all in one container run/same state persistence. However thinking about it now, it should probably mirror nextflow where each step is probably separate docker run.
I also wonder if this is the reason why I'm seeing differences in the outputs compared to outputs when I run the nextflow pipeline (something I am investigating ...)
There was a problem hiding this comment.
Differences in outputs?! What are the differences?
There was a problem hiding this comment.
All in the clinical consortium files. It's dropping more records for some reason (at least more compared to the nextflow pipeline run)
| - '**.md' # All Markdown files | ||
| - '**/docs/**' # Documentation directory | ||
| - '.github/workflows/codeql.yml' # Code scanner | ||
| - '.github/workflows/build_docs.yml' # mkdocs GH workflow |
There was a problem hiding this comment.
There are some paths I think we can ignore as well: tests/testthat, and .pre-commit-config.yaml. There maybe more. Since it's mainly about running through the pipeline, do you think we can use path instead to only point to related scripts folder?
| run: | | ||
| docker pull ghcr.io/sage-bionetworks/genie:${{ env.BRANCH_NAME }} | ||
|
|
||
| - name: Start Docker Container |
There was a problem hiding this comment.
Just FYI, I saw https://docs.github.com/en/actions/writing-workflows/choosing-where-your-workflow-runs/running-jobs-in-a-container. It seems to be a way to spin up the container.
Purpose: This is a draft PR. This adds automated integration tests for each step of the genie pipeline into our GH actions workflow. Still need to do the following:
build-containerstep should be pre-req to this stepTesting:
Ran all testing pipeline steps here: https://github.com/Sage-Bionetworks/Genie/actions/runs/12662104568. It took close to an hour (mainly validation and consortium release steps taking the longest). Still need to look into why the case (separate ticket)