Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create the CIs for Icy and deepImageJ using JDLL #82

Draft
wants to merge 86 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
38df8e7
start adding workflow for icy
carlosuc3m Jun 13, 2024
fb0477c
update jdll version
carlosuc3m Jun 13, 2024
298bfc1
listen to folder to trigger workflow
carlosuc3m Jun 13, 2024
0ce6e42
add class to download engines, different for Icy and DIJ
carlosuc3m Jun 13, 2024
5aeb5c4
download engines from action
carlosuc3m Jun 13, 2024
4acdc03
install first
carlosuc3m Jun 13, 2024
4dbed35
correct error in syntax
carlosuc3m Jun 13, 2024
be38fa1
remove problematic code
carlosuc3m Jun 13, 2024
f8145d1
check the directory
carlosuc3m Jun 13, 2024
cfccb6b
correct directory
carlosuc3m Jun 13, 2024
0a4b3e4
remove unnecessary classes for logging
carlosuc3m Jun 13, 2024
6ed2651
do not trigger build
carlosuc3m Jun 13, 2024
af205f6
avoid triggering it while developping
carlosuc3m Jun 13, 2024
fbc24c1
correct name of folder
carlosuc3m Jun 13, 2024
10268b9
correct dir name
carlosuc3m Jun 13, 2024
a9b76be
add useful prints
carlosuc3m Jun 13, 2024
0bf128a
ensure the change happens on every subfodler
carlosuc3m Jun 13, 2024
305374e
add job that runs models
carlosuc3m Jun 13, 2024
4bb3619
needs previous run
carlosuc3m Jun 13, 2024
41d53e3
correct where to put needs
carlosuc3m Jun 13, 2024
132dfa7
just a single job
carlosuc3m Jun 13, 2024
239b1d6
update jdll version
carlosuc3m Jun 13, 2024
d4aad65
do two directories back
carlosuc3m Jun 13, 2024
b9a57fb
check the artifacts that are produced
carlosuc3m Jun 19, 2024
ed329cc
add part to upload changesusing python (python script still missing)
carlosuc3m Jun 25, 2024
03c8b8e
start creating the python code to upload dij and icy tests
carlosuc3m Jun 26, 2024
f2877b3
keep working towards convergence of CI
carlosuc3m Jul 2, 2024
6a705db
separate the tests for icy and deepimagej
carlosuc3m Jul 2, 2024
82fd7b6
install every engine for Icy
carlosuc3m Jul 2, 2024
41c2c79
do not fail when an engine is not properly installed
carlosuc3m Jul 3, 2024
313145e
remove bioimage.io python deps
carlosuc3m Jul 3, 2024
3d332bf
correct code to make the results depend on software and version
carlosuc3m Jul 3, 2024
39a9fd3
renew the way tests are created
carlosuc3m Jul 4, 2024
9c8380d
keep improving the java part of testing
carlosuc3m Jul 4, 2024
346fbe8
stabilize java part
carlosuc3m Jul 4, 2024
8da01d5
create script to retrieve the dij and icy versions
carlosuc3m Jul 4, 2024
027e1d7
keep iterating
carlosuc3m Jul 4, 2024
2964cb6
add code to understand JDLL and java software versions tested
carlosuc3m Jul 5, 2024
950b1c9
print version so github action can use it
carlosuc3m Jul 5, 2024
632e15e
update actions to use new script retrieving version dict
carlosuc3m Jul 5, 2024
2733da6
add installation of deps
carlosuc3m Jul 5, 2024
e4a03d6
corect small details
carlosuc3m Jul 5, 2024
e1e904b
add listener to another file
carlosuc3m Jul 5, 2024
7af4a00
try to change something
carlosuc3m Jul 5, 2024
8f4139e
debug
carlosuc3m Jul 5, 2024
72f985a
debug
carlosuc3m Jul 5, 2024
a443fa1
debug
carlosuc3m Jul 5, 2024
4ce9640
try new changes
carlosuc3m Jul 5, 2024
d7fd1c1
try more changes
carlosuc3m Jul 5, 2024
df64bd6
more changes
carlosuc3m Jul 5, 2024
1344fa5
use a list of entries instead of a dict
carlosuc3m Jul 5, 2024
3d29be5
redo map
carlosuc3m Jul 5, 2024
1b016c7
stabilize version management
carlosuc3m Jul 5, 2024
f03fff9
che k the dir
carlosuc3m Jul 5, 2024
2904eb6
move to the correct dir to execute java code
carlosuc3m Jul 5, 2024
c832bab
orint jarfile
carlosuc3m Jul 5, 2024
de04824
correct paths again
carlosuc3m Jul 5, 2024
1fbc127
check if jarfile persits
carlosuc3m Jul 5, 2024
8c4b5ca
preserve jarfile name
carlosuc3m Jul 5, 2024
e333e6a
add forgotten id
carlosuc3m Jul 5, 2024
707eb22
need to jobs
carlosuc3m Jul 5, 2024
7a8892c
install deps
carlosuc3m Jul 5, 2024
8a17d95
correct version number
carlosuc3m Jul 5, 2024
4cff69f
update python code
carlosuc3m Jul 5, 2024
f397ab5
run as modules to avoid import errors
carlosuc3m Jul 9, 2024
241d6dd
check if this change eas the error
carlosuc3m Jul 9, 2024
e49dbce
improve robustness
carlosuc3m Jul 9, 2024
6be07cf
create the collection backoffice plus dependencies
carlosuc3m Jul 10, 2024
b521fe1
improving naming of the paths of test summaries
carlosuc3m Jul 10, 2024
fc4cb0d
correct installation of backoffice
carlosuc3m Jul 10, 2024
89951e4
Merge branch 'bioimage-io:main' into main
carlosuc3m Jul 10, 2024
0ff8c70
update gitignore again and remove silly print
carlosuc3m Jul 10, 2024
d700c34
provide the arguments correctly
carlosuc3m Jul 10, 2024
a64cc1d
correct error that was failing passed tests
carlosuc3m Jul 10, 2024
b90e825
fail the workflow on purpose
carlosuc3m Jul 10, 2024
8f4ea3e
fail the other workflow on purpose
carlosuc3m Jul 10, 2024
7f2c972
test fail
carlosuc3m Jul 10, 2024
7a04b2e
restore build yaml
carlosuc3m Jul 10, 2024
b5a4014
restore default
carlosuc3m Jul 10, 2024
7195192
remove intentional fail
carlosuc3m Jul 10, 2024
06da2c4
remove default path forgotten
carlosuc3m Jul 10, 2024
1ee4bcb
make the details more understandable
carlosuc3m Jul 11, 2024
0fe821e
remove intentional fail
carlosuc3m Jul 11, 2024
efd911c
also catch errors
carlosuc3m Jul 11, 2024
8924cae
at the moment only use one version per engine
carlosuc3m Jul 11, 2024
854f356
correct a couple of details of the action
carlosuc3m Jul 11, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 105 additions & 0 deletions .github/workflows/check_compatibility_deepimagej.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
name: check compatibility deepimagej

concurrency: deepimagej

on:
push:
branches:
- main
paths:
- .github/workflows/check_compatibility_deepimagej.yaml
- scripts/get_java_software_versions.py
- scripts/check_compatibility_java_software/**/*
workflow_dispatch:
schedule:
- cron: 0 1 * * * # update compatibility once a day

jobs:
generate-version-dict:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-dict.outputs.matrix }}
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.x'
- name: Install deps
run: pip install requests==2.28.2 beautifulsoup4==4.12.3 packaging==23.0
- name: Generate dict
id: set-dict
run: |
matrix_output=$(python scripts/get_java_software_versions.py deepimagej)
echo "matrix=${matrix_output}" >> $GITHUB_OUTPUT
- name: Versions matrix
run: echo '${{ steps.set-dict.outputs.matrix }}'

run:
needs: generate-version-dict
runs-on: ubuntu-latest
strategy:
matrix: ${{ fromJson(needs.generate-version-dict.outputs.matrix) }}
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v3
with:
repository: bioimage-io/collection-bioimage-io
ref: gh-pages
path: bioimageio-gh-pages
- name: Setup Maven Action
uses: stCarolas/[email protected]
with:
java-version: 11
maven-version: 3.9.5
- name: Build with Maven
run: |
cd scripts/check_compatibility_java_software
mvn clean install org.apache.maven.plugins:maven-shade-plugin:3.2.4:shade -Dshade.mainClass=io.bioimage.modelrunner.ci.ContinuousIntegration
- name: Get jar file name
id: get-jar
run: |
cd scripts/check_compatibility_java_software
jarfile=$(mvn -q exec:exec -Dexec.executable=echo -Dexec.args='${project.build.finalName}.jar')
if [ -z "$jarfile" ]; then
echo "Failed to get jar file name"
exit 1
fi
if [ ! -f "target/$jarfile" ]; then
echo "Jar file not found: target/$jarfile"
exit 1
fi
echo "Jar file found: target/$jarfile"
echo "jarfile=$jarfile" >> $GITHUB_OUTPUT
- name: Download engines
run: |
java -cp "scripts/check_compatibility_java_software/target/${{ steps.get-jar.outputs.jarfile }}" io.bioimage.modelrunner.ci.DownloadEngines deepimagej
- name: Run models
run: |
java -cp "scripts/check_compatibility_java_software/target/${{ steps.get-jar.outputs.jarfile }}" io.bioimage.modelrunner.ci.ContinuousIntegration deepimagej ${{ matrix.key }}
- name: Upload artifacts
uses: actions/upload-artifact@v3
with:
path: test_summaries_deepimagej_${{ matrix.key }}

sendreport:
runs-on: ubuntu-latest
needs: [generate-version-dict, run]
strategy:
matrix: ${{ fromJson(needs.generate-version-dict.outputs.matrix) }}
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: '3.10'
- name: Get test results
uses: actions/download-artifact@v3
with:
path: test_summaries_deepimagej_${{ matrix.key }}
- name: Install deps
run: |
pip install .
- name: Send deepimagej ${{ matrix.key }} tests
shell: bash -l {0}
run: python scripts/check_compatibility_java_software.py deepimagej ${{ matrix.key }} --summaries_dir "test_summaries_deepimagej_${{ matrix.key }}/artifact"
105 changes: 105 additions & 0 deletions .github/workflows/check_compatibility_icy.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
name: check compatibility icy

concurrency: icy

on:
push:
branches:
- main
paths:
- .github/workflows/check_compatibility_icy.yaml
- scripts/get_java_software_versions.py
- scripts/check_compatibility_java_software/**/*
workflow_dispatch:
schedule:
- cron: 0 1 * * * # update compatibility once a day

jobs:
generate-version-dict:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-dict.outputs.matrix }}
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.x'
- name: Install deps
run: pip install requests==2.28.2 beautifulsoup4==4.12.3 packaging==23.0
- name: Generate dict
id: set-dict
run: |
matrix_output=$(python scripts/get_java_software_versions.py icy)
echo "matrix=${matrix_output}" >> $GITHUB_OUTPUT
- name: Versions matrix
run: echo '${{ steps.set-dict.outputs.matrix }}'

run:
needs: generate-version-dict
runs-on: ubuntu-latest
strategy:
matrix: ${{ fromJson(needs.generate-version-dict.outputs.matrix) }}
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v3
with:
repository: bioimage-io/collection-bioimage-io
ref: gh-pages
path: bioimageio-gh-pages
- name: Setup Maven Action
uses: stCarolas/[email protected]
with:
java-version: 11
maven-version: 3.9.5
- name: Build with Maven
run: |
cd scripts/check_compatibility_java_software
mvn clean install org.apache.maven.plugins:maven-shade-plugin:3.2.4:shade -Dshade.mainClass=io.bioimage.modelrunner.ci.ContinuousIntegration
- name: Get jar file name
id: get-jar
run: |
cd scripts/check_compatibility_java_software
jarfile=$(mvn -q exec:exec -Dexec.executable=echo -Dexec.args='${project.build.finalName}.jar')
if [ -z "$jarfile" ]; then
echo "Failed to get jar file name"
exit 1
fi
if [ ! -f "target/$jarfile" ]; then
echo "Jar file not found: target/$jarfile"
exit 1
fi
echo "Jar file found: target/$jarfile"
echo "jarfile=$jarfile" >> $GITHUB_OUTPUT
- name: Download engines
run: |
java -cp "scripts/check_compatibility_java_software/target/${{ steps.get-jar.outputs.jarfile }}" io.bioimage.modelrunner.ci.DownloadEngines icy
- name: Run models
run: |
java -cp "scripts/check_compatibility_java_software/target/${{ steps.get-jar.outputs.jarfile }}" io.bioimage.modelrunner.ci.ContinuousIntegration icy ${{ matrix.key }}
- name: Upload artifacts
uses: actions/upload-artifact@v3
with:
path: test_summaries_icy_${{ matrix.key }}

sendreport:
runs-on: ubuntu-latest
needs: [generate-version-dict, run]
strategy:
matrix: ${{ fromJson(needs.generate-version-dict.outputs.matrix) }}
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: '3.10'
- name: Get test results
uses: actions/download-artifact@v3
with:
path: test_summaries_icy_${{ matrix.key }}
- name: Install deps
run: |
pip install .
- name: Send icy ${{ matrix.key }} tests
shell: bash -l {0}
run: python scripts/check_compatibility_java_software.py icy ${{ matrix.key }} --summaries_dir "test_summaries_icy_${{ matrix.key }}/artifact"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,7 @@ __pycache__/
*.egg-info/
docs/
*.pyc
TEMP/
bioimageio-gh-pages
scripts/check_compatibility_java_software/test_summaries_null_null
scripts/check_compatibility_java_software/test_summaries_default_default
Loading