Add PAVICS homepage notebooks to Jenkins testsuite and new Jupyter env build #79
Conversation
In addition to the writable-workspace/ folder, we also need to re-create the pavics-homepage/ layout. To fix ``` DriverError: /notebook_dir/pavics-homepage/tutorial_data/test_regions.geojson: No such file or directory ``` Path to those .geojson files are hardcoded so users can copy the nb to writable-workspace/ dir and still be able to run them seemlessly from the Jupyter env (without having to also copy those *.geojson files with the notebooks). The layout creation will be delegated to pavics-landing repo so this repo is generic.
…se testdata from prod Add "TEST_USE_PROD_DATA" on any lines that we need to regex replace "pavics" to "boreas". For pavics-landing notebooks that use huge data from .ncml: ``` HTTPError: 404 Client Error: for url: http://lvupavicsdev.ouranos.ca:8083/twitcher/ows/proxy/thredds/catalog/datasets/simulations/bias_adjusted/cmip5/ouranos/cb-oura-1.0/catalog.xml ``` Can not replicate all those .nc files that provides those .ncml.
unzip needed to test PAVICS-landing notebooks under Jenkins. No other packages update. See PR Ouranosinc/PAVICS-e2e-workflow-tests#79 for more details.
… public This makes the cronjob setup easier since no ssh keys to worry about. pavics-landing repo is made public for Ouranosinc/PAVICS-e2e-workflow-tests#79.
… public This makes the cronjob setup easier since no ssh keys to worry about. pavics-landing repo is made public for Ouranosinc/PAVICS-e2e-workflow-tests#79.
deploy pavics-landing: update to https repo url since the repo is now public This makes the cronjob setup easier since no ssh keys to worry about. pavics-landing repo is made public for Ouranosinc/PAVICS-e2e-workflow-tests#79.
…to-jenkins-testsuite Add PAVICS homepage notebooks to Jenkins testsuite Various changes so the notebooks pass Jenkins: * Stable interface to setup the layout (unzip output.zip, .geojson files path are hardcoded) required by the notebooks. With this stable interface, future layout change will be transparent to Jenkins * Make all prod Thredds access go through httpS so Jenkins test server outside of Ouranos can still run the notebooks (CRIM) * Add special markup for all Thredds url to get data directly from prod Pavics since that's too much `.ncml` (and `.nc`) files to replicate to all the test servers * Disable output checking for failing output that is not critical to the notebooks so we do not need to refresh too often the notebook output See each commit description for more detailed info. Not regenerating all the .html files since the code change is really minimal (adding comments and one line switch the Thredds url from direct url to behind twitcher). Matching PR that actually add the homepage notebooks to Jenkins: Ouranosinc/PAVICS-e2e-workflow-tests#79
|
@tlvu By curiosity, what's the difference between https://github.com/bird-house/birdhouse-deploy-ouranos and https://github.com/Ouranosinc/birdhouse-deploy? |
|
@matprov the birdhouse-deploy-ouranos is our custom override of generic birdhouse-deploy. These are the stuff that can not even be in any kind of optional-components because they are too specific to Ouranos, example this https://github.com/bird-house/birdhouse-deploy-ouranos/blob/e284ab8ad99419cd0d7efde8884e3d3e1bdeb95b/ouranos-config/proxy/cruesdeca-redirect.conf.template#L1-L3 or our extra scheduler jobs that is only ever relevant to us https://github.com/bird-house/birdhouse-deploy-ouranos/tree/master/scheduler-jobs. testdata deploy jobs are included in the generic one because that can potentially be of interest to others. |
…k-deploy-branch Notebook deployment: allow to specify required branch for any tutorial notebook repos in env.local ## Changes **Non-breaking changes** - Notebook deployment: allow to specify required branch for any tutorial notebook repos in `env.local` Example: set `WORKFLOW_TESTS_BRANCH` and any other notebook deploy config like `PAVICS_LANDING_BRANCH` in `env.local` - jupyter: minor update to add `unzip` package `unzip` needed to test PAVICS-landing notebooks under Jenkins. No other package updates. To support this PR Ouranosinc/PAVICS-e2e-workflow-tests#79
|
@tlvu Oh yes, my bad, given the repo name I was confused but looking at the actual files in there makes sense. What do you think about moving this under |
That would be ideal but not feasible at the moment since this e2e-workflow-test is not pluggable ! We have previously discussed in #57. Making this e2e-workflow-test repo pluggable will also allow you guys to test your EO and NLP image against your matching notebooks. Right now, the Jupyter image is hardcoded to ours, same as the notebooks :( |
Overview
Add PAVICS homepage notebooks to Jenkins testsuite and those notebooks enabled by default.
Also released new Jupyter build for the homepage notebook. Only
unzipis newly added to the Jupyter env. No other update.Matching required notebook changes PR Ouranosinc/PAVICS-landing#29
Matching PAVICS PR deployment change to test this PR bird-house/birdhouse-deploy#180
Matching PR to update pavics-landing deployment config bird-house/birdhouse-deploy-ouranos#12
Other Changes
pavics.ouranos.cain notebooks. Pavics homepage notebooks use a lot of .ncml so it was too much of .nc files to replicated to all the test servers.Related Issue / Discussion
Homepage notebooks are also deployed on Binder under
tutorial-notebooks. They are already previously deployed to Jupyter env outside oftutorial-notebooksso it is kept as-isThere is a stable interface between the homepage notebooks and Jenkins for setup the layout the notebooks requires. The stable interface means future layout changes will be transparent to Jenkins.
Additional Information
Test passing against prod Pavics: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/add-pavics-homepage-notebooks-to-jenkins-testsuite/13/console
Test passing against my dev server: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/add-pavics-homepage-notebooks-to-jenkins-testsuite/14/console
Test passing against CRIM server to test remote server and ensure the IAC pipeline will also work: https://daccs-jenkins.crim.ca/job/PAVICS-e2e-workflow-tests/job/add-pavics-homepage-notebooks-to-jenkins-testsuite/3/console (one failure unrelated).