Use Papermill to run notebook

[SammyAgrawal]

Can you run a notebook as a job and save the output of each cell?

[norlandrhagen]

My co-worker used papermill a while ago on our CMIP6 work. This example notebook might be helpful: https://github.com/carbonplan/cmip6-downscaling/blob/7957fc1e086574ba8a9ff16cb5d810b580f742f5/notebooks/run_analyses.ipynb#L10

[jbusecke]

Anything you find here would be very valuable to document, even if it is just linking existing resources.

[SammyAgrawal]

Running Papermill example notebook from the docs worked great, as did running from python file.

import papermill as pm
import json

pdict = dict(alpha=0.6, num_steps=10)
with open('param_test.json', 'w') as f:
    json.dump(pdict, f, indent=2)

with open('param_test.json', 'r') as f:
    pdict_loaded = json.load(f)


pm.execute_notebook(
    'notebook_test.ipynb',
    'output.ipynb', 
    parameters = pdict_loaded
)

Output is saved to external notebook.
My next tests:

• upload papermill to a custom docker image on quay
• test alongside kbatch

[SammyAgrawal]

blocked: how to load output ipynb and place into user directory