Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chemical renaming #59

Merged
merged 295 commits into from
Feb 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
295 commits
Select commit Hold shift + click to select a range
6dd552a
Use config instead of local variables
mjaehn Dec 20, 2023
7ec7bc1
Add logging for whole chain
mjaehn Dec 20, 2023
7c1a58d
GitHub Action: Apply Pep8-formatting
invalid-email-address Dec 20, 2023
59393b3
Add logging for jobs
mjaehn Dec 20, 2023
0c01df7
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Dec 20, 2023
d2fa410
GitHub Action: Apply Pep8-formatting
invalid-email-address Dec 20, 2023
47da701
Small fix
mjaehn Dec 20, 2023
1c65c5b
GitHub Action: Apply Pep8-formatting
invalid-email-address Dec 20, 2023
b0dd12a
Format duration
mjaehn Dec 21, 2023
d2df141
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Dec 21, 2023
54b21a6
Remove --wait from submit function
mjaehn Dec 21, 2023
5769f8a
Let geosp be a separate job
mjaehn Dec 21, 2023
df4fb2c
GitHub Action: Apply Pep8-formatting
invalid-email-address Dec 21, 2023
f9a8060
Add run_chain argument to process specified chunk
mjaehn Jan 3, 2024
bbad6e9
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 3, 2024
f6e75b3
Fix datetime comparison in int2lm
mjaehn Jan 3, 2024
a68b4ae
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 3, 2024
d885c39
Submit all jobs as sbatch
mjaehn Jan 4, 2024
3181262
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 4, 2024
f1c757d
Use existing submit() function and only return job_id
mjaehn Jan 4, 2024
6661ce5
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 4, 2024
bb8844f
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 4, 2024
0b5faf1
Fix prepare_data
mjaehn Jan 4, 2024
81047bf
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 4, 2024
e64cd48
Disable logging for individual jobs
mjaehn Jan 4, 2024
efd94d6
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 4, 2024
6871eff
Write casename into logfile
mjaehn Jan 4, 2024
d7d8c08
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 4, 2024
cee8c44
Fix timers for jobs
mjaehn Jan 4, 2024
f60cd9d
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 4, 2024
aefab40
Use UTC times
mjaehn Jan 4, 2024
7f2982d
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 4, 2024
23cfddf
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 4, 2024
f61b8d5
More async jobs and workflows
mjaehn Jan 5, 2024
08aeca4
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 5, 2024
5ce30de
Add logging to new jobs
mjaehn Jan 5, 2024
dfd80b0
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 5, 2024
98ca64b
Remove unused imports
mjaehn Jan 5, 2024
a8ad905
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 5, 2024
321e629
Remove global icon-art from prepare_data
mjaehn Jan 5, 2024
8b95db7
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 5, 2024
cdbb692
Replace os.path with Pathlib
mjaehn Jan 5, 2024
18e3c54
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 5, 2024
3221bb7
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 5, 2024
7c4ab07
Fix for PosixPaths
mjaehn Jan 5, 2024
837334d
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 5, 2024
2a43496
ADD: convenience function for slurm job info
leclairm Jan 8, 2024
9956a7d
REF: move get_job_info method to the end of the class definition
leclairm Jan 8, 2024
d1fb2f7
Merge branch 'async' of github.com:C2SM/processing-chain into async
leclairm Jan 8, 2024
44b9cd0
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 8, 2024
192383d
Add some docstrings
mjaehn Jan 8, 2024
8ded708
Fix quotes for f-strings
mjaehn Jan 8, 2024
1d8f314
Add imports for art-global job
mjaehn Jan 8, 2024
f9bf445
Add dependencies for all icon workflows
mjaehn Jan 8, 2024
cc162dd
Fix pathname
mjaehn Jan 8, 2024
02e610a
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 8, 2024
ae3a814
Fix chunk calculation
mjaehn Jan 8, 2024
971ae39
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 8, 2024
75e306f
Add icontools job
mjaehn Jan 8, 2024
c7d7c72
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 8, 2024
c3024a0
Add icontools to dependencies
mjaehn Jan 8, 2024
5c43231
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 8, 2024
767ec08
Modify docstrings
mjaehn Jan 8, 2024
fd8b603
Some code cleanup
mjaehn Jan 9, 2024
d7033a5
fix(icontools): remove unsused packages + fix dependency logic
leclairm Jan 9, 2024
d2dd23d
Make prepare_art_global dependent on previous icon
mjaehn Jan 9, 2024
3ce57d6
Fix import
mjaehn Jan 9, 2024
8df44e8
Merge branch 'async' of github.com:C2SM/processing-chain into async
leclairm Jan 9, 2024
99b96cf
Explicitly set async variable
mjaehn Jan 9, 2024
07dac89
Add logfile for icontools
mjaehn Jan 9, 2024
7169901
Set variables for icontools
mjaehn Jan 9, 2024
0c9f906
Revert "Explicitly set async variable"
mjaehn Jan 9, 2024
f3e9e47
Define sequential workflow in icon-seq-test
mjaehn Jan 9, 2024
8040ee7
Make cosmo-ghg workflow async
mjaehn Jan 9, 2024
2acb4bf
Add info about (a)sync mode
mjaehn Jan 9, 2024
3d6e991
fix: also launch waiting job when some workflow jobs failed
leclairm Jan 9, 2024
52f06a1
add: other keys to default job info dict
leclairm Jan 9, 2024
9f4c456
Merge branch 'async' of github.com:C2SM/processing-chain into async
leclairm Jan 9, 2024
276a756
fix: empty current job ids before each chunk
leclairm Jan 9, 2024
1528d18
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 9, 2024
9b87be8
Split prepare_data into cosmo and icon
mjaehn Jan 9, 2024
8da59da
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 9, 2024
d4443bb
Remove unused imports
mjaehn Jan 9, 2024
39f0993
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 9, 2024
d65606e
Rename prepare_data job in workflows
mjaehn Jan 9, 2024
6ad7278
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 9, 2024
cfeb7e9
Rename prepare_data in icon-seq case
mjaehn Jan 9, 2024
7a7a7ad
Set additional cfg variables in prepare_icon
mjaehn Jan 9, 2024
7df686d
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 9, 2024
f55331d
Time logging for all jobs
mjaehn Jan 9, 2024
18d9fc3
Merge geosp into prepare_art
mjaehn Jan 9, 2024
c36f70d
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 9, 2024
729fb60
Include oem in icon-art-oem workflow
mjaehn Jan 9, 2024
32a557c
Fix icon job
mjaehn Jan 9, 2024
f23c337
Move geosp to icontools job
mjaehn Jan 9, 2024
83902f1
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 9, 2024
bae1374
add: placeholder for monitoring async jobs
leclairm Jan 9, 2024
379024f
Merge branch 'async' of github.com:C2SM/processing-chain into async
leclairm Jan 9, 2024
035ace8
Run geosp after icontools
mjaehn Jan 10, 2024
814eb5c
Remove oem job from icon-art-oem case
mjaehn Jan 10, 2024
8503340
Change to current logfile within jobs
mjaehn Jan 10, 2024
7a949cd
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 10, 2024
3ad7f3d
Add missing log inits
mjaehn Jan 10, 2024
4535b2b
Configure root logger
mjaehn Jan 10, 2024
122525a
set logger
mjaehn Jan 10, 2024
389b40a
Fix logger
mjaehn Jan 10, 2024
f49e395
Format logging output
mjaehn Jan 10, 2024
c257556
Introduce BASIC_PYTHON_JOB option to call jobs directly in async mode
mjaehn Jan 10, 2024
91e72f7
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 10, 2024
870289a
add: only submit basic python jobs through a nested run_chain
leclairm Jan 11, 2024
0b5e17a
Merge branch 'async' of github.com:C2SM/processing-chain into async
leclairm Jan 11, 2024
208e9e0
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 11, 2024
623e435
fix: loop over jobs in run_chunk
leclairm Jan 11, 2024
ceb1297
Merge branch 'async' of github.com:C2SM/processing-chain into async
leclairm Jan 11, 2024
b39e27d
ref: job_id becomes chunk_id
leclairm Jan 11, 2024
8962742
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 11, 2024
1c56a68
fix: leftover `job_id` -> `chunk_id`
leclairm Jan 11, 2024
71e7084
Merge branch 'async' of github.com:C2SM/processing-chain into async
leclairm Jan 11, 2024
da105eb
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 11, 2024
f853d98
add(untested): Slurm monitoring
leclairm Jan 11, 2024
986f3a0
Merge branch 'async' of github.com:C2SM/processing-chain into async
leclairm Jan 11, 2024
622ecf0
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 11, 2024
8d79b66
Add BASIC_PYHTON_JOB to missing jobs
mjaehn Jan 11, 2024
4391856
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 11, 2024
6389bdc
Fix function arguments
mjaehn Jan 11, 2024
29fcbc0
Small fix and hint to KeyError
mjaehn Jan 11, 2024
7de963d
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 11, 2024
87edb3b
ref(slurm summary): streamline code a bit
leclairm Jan 12, 2024
f4bf94d
Merge branch 'async' of github.com:C2SM/processing-chain into async
leclairm Jan 12, 2024
390b499
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 12, 2024
dffb13e
Some settings for cosmo-ghg case
mjaehn Jan 12, 2024
9da09f4
afterany -> afterok for wait job
mjaehn Jan 12, 2024
38d40ce
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 12, 2024
4f40ec0
Comment some function calls
mjaehn Jan 12, 2024
c1b1ca9
add:ref: remove unused `info_requests` + print failing jobs
leclairm Jan 15, 2024
bbb41be
Merge branch 'async' of github.com:C2SM/processing-chain into async
leclairm Jan 15, 2024
f001103
fix: job summary for previous chunk, not current
leclairm Jan 15, 2024
91e2251
fix: reactivate empty current job ids at beginning of chunk
leclairm Jan 15, 2024
1cfcd20
fix: only wait and monitor if not very first chunk
leclairm Jan 15, 2024
db2b393
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 15, 2024
3d0fb48
ref: move icon-art error handling in slurm job itself
leclairm Jan 15, 2024
5f87d55
Merge branch 'async' of github.com:C2SM/processing-chain into async
leclairm Jan 15, 2024
648eb8d
fix: remove BASIC_PYTHON_JOB workaround for icon
leclairm Jan 15, 2024
97b6cff
fix(icon.py): escape curly brackets for string formatting
leclairm Jan 15, 2024
6c3b540
fix(icon): escape curly brackets in python way
leclairm Jan 15, 2024
ea59d35
Add walltime and remove conda activation
mjaehn Jan 15, 2024
b14e29f
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 15, 2024
4a50fa0
Set walltime correctly
mjaehn Jan 15, 2024
51c06c3
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 15, 2024
e9f4d71
Fix call to handle_error function
mjaehn Jan 16, 2024
16d5b2d
Set walltimes for icon workflows
mjaehn Jan 16, 2024
2b99c66
Debug
mjaehn Jan 16, 2024
eb3d856
Remove job_ids override
mjaehn Jan 16, 2024
c5cce6a
ref(config): clean up
leclairm Jan 16, 2024
4a08212
Merge branch 'async' of github.com:C2SM/processing-chain into async
leclairm Jan 16, 2024
dee4c0e
Add smaller walltime to wait job
mjaehn Jan 16, 2024
726be9b
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 16, 2024
add761e
Remove prints
mjaehn Jan 16, 2024
780c66d
Some fixes for cosmo case
mjaehn Jan 16, 2024
0e6de77
Remove model check
mjaehn Jan 16, 2024
2e6a45f
Don't pass logfile anymore
mjaehn Jan 16, 2024
6f88cdf
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 16, 2024
5b560a0
Submit int2lm and cosmo jobs correctly
mjaehn Jan 16, 2024
5c8d9c8
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 16, 2024
7e4e810
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 16, 2024
d615f73
Small fixes for int2lm and cosmo
mjaehn Jan 16, 2024
2c72a73
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 16, 2024
a8b1c09
Store job scripts in separate directory
mjaehn Jan 18, 2024
763efee
Add BASIC_PYTHON_JOB to int2lm
mjaehn Jan 18, 2024
089ec31
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 18, 2024
7800847
Add post_cosmo dependency
mjaehn Jan 18, 2024
cfbadb7
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 18, 2024
019f8f7
Bugfixes for cosmo jobs
mjaehn Jan 18, 2024
21d3256
Further fixes
mjaehn Jan 18, 2024
4f87ba3
Make post_cosmo a submit job
mjaehn Jan 18, 2024
4e79263
Fix post_cosmo
mjaehn Jan 18, 2024
58727f4
Fix post_cosmo
mjaehn Jan 18, 2024
5ebefb5
ref: refactor cycling and monitoring
leclairm Jan 25, 2024
1f0f524
Merge branch 'async' of github.com:C2SM/processing-chain into async
leclairm Jan 25, 2024
f5be54c
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 25, 2024
1a3629a
Fix config variables
mjaehn Jan 26, 2024
f51a336
Fix log file output
mjaehn Jan 26, 2024
30a81a0
Formatting
mjaehn Jan 26, 2024
d603c07
Remove time logging from jobs
mjaehn Jan 26, 2024
532af08
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 26, 2024
70204ca
Fix for icon job
mjaehn Jan 26, 2024
0556121
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 26, 2024
58fd928
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 26, 2024
8810dcf
ref: exception handling in Config.submit()
leclairm Jan 29, 2024
1c4d9f0
Change table cell widths
mjaehn Jan 29, 2024
f036e43
Remove old way of logging
mjaehn Jan 29, 2024
58fc17c
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 29, 2024
bb51303
Merge branch 'v3.1-rc' of github.com:C2SM/processing-chain into async
mjaehn Jan 29, 2024
19d45b0
Cleanup
mjaehn Jan 29, 2024
fb19c02
NNodes -> N
mjaehn Jan 29, 2024
7b618ee
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 29, 2024
5a314f4
Fix for N/NNodes
mjaehn Jan 29, 2024
7933fb4
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 29, 2024
c4f30ae
Yet another fix for N/NNodes
mjaehn Jan 29, 2024
ce9369b
Simplify job names
mjaehn Jan 29, 2024
86b5f68
Change missing job names
mjaehn Jan 29, 2024
4a59dd8
Merge branch 'async' of github.com:C2SM/processing-chain into async
leclairm Jan 29, 2024
97b2fa5
Fix syntax error from commit 8810dcf
mjaehn Jan 30, 2024
4b01685
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 30, 2024
3a65efe
Just jobname for jobs
mjaehn Jan 30, 2024
90793ab
Cleaner console output
mjaehn Jan 30, 2024
f8551b9
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 30, 2024
06729b4
Unify restart and spinup runs
mjaehn Jan 30, 2024
2d3fb0b
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 30, 2024
7463009
Complete config file for spinup test
mjaehn Jan 30, 2024
28ac52c
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 30, 2024
44e3c74
Fix for spinup
mjaehn Jan 30, 2024
79d5f94
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 30, 2024
287ff6b
Fix 2 for spinup
mjaehn Jan 30, 2024
365cad8
Fancy formatting
mjaehn Jan 30, 2024
c7f6826
Define separate spinup workflow
mjaehn Jan 30, 2024
5afb56e
Fix formatting (hopefully)
mjaehn Jan 30, 2024
0cf4a03
Aligning case + workflow
mjaehn Jan 30, 2024
b1448d3
Custom workflow_name and improve check
mjaehn Jan 30, 2024
2831e98
Remove workflow_name check in jobs
mjaehn Jan 30, 2024
126b9c1
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 30, 2024
74cd9f8
Bugfix in cosmo job
mjaehn Jan 30, 2024
0cd732e
Remove restart info
mjaehn Jan 30, 2024
a676d75
Compute chunks separately
mjaehn Jan 30, 2024
99e5cba
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 30, 2024
f1d3caf
Add function to get previous chunk ID
mjaehn Jan 30, 2024
077dc47
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 30, 2024
121c44f
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 30, 2024
38e2c36
Directly get previous chunk id
mjaehn Jan 30, 2024
0a42d3e
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 30, 2024
59521fa
Further cleanup and refactoring
mjaehn Jan 30, 2024
c9846be
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 30, 2024
8f700a1
Fix for chunk_id_prev
mjaehn Jan 30, 2024
e8fd4a7
Merge branch 'async' of github.com:C2SM/processing-chain into async
mjaehn Jan 30, 2024
f223a7c
Fix
mjaehn Jan 30, 2024
3c79da1
Save total chunk list
mjaehn Jan 31, 2024
03370b5
Fix restart variables
mjaehn Jan 31, 2024
45eec53
Don't print chunk list twice
mjaehn Jan 31, 2024
5174a92
Fix for cosmo_restart_out
mjaehn Jan 31, 2024
7505cc0
Remove sequential part
mjaehn Jan 31, 2024
951ad52
GitHub Action: Apply Pep8-formatting
invalid-email-address Jan 31, 2024
3fe7fcc
Remove sequential case
mjaehn Jan 31, 2024
ee98de1
Add dependencies to all workflows
mjaehn Jan 31, 2024
f4533bb
Remove is_async config variable
mjaehn Jan 31, 2024
c5e5457
Include renaming scheme for chemicals for OEM
efmkoene Feb 5, 2024
6ede4bb
GitHub Action: Apply Pep8-formatting
invalid-email-address Feb 5, 2024
2140049
re-order in and output tracers
efmkoene Feb 5, 2024
0325f66
Shortened variable name
efmkoene Feb 5, 2024
483193b
GitHub Action: Apply Pep8-formatting
invalid-email-address Feb 5, 2024
c32d633
Merge branch 'v3.1-rc' of github.com:C2SM/processing-chain into chemi…
mjaehn Feb 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions cases/icon-art-oem-test/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ chem:
nameformat: '%Y%m%d_%H'
suffix: .grb
inc: 3
remap_tracers:
CH4_BG: TRCH4_chemtr

icontools_runjobs:
- icontools_remap_ic_runjob.cfg
Expand Down
5 changes: 4 additions & 1 deletion jobs/prepare_art_oem.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,10 @@ def main(cfg):
ds_chem['PS'] = ds_chem['LNPS']
ds_chem['PS'].attrs = ds_chem['LNPS'].attrs
ds_chem['PS'].attrs["long_name"] = 'surface pressure'
ds_chem['TRCH4_chemtr'] = ds_chem['CH4_BG']
# Remapping chemical tracer names
if "remap_tracers" in cfg.chem:
for chem_in, chem_out in cfg.chem['remap_tracers'].items():
ds_chem[chem_out] = ds_chem[chem_in]
# merge:
ds_merged = xr.merge([ds_meteo, ds_chem], compat="override")
#ds_merged.attrs = ds.attrs
Expand Down
Loading