Skip to content

Commit

Permalink
Add initial files from init-repo-code
Browse files Browse the repository at this point in the history
Add files from `examples` directory.

Add previous `example_config.yaml` and modify that it works with new pydantic validation.

Add Python gitignore template from GitHub.

First version to make `to_core_*` methods work

Starting from the pydantic config representations. Ideally, I think one
should rather have `from_config_*` methods in `core.py`, rather than
`to_core_*` methods in `_yaml_data_models.py`, but chose this route as
it's simpler to adapt for now.

Acctually add fixed `example_config.yaml`

clean up
  • Loading branch information
GeigerJ2 authored and agoscinski committed Oct 16, 2024
1 parent 84e1e8a commit f11249d
Show file tree
Hide file tree
Showing 25 changed files with 1,499 additions and 19 deletions.
163 changes: 163 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/latest/usage/project/#working-with-version-control
.pdm.toml
.pdm-python
.pdm-build/

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/

12 changes: 12 additions & 0 deletions examples/README.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
Running examples
================

.. marker-example-text
To run the example you can use the command

.. code-block:: bash
wcflow files/configs/example_config.yaml
It is important to execute it from the examples folder as the relative paths are hardcoded.
182 changes: 182 additions & 0 deletions examples/files/configs/example_config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,182 @@
---
start_date: "2026-01-01T00:00"
end_date: "2027-01-01T00:00"
cycles:
- init:
tasks:
- extpar:
inputs:
- obs_data:
arg_option: "--input"
outputs:
- extpar_file
- icon_bimonthly:
period: "P2M"
tasks:
- preproc:
inputs:
- grid_file:
arg_option: "-g"
- extpar_file:
date: "2026-01-01T00:00"
arg_option: "-p"
- ERA5:
arg_option: "-e"
outputs:
- icon_input
depends:
- icon:
lag: "-P4M"
- icon:
inputs:
- grid_file:
arg_option: "-g"
- icon_input:
arg_option: "--input"
#- icon_restart:
# lag: '-P2M'
outputs:
- stream_1
- stream_2
#- icon_restart
- postproc_1:
inputs:
- stream_1:
arg_option: "--input"
outputs:
- postout_1
- store_and_clean_1:
inputs:
- postout_1:
arg_option: "--input"
- stream_1:
arg_option: "--stream"
- icon_input:
arg_option: "--icon_input"
outputs:
- stored_data_1
- yearly:
period: "P1Y"
tasks:
- postproc_2:
inputs:
- stream_2:
lag:
- "P0M"
arg_option: "--input"
outputs:
- postout_2
- store_and_clean_2:
inputs:
- postout_2:
arg_option: "--input"
- stream_2:
lag:
- "P0M"
outputs:
- stored_data_2
# Each task and piece of data (input and output of tasks) used to
# define the graph is described in that section
tasks:
- ROOT:
# All tasks inherit the root task properties
host: santis
account: g110
- extpar:
plugin: extpar
command: $PWD/files/scripts/extpar
command_option: "--verbose" # todo implement support
config: path/to/namelists/dir
uenv:
squashfs: path/to/squashfs
mount_point: runtime/mount/point
nodes: 1
walltime: "00:02:00"
- preproc:
plugin: AiiDA Shell
command: $PWD/files/scripts/cleanup.sh
nodes: 4
walltime: "00:02:00"
config: path/to/config/dir
uenv:
squashfs: path/to/squashfs
mount_point: runtime/mount/point
- icon:
plugin: icon
command: $PWD/files/scripts/icon
nodes: 40
walltime: "23:59:59"
config: path/to/namelists/dir
uenv:
squashfs: path/to/squashfs
mount_point: runtime/mount/point
- postproc_1:
plugin: AiiDA Shell
command: $PWD/files/scripts/main_script_ocn.sh
nodes: 2
walltime: "00:05:00"
conda_env: path/to/yaml/env/file
uenv:
squashfs: path/to/squashfs
mount_point: runtime/mount/point
- postproc_2:
plugin: AiiDA Shell
command: $PWD/files/scripts/main_script_atm.sh
nodes: 2
walltime: "00:05:00"
src: path/to/src/dir
conda_env: path/to/yaml/env/file
uenv:
squashfs: path/to/squashfs
mount_point: runtime/mount/point
- store_and_clean_1:
plugin: AiiDA Shell
command: $PWD/files/scripts/post_clean.sh
nodes: 1
walltime: "00:01:00"
- store_and_clean_2:
plugin: AiiDA Shell
command: $PWD/files/scripts/post_clean.sh
nodes: 1
walltime: "00:01:00"
data:
- preproc_output:
type: file
src: $PWD/files/data/file.ncdf
- grid_file:
type: file
src: $PWD/files/data/grid
- obs_data:
type: file
src: $PWD/files/data/obs_data
- ERA5:
type: file
src: $PWD/files/data/era5
- extpar_file:
type: file
src: output
- icon_input:
type: file
src: output
- icon_restart:
type: file
format: ncdf
src: restart
- stream_1:
type: file
src: output_1
- stream_2:
type: file
src: output_2
- postout_1:
type: file
src: postout
- postout_2:
type: file
src: postout
- stored_data_1:
type: file
src: stored_data
- stored_data_2:
type: file
src: stored_data
Empty file added examples/files/data/era5
Empty file.
Empty file added examples/files/data/file.ncdf
Empty file.
Empty file added examples/files/data/grid
Empty file.
Empty file added examples/files/data/input
Empty file.
Empty file added examples/files/data/obs_data
Empty file.
Empty file added examples/files/scripts/argparse
Empty file.
9 changes: 9 additions & 0 deletions examples/files/scripts/cleanup.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#!/usr/bin/env python

def main():
# Main script execution continues here
print("Cleaning")


if __name__ == '__main__':
main()
1 change: 1 addition & 0 deletions examples/files/scripts/cleanup.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
echo "cleanup" > output
1 change: 1 addition & 0 deletions examples/files/scripts/extpar
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
echo "extpar" > output
4 changes: 4 additions & 0 deletions examples/files/scripts/icon
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
echo "icon" > restart
echo "icon" > output
echo "icon" > output_1
echo "icon" > output_2
Loading

0 comments on commit f11249d

Please sign in to comment.