Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support area masking of boundary forcing #52

Closed
wants to merge 181 commits into from
Closed
Show file tree
Hide file tree
Changes from 173 commits
Commits
Show all changes
181 commits
Select commit Hold shift + click to select a range
38f8d15
feat: Initial implementation of global graphs
theissenhelen Jun 24, 2024
9dc2cec
add dependencies
theissenhelen Jun 25, 2024
f1fe18f
add cli command
JPXKQX Jun 25, 2024
b8b558d
Ignore .pt files
JPXKQX Jun 25, 2024
7f6f4bd
run pre-commit
JPXKQX Jun 25, 2024
d5f67fd
docstring + log erros
JPXKQX Jun 26, 2024
b12272d
initial tests
JPXKQX Jun 26, 2024
cce5ea6
feat: initial version of AttributeBuilder
theissenhelen Jun 26, 2024
9ba0391
refactor: separate into node edge attribute builders
theissenhelen Jun 26, 2024
9a47184
feat: edge_length moved to edges/attributes.py
JPXKQX Jun 27, 2024
384adc7
remove __init__
JPXKQX Jun 27, 2024
0bc176c
feat: test edge builders
JPXKQX Jun 27, 2024
d16934b
add blank lines
JPXKQX Jun 27, 2024
0f82ea7
dep: hydra-core
JPXKQX Jun 27, 2024
a9c5ada
bugfix (encoder edge lengths) + refector
JPXKQX Jun 27, 2024
66ef5dc
deps: == to >=
JPXKQX Jun 27, 2024
fcd0b74
rename node builder classes
JPXKQX Jun 27, 2024
b28b0ff
fix: tests
JPXKQX Jun 27, 2024
ec8c9c5
feat: support path and dict for `config` argument
JPXKQX Jun 28, 2024
9b9d805
fix: error
JPXKQX Jun 28, 2024
2b67bf3
refactor: naming
theissenhelen Jul 1, 2024
cdeaa03
fix: pre-commit
theissenhelen Jul 1, 2024
f07434c
feat: builders icosahedral
theissenhelen Jun 28, 2024
52403d7
feat: Add icosahedral graph generation
theissenhelen Jun 28, 2024
2ef63c2
refactor: remove create_shere
theissenhelen Jun 28, 2024
1fe76bb
feat: Icosahedral edge builder
theissenhelen Jun 28, 2024
fde0fe6
feat: hexagonal graph generation
theissenhelen Jun 28, 2024
97ef0da
feat: hexagonal builders
theissenhelen Jun 28, 2024
8dcda40
fix: AOI not implemented yet
theissenhelen Jun 28, 2024
63be6af
fix: abstractmethod and renaming
theissenhelen Jul 1, 2024
b175585
chore: add dependencies
theissenhelen Jul 1, 2024
86c5e35
test: add tests for trimesh
theissenhelen Jul 1, 2024
19461a1
test: add tests for hex (h3)
theissenhelen Jul 1, 2024
39ee3ad
fix: imports
theissenhelen Jul 1, 2024
f00fd72
fix: output type
theissenhelen Jul 1, 2024
75a82c8
refactor: delete unused file
theissenhelen Jul 1, 2024
f45b900
refactor: renaming and positioning
theissenhelen Jul 1, 2024
9f2c052
feat: ensure src and dst always the same
theissenhelen Jul 1, 2024
e410bf5
fix: imports
theissenhelen Jul 1, 2024
ef1c110
fix: edge_name not supported
theissenhelen Jul 1, 2024
2e6830f
test: add tests for TriIcosahedralEdges
theissenhelen Jul 1, 2024
a59f5d1
fix: assert missing for Hexagonal edges
theissenhelen Jul 2, 2024
59bac56
test: hexagonal edges
theissenhelen Jul 2, 2024
bd729c9
fix: avoid same name
theissenhelen Jul 2, 2024
9cce37a
feat: LimitedAreaZarrNodes
theissenhelen Jul 3, 2024
745709f
feat: add KNNMaskBuilder for use with LAM
theissenhelen Jul 3, 2024
bc735cd
feat: add KNNMaskBuilder to TriIcosahedral
theissenhelen Jul 3, 2024
03fbf9f
feat: AreaNPZFileNodes
theissenhelen Jul 3, 2024
3731e82
fix: KNNAreaMaskBuilder working with NPZ
theissenhelen Jul 3, 2024
ed64a7e
fix: imports and naming
theissenhelen Jul 3, 2024
1e2c37a
fix: TriIocsahedral working for area masks
theissenhelen Jul 5, 2024
c07c583
feat: debugging purposes
theissenhelen Jul 5, 2024
980ed8d
refactor: rename tests
theissenhelen Jul 5, 2024
3609681
Global Encoder-Processor-Decoder graph (#9)
JPXKQX Jul 5, 2024
a973c2d
fix: attributes as torch.float32
JPXKQX Jul 6, 2024
af111a6
new test: attributes must be float32
JPXKQX Jul 6, 2024
a8a1620
fix typo
JPXKQX Jul 6, 2024
926c75b
Homogeneize base builders
JPXKQX Jul 6, 2024
f342073
improve test docstrings
JPXKQX Jul 6, 2024
9d9fea8
homogeneize (name as class attribute)
JPXKQX Jul 6, 2024
1b20845
new input config
JPXKQX Jul 8, 2024
4e62431
new default
JPXKQX Jul 8, 2024
6825850
Merge branch 'develop' into fix/global-version
JPXKQX Jul 8, 2024
d25c47b
feat: Initial implementation of global graphs
theissenhelen Jun 24, 2024
8f0415e
add cli command
JPXKQX Jun 25, 2024
22fba0e
Ignore .pt files
JPXKQX Jun 25, 2024
2879c7c
run pre-commit
JPXKQX Jun 25, 2024
1ca0633
docstring + log erros
JPXKQX Jun 26, 2024
277c231
initial tests
JPXKQX Jun 26, 2024
7a4e9cb
feat: initial version of AttributeBuilder
theissenhelen Jun 26, 2024
96a6ed5
refactor: separate into node edge attribute builders
theissenhelen Jun 26, 2024
1c88ee8
feat: edge_length moved to edges/attributes.py
JPXKQX Jun 27, 2024
92988d7
remove __init__
JPXKQX Jun 27, 2024
de07488
bugfix (encoder edge lengths) + refector
JPXKQX Jun 27, 2024
c900bfd
feat: support path and dict for `config` argument
JPXKQX Jun 28, 2024
9768204
fix: error
JPXKQX Jun 28, 2024
74ef4cc
refactor: naming
theissenhelen Jul 1, 2024
91e15b3
fix: pre-commit
theissenhelen Jul 1, 2024
66d3aa1
feat: builders icosahedral
theissenhelen Jun 28, 2024
23b3110
feat: Add icosahedral graph generation
theissenhelen Jun 28, 2024
469b56b
refactor: remove create_shere
theissenhelen Jun 28, 2024
4e864c3
feat: Icosahedral edge builder
theissenhelen Jun 28, 2024
d573905
feat: hexagonal graph generation
theissenhelen Jun 28, 2024
edd9c43
feat: hexagonal builders
theissenhelen Jun 28, 2024
b7002e3
fix: AOI not implemented yet
theissenhelen Jun 28, 2024
1d55ef6
fix: abstractmethod and renaming
theissenhelen Jul 1, 2024
7dcfb98
chore: add dependencies
theissenhelen Jul 1, 2024
a35b76f
test: add tests for trimesh
theissenhelen Jul 1, 2024
bfc3e84
test: add tests for hex (h3)
theissenhelen Jul 1, 2024
fc2f707
fix: imports
theissenhelen Jul 1, 2024
7c3dca3
fix: output type
theissenhelen Jul 1, 2024
d627e58
refactor: delete unused file
theissenhelen Jul 1, 2024
735fbc1
refactor: renaming and positioning
theissenhelen Jul 1, 2024
46a2c07
feat: ensure src and dst always the same
theissenhelen Jul 1, 2024
c9e4fde
fix: imports
theissenhelen Jul 1, 2024
7a8b316
fix: edge_name not supported
theissenhelen Jul 1, 2024
02496ec
test: add tests for TriIcosahedralEdges
theissenhelen Jul 1, 2024
59fcad3
fix: assert missing for Hexagonal edges
theissenhelen Jul 2, 2024
b41f88e
test: hexagonal edges
theissenhelen Jul 2, 2024
5a43185
fix: avoid same name
theissenhelen Jul 2, 2024
a0259f8
fix: imports
theissenhelen Jul 8, 2024
7289e32
fix: conflicts
theissenhelen Jul 8, 2024
9a6decc
Merge branch 'develop' into 6-generate-graphs-from-icosahedral-meshes
theissenhelen Jul 8, 2024
4ca717b
update tests
JPXKQX Jul 9, 2024
fb93b8b
Merge branch 'develop' into 6-generate-graphs-from-icosahedral-meshes
theissenhelen Jul 9, 2024
e44ae0d
Merge branch '6-generate-graphs-from-icosahedral-meshes' into 7-local…
theissenhelen Jul 9, 2024
fe8a8e5
Include xhops to hexagonal edges
JPXKQX Jul 9, 2024
5865b35
Merge pull request #14 from ecmwf/hotfix/config_node_name
JPXKQX Jul 9, 2024
633091f
Merge remote-tracking branch 'origin/develop' into 6-generate-graphs-…
theissenhelen Jul 9, 2024
b2820e2
Merge remote-tracking branch 'origin/develop' into 6-generate-graphs-…
theissenhelen Jul 9, 2024
463911c
docs: update docstrings
theissenhelen Jul 9, 2024
ae5a8a7
Merge remote-tracking branch 'origin/develop' into 7-local-area-model…
theissenhelen Jul 9, 2024
b0a35b8
fix: update attribute name
JPXKQX Jul 10, 2024
4f445a9
refactor: rename multiscale nodes
theissenhelen Jul 11, 2024
fa812eb
refactor: rename icosahedral nodes
theissenhelen Jul 11, 2024
c7bd45c
Merge remote-tracking branch 'origin/6-generate-graphs-from-icosahedr…
theissenhelen Jul 11, 2024
d190758
refactor: LimitedArea prefix
theissenhelen Jul 11, 2024
dca3729
feat: add aoi_mask_builder to edge builder
theissenhelen Jul 11, 2024
86a5ce5
Merge remote-tracking branch 'origin/develop' into 7-local-area-model…
JPXKQX Jul 26, 2024
e73c203
Merge branch 'develop' into 7-local-area-modelling-graphs
theissenhelen Aug 5, 2024
3b26473
Merge branch 'develop' into 7-local-area-modelling-graphs
JPXKQX Aug 6, 2024
9d015a3
docs & default values
JPXKQX Aug 6, 2024
41bd3f5
create LimitedAreaHEALPixNodes
JPXKQX Aug 6, 2024
030168c
fix: import HEALPixNodes
JPXKQX Aug 6, 2024
9493933
fix: avoid runtimeError when deleting a key
JPXKQX Aug 6, 2024
986efe7
fix: set config arg to pathlib.Path
JPXKQX Aug 6, 2024
c9c736d
more logging
JPXKQX Aug 6, 2024
6e571b3
create LimitedAreaIcoshaderalNodes
JPXKQX Aug 6, 2024
81a3419
refactor LimiteAreaIcosahedralNodes class
JPXKQX Aug 6, 2024
8460e06
types & docstrings
JPXKQX Aug 6, 2024
c35c4ff
fix(lam): icosahedral nodes in lam
JPXKQX Aug 7, 2024
60118dd
fix: style
JPXKQX Aug 7, 2024
6c0585e
feat: icosahedral & hexagonal lam multiscale edges for lam
JPXKQX Aug 7, 2024
be72653
fixs(docs): typo
JPXKQX Aug 8, 2024
dcbd4ed
fix: remove redundant code
JPXKQX Aug 9, 2024
e2cb2b4
refactor: remove edge attr computation during graph creation
JPXKQX Aug 9, 2024
74533d4
refactor: split builder.py in several files
JPXKQX Aug 9, 2024
045ab09
fix: rename node builder class
JPXKQX Aug 9, 2024
2137f10
Merge branch 'develop' into 7-local-area-modelling-graphs
JPXKQX Aug 9, 2024
18dc2a3
fix: test imports
JPXKQX Aug 9, 2024
fee2160
Updated CHANGELOG.md
JPXKQX Aug 9, 2024
6a19a62
fix: imports
JPXKQX Aug 9, 2024
b4df296
fix: move masks.py to generate/
JPXKQX Aug 9, 2024
5d54f2f
refactor: update resolutions argument to resolution
JPXKQX Aug 9, 2024
c3dc568
Merge branch 'develop' into 7-local-area-modelling-graphs
JPXKQX Aug 9, 2024
e641d74
Merge branch 'develop' into 7-local-area-modelling-graphs
JPXKQX Aug 15, 2024
b9ae468
Merge 'develop' branch into 7-local-area-modelling-graphs
JPXKQX Aug 19, 2024
ddef154
fix: import annotations (py3.9)
JPXKQX Aug 19, 2024
6acfc32
tests: new tests for CutOutZarrDatasetNodes
JPXKQX Aug 19, 2024
36aa407
test: new test for KNNAreaMaskBuilder
JPXKQX Aug 19, 2024
d6a8a20
feat: mask_attr_name should be optional
JPXKQX Aug 21, 2024
7a53363
refactor: KNNAreaMAskBuilder
JPXKQX Aug 21, 2024
8383496
Merge branch 'develop' into 7-local-area-modelling-graphs
JPXKQX Aug 23, 2024
19e38b0
docs: remove duplication
theissenhelen Sep 2, 2024
5d247de
refactor: remove second return
theissenhelen Sep 2, 2024
ffb95a0
Merge branch 'develop' into 7-local-area-modelling-graphs
JPXKQX Sep 3, 2024
316cf7a
Merge branch '7-local-area-modelling-graphs' of github.com:ecmwf/anem…
JPXKQX Sep 3, 2024
ed6f960
docs: output folder missing
theissenhelen Sep 3, 2024
23a1a41
docs: update changelog
theissenhelen Sep 3, 2024
af1e520
docs: add PR to changelog
theissenhelen Sep 11, 2024
3b65701
docs: added missing PRs in changelog
theissenhelen Sep 11, 2024
524610a
fix: address @mchantry's comments
JPXKQX Sep 12, 2024
24ff1fc
fix: rename according to the refinement
JPXKQX Sep 12, 2024
51af55c
Merge branch '7-local-area-modelling-graphs' of github.com:ecmwf/anem…
JPXKQX Sep 12, 2024
e2601b4
fix: edge case 1 set of nodes with 1 node attribute
JPXKQX Sep 13, 2024
4762f3f
fix: delete repeated code
JPXKQX Sep 16, 2024
fac6f36
[Feature] Support node masking in edge builder (#50)
JPXKQX Sep 17, 2024
2fdb8c3
feat: support area masking of boundary forcing
JPXKQX Sep 24, 2024
669d201
docs: added CutOutZarrDatasetNodes docstring
JPXKQX Sep 24, 2024
0deb886
fix: cast to tuple
JPXKQX Sep 24, 2024
5efac37
Merge branch 'develop' into feature/support-forcing-area-in-cutout
JPXKQX Oct 3, 2024
996514d
feat: update CHANGELOG
JPXKQX Oct 3, 2024
b632221
feat: support None as default argument for forcing area
JPXKQX Oct 3, 2024
5dedb0a
feat: refactor args of CutOutZarrDatasetNodes
JPXKQX Oct 15, 2024
24891b9
Merge branch 'develop' into feature/support-forcing-area-in-cutout
JPXKQX Oct 16, 2024
820ca2f
fix: tests
JPXKQX Oct 16, 2024
d1dd700
fix: update changelog
JPXKQX Oct 16, 2024
5a0ee5e
fix: type hint
JPXKQX Oct 17, 2024
caa9002
Merge branch 'develop' into feature/support-forcing-area-in-cutout
JPXKQX Oct 21, 2024
76f42f3
Merge branch 'develop' into feature/support-forcing-area-in-cutout
JPXKQX Oct 23, 2024
d710a4b
Merge branch 'develop' into feature/support-forcing-area-in-cutout
JPXKQX Oct 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ Keep it human-readable, your future self will thank you!

### Changed
- ci: small fixes and updates pre-commit, downsteam-ci (#49)
- feat: New argument 'forcing_area' in the CutOutZarDatasetNodes class. (#52)

## [0.3.0 Anemoi-graphs, minor release](https://github.com/ecmwf/anemoi-graphs/compare/0.2.1...0.3.0) - 2024-09-03

Expand Down
49 changes: 43 additions & 6 deletions src/anemoi/graphs/nodes/builders/from_file.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,15 +52,52 @@ def get_coordinates(self) -> torch.Tensor:


class CutOutZarrDatasetNodes(ZarrDatasetNodes):
"""Nodes from Zarr dataset."""
"""Nodes from Zarr dataset.

Attributes
----------
lam_dataset : str
The limited area dataset.
forcing_dataset : str
The forcing dataset.
thinning : int, optional
The thinning factor. Defaults to 1, which means no thinning.
forcing_area : list[float], optional
The area of the forcing dataset. Specify the longitude and
latitudes boundaries as (north, west, south, east). Defaults
to None, which means the forcing dataset is not cropped.

Methods
-------
get_coordinates()
Get the lat-lon coordinates of the nodes.
register_nodes(graph, name)
Register the nodes in the graph.
register_attributes(graph, name, config)
Register the attributes in the nodes of the graph specified.
update_graph(graph, name, attr_config)
Update the graph with new nodes and attributes.
"""

def __init__(
self, name: str, lam_dataset: str, forcing_dataset: str, thinning: int = 1, adjust: str = "all"
self,
name: str,
lam_dataset: str,
forcing_dataset: str,
thinning: int = 1,
forcing_area: list[float] | None = None,
adjust: str = "all",
) -> None:
dataset_config = {
"cutout": [{"dataset": lam_dataset, "thinning": thinning}, {"dataset": forcing_dataset}],
"adjust": adjust,
}
lam_config = {"dataset": lam_dataset, "thinning": thinning}
forcing_config = {"dataset": forcing_dataset}
JPXKQX marked this conversation as resolved.
Show resolved Hide resolved

# Add area argument to crop the boundary forcing
if forcing_area is not None:
forcing_config["area"] = tuple(forcing_area)
assert len(forcing_area) == 4, "The forcing area must be a list of 4 elements (north, west, south, east)."
LOGGER.info("Forcing dataset is cropped to area: %s", forcing_area)

dataset_config = {"cutout": [lam_config, forcing_config], "adjust": adjust}
super().__init__(dataset_config, name)
self.n_cutout, self.n_other = self.dataset.grids

Expand Down
Loading