Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Features/#450 add metadata #568

Open
wants to merge 117 commits into
base: dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 28 commits
Commits
Show all changes
117 commits
Select commit Hold shift + click to select a range
f981236
Add function for list of sources
ClaraBuettner Dec 13, 2021
e21500a
Add BGR datasource
fwitte Dec 13, 2021
4ab5946
Add osm, mastr, hotmaps to sources dict
IlkaCu Dec 13, 2021
648ab00
Merge branch 'features/#450-add-metadata' of https://github.com/opene…
IlkaCu Dec 13, 2021
a4eb6bc
Insert individual sources for BGR and openffe gas
fwitte Dec 13, 2021
a994a6e
Sort sources alphabetically
ClaraBuettner Dec 13, 2021
c6612fc
Add demandregio, nep, era5, tyndp, peta to sources list
ClaraBuettner Dec 13, 2021
0d886f8
Add authors
IlkaCu Dec 13, 2021
9213ac8
Merge branch 'features/#450-add-metadata' of https://github.com/opene…
IlkaCu Dec 13, 2021
0d78d2b
Add seenergies and egon-data to sources dict
IlkaCu Dec 13, 2021
e6edf5e
Add missing }
ClaraBuettner Dec 13, 2021
99262f6
Add metadata for scenario capacities
ClaraBuettner Dec 13, 2021
80df673
Add licenses in list
ClaraBuettner Dec 13, 2021
c14d16b
Add documentation to add_metadata
ClaraBuettner Dec 13, 2021
60d13a1
Add Scigrid_gas, Einspeiseatlas and pipepline_classification to sourc…
Dec 13, 2021
6262699
Add egon-data to metadata soutces
ClaraBuettner Dec 13, 2021
68a9165
Update metadata
ClaraBuettner Dec 13, 2021
be8a0f8
Merge branch 'features/#450-add-metadata' of https://github.com/opene…
ClaraBuettner Dec 13, 2021
7545890
Add metadata to renewable_feedin
ClaraBuettner Dec 13, 2021
e04a3b1
Update metadata for district_heating areas
ClaraBuettner Dec 13, 2021
55ff8a2
Define method to retrieve etrago table metadata
fwitte Dec 13, 2021
775cd54
Add metadata for zenszs map vg250
ClaraBuettner Dec 13, 2021
edee1f0
Add Storage key to components dict
fwitte Dec 14, 2021
41aab41
Merge remote-tracking branch 'origin/features/#449-add-etrago-metadat…
fwitte Dec 14, 2021
46a1605
Add contributor dict with example usage
fwitte Dec 15, 2021
95247fe
Fix example usage
fwitte Jan 19, 2022
f5a5b19
Apply black and isort
ClaraBuettner Mar 1, 2023
7ce6add
Merge branch 'dev' into features/#450-add-metadata
ClaraBuettner Mar 1, 2023
0277794
Adapt contributors function
AmeliaNadal Mar 6, 2023
8805523
Apply black and isort
AmeliaNadal Mar 6, 2023
196bff7
Rename 'license' to 'licenses'
ClaraBuettner Mar 13, 2023
4fbe7d1
Fix name
ClaraBuettner Mar 13, 2023
066f317
Fill metadata for static tables
ClaraBuettner Mar 13, 2023
2347b06
Apply black and isort
ClaraBuettner Mar 13, 2023
8feefdd
Add gas sector sources and contributors
AmeliaNadal Mar 13, 2023
b65f717
Adjust metadata sources
AmeliaNadal Mar 13, 2023
7bee98e
Add metadata for gas voronois and gas mapping tables
AmeliaNadal Mar 14, 2023
80b8687
Add technology data and dsm source
ClaraBuettner Mar 14, 2023
e341fff
Add metadata for generaotr and bus timeseries tables
ClaraBuettner Mar 14, 2023
83ad9bf
Add sources for gas metadata
AmeliaNadal Mar 15, 2023
d8f8ea1
Remane 'license' to 'licenses'
AmeliaNadal Mar 15, 2023
069e284
add DL-DE-BY-2.0 license to metadata.py
khelfen Mar 15, 2023
9123cc7
Add time series meta data automised
ClaraBuettner Mar 15, 2023
19ad658
Delete unused imports
ClaraBuettner Mar 15, 2023
f857b09
Add dsm sources
ClaraBuettner Mar 15, 2023
baa66ad
Apply black and isort
ClaraBuettner Mar 15, 2023
082dbd1
Add vg250 to generator sources
ClaraBuettner Mar 15, 2023
2c6f96e
Update default values for meta data
ClaraBuettner Mar 15, 2023
0be7172
Fix licenses list in metadata
ClaraBuettner Mar 15, 2023
928b611
Change contributors of egon-data source
ClaraBuettner Mar 15, 2023
01df06f
Add era5 sources to line timeseries metadata
ClaraBuettner Mar 15, 2023
884cce9
Remove unused sources
ClaraBuettner Mar 15, 2023
9dee3d0
Merge pull request #1117 from openego/features/#450-metadata-etrago-t…
ClaraBuettner Mar 15, 2023
191c319
add me to contributors function in metadata.py
khelfen Mar 16, 2023
93508f4
Add data bundle as source
IlkaCu Mar 16, 2023
acdea7f
Update license for etrago tables
ClaraBuettner Mar 16, 2023
21d0f7a
Fix metadata for scenario capacities
ClaraBuettner Mar 16, 2023
3a925dc
Delete quotation marks from metadata entry
ClaraBuettner Mar 16, 2023
c39afdf
Remove unused imports
ClaraBuettner Mar 16, 2023
6e7155f
Merge branch 'features/#450-add-metadata' into features/#450-metadata…
AmeliaNadal Mar 16, 2023
3bcdd78
Merge branch 'dev' into features/#450-add-metadata
khelfen Mar 16, 2023
c8fcad1
Adapt license to odbl
AmeliaNadal Mar 16, 2023
8e68920
Add metadata
ClaraBuettner Mar 16, 2023
4ab049c
Apply black and isort
ClaraBuettner Mar 16, 2023
e706d7e
Add metadata
ClaraBuettner Mar 16, 2023
227a735
Apply black and isort
ClaraBuettner Mar 16, 2023
abb5d11
Add metadata
ClaraBuettner Mar 16, 2023
a39ce0c
Apply black and isort
ClaraBuettner Mar 16, 2023
3642f3b
Merge branch 'features/#450-add-metadata' into features/#450-metadata…
AmeliaNadal Mar 17, 2023
a65f9b0
Merge pull request #1118 from openego/features/#450-metadata_gas
AmeliaNadal Mar 17, 2023
1f8f27c
Add json metadata upload
nailend Mar 17, 2023
0397f51
Black&isort
nailend Mar 17, 2023
3692c2e
Add alternative download and unzip
nailend Mar 17, 2023
b0554a2
Add json metadata files to repo
nailend Mar 17, 2023
955e9eb
Adjust upload to files in repo
nailend Mar 17, 2023
e78ecde
Adjust dependencies
nailend Mar 17, 2023
e79ebaa
Merge pull request #1123 from openego/features/#450-metadata_nailend
nailend Mar 17, 2023
bb63c72
Fix typo
nailend Mar 17, 2023
b44c012
Update imports
nailend Mar 17, 2023
f767c38
Add metadata for power plants table
ClaraBuettner Mar 17, 2023
22d54d1
Merge pull request #1121 from openego/features/#450-add-metadata-heat…
ClaraBuettner Mar 17, 2023
dd7f2df
Fix calling meta data function
ClaraBuettner Mar 17, 2023
21e6c59
Run black on "heat_demand_timeseries/__init__.py"
gnn Mar 18, 2023
53e43ee
Remove unused imports
gnn Mar 18, 2023
7005675
Wrap overly long lines
gnn Mar 18, 2023
0d09fd5
Replace hard tabs with spaces
gnn Mar 18, 2023
83b03aa
Remove trailing spaces
gnn Mar 18, 2023
0206131
Use normal strings when there is no substitution
gnn Mar 18, 2023
4ad0ec3
Merge printed expression into f-string
gnn Mar 18, 2023
cef725e
Fix docstring
gnn Mar 18, 2023
cfa2e36
Fix assertion messages
gnn Mar 18, 2023
ad79e64
Fix `datetime` usage
gnn Mar 18, 2023
3bc50ac
Run `black` on "src/egon/data/metadata.py"
gnn Mar 19, 2023
1294a0f
Run `isort` on "src/egon/data/metadata.py"
gnn Mar 19, 2023
c86f814
Move separating space to the beginning of the line
gnn Mar 19, 2023
f10d76e
Make long dictionary string values more readable
gnn Mar 19, 2023
2045d84
Wrap long strings to 72 characters
gnn Mar 19, 2023
31b3f50
Fix "zensus" URL
gnn Mar 19, 2023
c0f3485
Don't convert `Path` object to string
gnn Mar 19, 2023
408efaa
Invert `file.endswith(".json")` test
gnn Mar 19, 2023
a895844
Make logging message more precise
gnn Mar 19, 2023
b84d5ac
Replace string concatenation with an f-string
gnn Mar 19, 2023
c0e244e
Fix incomplete file paths
gnn Mar 19, 2023
86fa23e
Reformat "json_metadata/*.json" files
gnn Mar 22, 2023
9ae851c
Removed parts of metadata which lead to omi error
nailend Mar 24, 2023
4d53166
Readd start and end information
nailend Mar 24, 2023
1275fa8
Replace start and end with isodates for the whole year
nailend Mar 24, 2023
dfe1d8e
Move duplicate metadata entries into variables
gnn Mar 28, 2023
7a2a21d
Convert 'egon.data.metadata' into a package
gnn Mar 28, 2023
ceec82f
Move JSON metadata files to the "metadata" package
gnn Mar 28, 2023
43c6420
Move dialect creation out of the loop
gnn Mar 28, 2023
15e6753
Fix typo in filename: "populaiton" -> "population"
gnn Mar 28, 2023
7b8abe0
Fix typo in filename: "fitered" -> "filtered"
gnn Mar 28, 2023
7c83cb2
Fix end timestep
nailend Mar 29, 2023
12f6665
Merge branch 'dev' into features/#450-add-metadata
ClaraBuettner Oct 10, 2023
c1f0966
Import missing package
ClaraBuettner Oct 10, 2023
bb1f08c
Import json package
ClaraBuettner Oct 11, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AUTHORS.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
Authors
=======

* Guido Pleßmann, Ilka Cußmann, Stephan Günther, Jonathan Amme, Julian Endres, Kilian Helfenbein - https://github.com/openego/eGon-data
* Jonathan Amme, Clara Büttner, Ilka Cußman, Julian Endres, Carlos Epia, Kilian Helfenbein, Stephan Günther, Ulf Müller, Amélia Nadal, Guido Pleßmann, Francesco Witte - https://github.com/openego/eGon-data
132 changes: 51 additions & 81 deletions src/egon/data/datasets/district_heating_areas/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,15 @@
)

# for metadata creation
import time
import datetime
from egon.data.metadata import (
context,
meta_metadata,
license_ccby,
sources,
generate_resource_fields_from_sqla_model,
)
import json

# import time
Expand All @@ -45,7 +54,7 @@ def __init__(self, dependencies):
super().__init__(
name="district-heating-areas",
# version=self.target_files + "_0.0",
version="0.0.1", # maybe rethink the naming
version="0.0.2", # maybe rethink the naming
dependencies=dependencies,
tasks=(create_tables, demarcation),
)
Expand Down Expand Up @@ -687,29 +696,12 @@ def add_metadata():
"""
Writes metadata JSON string into table comment.

TODO
----

Meta data must be check and adjusted to the egon_data standard:
- Add context
- authors and institutions

"""

# Prepare variables
license_district_heating_areas = [
{
# this could be the license of the "district_heating_areas"
"name": "Creative Commons Attribution 4.0 International",
"title": "CC BY 4.0",
"path": "https://creativecommons.org/licenses/by/4.0/",
"instruction": (
"You are free: To Share, To Adapt;"
" As long as you: Attribute!"
),
"attribution": "© Europa-Universität Flensburg", # if all agree
# "attribution": "© ZNES Flensburg", # alternative
}
license_ccby("© Europa-Universität Flensburg")
]

# Metadata creation for district heating areas (polygons)
Expand All @@ -719,25 +711,14 @@ def add_metadata():
"description": "Modelled future district heating areas for "
"the supply of residential and service-sector heat demands",
"language": ["EN"],
"publicationDate": datetime.date.today().isoformat(),
"context": context(),
"spatial": {"location": "", "extent": "Germany", "resolution": ""},
"temporal": {
"referenceDate": "scenario-specific",
"timeseries": {
"start": "",
"end": "",
"resolution": "",
"alignment": "",
"aggregationType": "",
},
},
"sources": [
{
# eGon scenario specific heat demand distribution based
# on Peta5_0_1, using vg250 boundaries
},
{
# Census gridded apartment data
},
sources()["peta"],
sources()["egon-data"],
sources()["zensus"],
sources()["vg250"],
],
"resources": [
{
Expand Down Expand Up @@ -796,21 +777,21 @@ def add_metadata():
"licenses": license_district_heating_areas,
"contributors": [
{
"title": "Eva, Clara",
"email": "",
"date": "2021-05-07",
"object": "",
"comment": "Processed data",
}
],
"metaMetadata": { # https://github.com/OpenEnergyPlatform/oemetadata
"metadataVersion": "OEP-1.4.0",
"metadataLicense": {
"name": "CC0-1.0",
"title": "Creative Commons Zero v1.0 Universal",
"path": ("https://creativecommons.org/publicdomain/zero/1.0/"),
"title": "EvaWie",
"email": "http://github.com/EvaWie",
"date": time.strftime("%Y-%m-%d"),
"object": None,
"comment": "Imported data",
},
},
{
"title": "Clara Büttner",
"email": "http://github.com/ClaraBuettner",
"date": time.strftime("%Y-%m-%d"),
"object": None,
"comment": "Updated metadata",
},
],
"metaMetadata": meta_metadata(),
}
meta_json = "'" + json.dumps(meta) + "'"

Expand All @@ -824,25 +805,14 @@ def add_metadata():
" for supply of residential and service-sector heat demands"
" assigned to zensus_population_ids",
"language": ["EN"],
"publicationDate": datetime.date.today().isoformat(),
"context": context(),
"spatial": {"location": "", "extent": "Germany", "resolution": ""},
"temporal": {
"referenceDate": "scenario-specific",
"timeseries": {
"start": "",
"end": "",
"resolution": "",
"alignment": "",
"aggregationType": "",
},
},
"sources": [
{
# eGon scenario specific heat demand distribution based
# on Peta5_0_1, using vg250 boundaries
},
{
# Census gridded apartment data
},
sources()["peta"],
sources()["egon-data"],
sources()["zensus"],
sources()["vg250"],
],
# Add the license for the map table
"resources": [
Expand Down Expand Up @@ -897,21 +867,21 @@ def add_metadata():
"licenses": license_district_heating_areas,
"contributors": [
{
"title": "Eva, Clara",
"email": "",
"date": "2021-05-07",
"object": "",
"comment": "Processed data",
}
],
"metaMetadata": { # https://github.com/OpenEnergyPlatform/oemetadata
"metadataVersion": "OEP-1.4.0",
"metadataLicense": {
"name": "CC0-1.0",
"title": "Creative Commons Zero v1.0 Universal",
"path": ("https://creativecommons.org/publicdomain/zero/1.0/"),
"title": "EvaWie",
"email": "http://github.com/EvaWie",
"date": time.strftime("%Y-%m-%d"),
"object": None,
"comment": "Imported data",
},
},
{
"title": "Clara Büttner",
"email": "http://github.com/ClaraBuettner",
"date": time.strftime("%Y-%m-%d"),
"object": None,
"comment": "Updated metadata",
},
],
"metaMetadata": meta_metadata(),
}
meta_json = "'" + json.dumps(meta) + "'"

Expand Down
Loading