Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Catalogue walk #755

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Catalogue walk #755

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
680e651
add walk method
tlogan2000 Jan 8, 2024
b2587a5
catalog walk tests
tlogan2000 Jan 8, 2024
33cf829
Update src/siphon/catalog.py
tlogan2000 Jan 8, 2024
cabb3a2
flake8 fixes
tlogan2000 Jan 9, 2024
9fb1ad8
more flake8 list comprehension
tlogan2000 Jan 9, 2024
eb0730d
Merge branch 'catalogue_walk' of https://github.com/tlogan2000/siphon…
tlogan2000 Jan 11, 2024
a7cf22a
more flake8
tlogan2000 Jan 11, 2024
6e2374d
more flake8
tlogan2000 Jan 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions src/siphon/catalog.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@
from datetime import datetime
import logging
import re
import requests

Check failure on line 14 in src/siphon/catalog.py

View workflow job for this annotation

GitHub Actions / Flake8 

[flake8] reported by reviewdog 🐶
I201 Missing newline between import groups. 'import requests' is identified as Third Party and 'import re' is identified as Stdlib.

Raw Output:
./src/siphon/catalog.py:14:1: I201 Missing newline between import groups. 'import requests' is identified as Third Party and 'import re' is identified as Stdlib.
import warnings

Check failure on line 15 in src/siphon/catalog.py

View workflow job for this annotation

GitHub Actions / Flake8 

[flake8] reported by reviewdog 🐶
I100 Import statements are in the wrong order. 'import warnings' should be before 'import requests' and in a different group.

Raw Output:
./src/siphon/catalog.py:15:1: I100 Import statements are in the wrong order. 'import warnings' should be before 'import requests' and in a different group.

Check failure on line 15 in src/siphon/catalog.py

View workflow job for this annotation

GitHub Actions / Flake8 

[flake8] reported by reviewdog 🐶
I201 Missing newline between import groups. 'import warnings' is identified as Stdlib and 'import requests' is identified as Third Party.

Raw Output:
./src/siphon/catalog.py:15:1: I201 Missing newline between import groups. 'import warnings' is identified as Stdlib and 'import requests' is identified as Third Party.
import xml.etree.ElementTree as ET # noqa:N814
try:
from urlparse import urljoin, urlparse
Expand Down Expand Up @@ -389,6 +390,37 @@
else:
self.datasets.pop(ds_name)

def walk(self, depth=1):

"""Return a generator walking a THREDDS data catalog for datasets.

Check failure on line 395 in src/siphon/catalog.py

View workflow job for this annotation

GitHub Actions / Flake8 

[flake8] reported by reviewdog 🐶
D201 No blank lines allowed before function docstring

Raw Output:
./src/siphon/catalog.py:395:1: D201 No blank lines allowed before function docstring

Parameters
----------
cat : TDSCatalog
THREDDS catalog.
depth : int
Maximum recursive depth.
Setting 0 will return only datasets within the top-level catalog.
If None, depth is set to 1000.

Yields
------
Dataset
Siphon catalog dataset.
"""
tlogan2000 marked this conversation as resolved.
Show resolved Hide resolved
yield from self.datasets.values()
if depth is None:
depth = 1000

if depth > 0:
for ref in self.catalog_refs.values():
try:
child = ref.follow()
yield from child.walk(depth=depth - 1)

except requests.HTTPError as exc:
log.exception(exc)

@property
def latest(self):
"""Get the latest dataset, if available."""
Expand Down
Loading
Loading