Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: fixed missing file handling for remote polling #55

Merged
merged 3 commits into from
Sep 3, 2024
Merged

BUG: fixed missing file handling for remote polling #55

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1fbf240
BUG: fixed missing file polling
cpelley Aug 30, 2024
6bcb43d
fixed remote data polling
cpelley Sep 3, 2024
a329d3e
Automated reference documentation update for PR 55 [skip ci]
github-actions[bot] Sep 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 10 additions & 11 deletions dagrunner/plugin_framework.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,17 +203,16 @@ def __call__(

if host:
# bash equivalent to python glob (glob on remote host)
expanded_paths = (
subprocess.run(
f"ssh {host} \"printf '%s\n' {pattern} | grep -v '*'\" || true",
shell=True,
check=True,
text=True,
capture_output=True,
)
.stdout.strip()
.split("\n")
)
expanded_paths = subprocess.run(
f'ssh {host} \'for file in {pattern}; do if [ -e "$file" ]; then '
'echo "$file"; fi; done\'',
shell=True,
check=True,
text=True,
capture_output=True,
).stdout.strip()
if expanded_paths:
expanded_paths = expanded_paths.split("\n")
else:
expanded_paths = glob(pattern)
if expanded_paths:
Expand Down
14 changes: 14 additions & 0 deletions dagrunner/tests/plugin_framework/test_DataPolling.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,3 +56,17 @@ def test_specified_host(tmp_file, capsys):
call_dp(host_tmp_file)
captured = capsys.readouterr()
assert f"The following files were polled and found: ['{tmp_file}']" in captured.out


def test_specified_host_missing_file(capsys):
"""<host>:<filepath>"""
filepath = "/dummy/file/path.dat"
host_tmp_file = f"{socket.gethostname()}:{filepath}"
# Mocking gethostname() so that our host doesn't match against our local host check
# internally.
msg = f"Timeout waiting for: '{filepath}'"
with patch(
"dagrunner.utils.socket.gethostname", return_value="dummy_host.dummy_domain"
):
with pytest.raises(FileNotFoundError, match=msg):
call_dp(host_tmp_file, verbose=False)
20 changes: 10 additions & 10 deletions docs/dagrunner.plugin_framework.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Raises:

## class: `Input`

[Source](../dagrunner/plugin_framework.py#L241)
[Source](../dagrunner/plugin_framework.py#L240)

### Call Signature:

Expand All @@ -64,7 +64,7 @@ that are 'node aware'.

### function: `__call__`

[Source](../dagrunner/plugin_framework.py#L242)
[Source](../dagrunner/plugin_framework.py#L241)

#### Call Signature:

Expand Down Expand Up @@ -144,7 +144,7 @@ Raises:

## class: `LoadJson`

[Source](../dagrunner/plugin_framework.py#L275)
[Source](../dagrunner/plugin_framework.py#L274)

### Call Signature:

Expand Down Expand Up @@ -175,7 +175,7 @@ Args:

### function: `load`

[Source](../dagrunner/plugin_framework.py#L278)
[Source](../dagrunner/plugin_framework.py#L277)

#### Call Signature:

Expand All @@ -197,7 +197,7 @@ Raises:

## class: `LoadPickle`

[Source](../dagrunner/plugin_framework.py#L314)
[Source](../dagrunner/plugin_framework.py#L313)

### Call Signature:

Expand Down Expand Up @@ -228,7 +228,7 @@ Args:

### function: `load`

[Source](../dagrunner/plugin_framework.py#L317)
[Source](../dagrunner/plugin_framework.py#L316)

#### Call Signature:

Expand Down Expand Up @@ -320,7 +320,7 @@ Returns:

## class: `SaveJson`

[Source](../dagrunner/plugin_framework.py#L286)
[Source](../dagrunner/plugin_framework.py#L285)

### Call Signature:

Expand All @@ -334,7 +334,7 @@ that are 'node aware'.

### function: `__call__`

[Source](../dagrunner/plugin_framework.py#L287)
[Source](../dagrunner/plugin_framework.py#L286)

#### Call Signature:

Expand All @@ -359,7 +359,7 @@ Returns:

## class: `SavePickle`

[Source](../dagrunner/plugin_framework.py#L325)
[Source](../dagrunner/plugin_framework.py#L324)

### Call Signature:

Expand All @@ -373,7 +373,7 @@ that are 'node aware'.

### function: `__call__`

[Source](../dagrunner/plugin_framework.py#L326)
[Source](../dagrunner/plugin_framework.py#L325)

#### Call Signature:

Expand Down