Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature - stdout live reporting #16975

Merged
merged 36 commits into from
Nov 18, 2024
Merged
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
58d24dc
Add option to load stdout and stderr when requesting job status
gecage952 Oct 17, 2023
89cc78a
Update api endpoint for jobs
gecage952 Nov 3, 2023
870917b
Add missing variables and import
gecage952 Nov 3, 2023
6dfbc7a
Add capability for UI to load live job stdout and stderr
gecage952 Nov 3, 2023
fba01ad
Merge pull request #1 from gecage952/dev
gecage952 Nov 3, 2023
de7cadf
Linting / format.
dannon Nov 6, 2023
31065a6
Fix jest test
dannon Nov 6, 2023
24e7d15
Update API schema
dannon Nov 6, 2023
d336cad
Configure pulsar runner to read stdout from file if finish message st…
gecage952 Nov 17, 2023
afa0b25
XMerge branch 'feature_stdout_live_reporting' of github.com:gecage952…
gecage952 Nov 17, 2023
d2be71f
Merge branch 'dev' into feature_stdout_live_reporting
gecage952 Nov 17, 2023
fd0237f
Add new api endpoint to fetch job stdout and stderr
gecage952 Dec 15, 2023
70bfcc4
Use stdout api endpoint in job info ui while job is running
gecage952 Dec 15, 2023
7516d6a
Merge upstream into branch
gecage952 Dec 15, 2023
7052e90
Merge dev into branch and fix conflicts
gecage952 Mar 21, 2024
89113d0
Update stdout reporting code to vue3
gecage952 Mar 22, 2024
791348a
Fix linting errors
gecage952 Mar 27, 2024
87f5126
Fix more formatting and schema issues
gecage952 Mar 27, 2024
1b2ba86
Merge branch 'dev' into feature_stdout_live_reporting
gecage952 Apr 19, 2024
f1f8eec
format imports
martenson May 3, 2024
6faf683
Fix issue where only one of stdout or stderr is present
gecage952 May 9, 2024
beb5d5b
Merge branch 'feature_stdout_live_reporting' of github.com:gecage952/…
gecage952 May 13, 2024
63ede70
Fix docstrings that refer to tool stdout as job stdout
gecage952 May 29, 2024
735d5ef
Restrict tool stdout updating to properly configured job destinations
gecage952 Jun 14, 2024
deaba39
update client api schema
gecage952 Jun 14, 2024
907328e
Fix merge conflicts
gecage952 Jul 12, 2024
2e472a9
Update client api schema
gecage952 Jul 12, 2024
811a897
Update client api schema
gecage952 Sep 11, 2024
53051e7
Update client api schema
gecage952 Sep 23, 2024
858a245
Merge remote-tracking branch 'origin/dev' into feature_stdout_live_re…
jmchilton Nov 12, 2024
a2bffb8
Update JobInformation.test.js for live console output.
jmchilton Nov 12, 2024
8063632
Improved error handling for live console for tool execution.
jmchilton Nov 12, 2024
be11412
xfail the test for the freeze.
jmchilton Nov 18, 2024
a69031b
Merge remote-tracking branch 'origin/dev' into feature_stdout_live_re…
jmchilton Nov 18, 2024
f0aa89a
linting fix
jmchilton Nov 18, 2024
394fb16
Ahhh... the set metadata file is JSON and cannot be appended.
jmchilton Nov 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Add option to load stdout and stderr when requesting job status
gecage952 committed Oct 17, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit 58d24dc6e438bc6d9383a5c0c0a0478a951eca3e
26 changes: 25 additions & 1 deletion lib/galaxy/managers/jobs.py
Original file line number Diff line number Diff line change
@@ -231,7 +231,7 @@ def update_job_lock(self, job_lock: JobLock):
)
return self.job_lock()

def get_accessible_job(self, trans, decoded_job_id):
def get_accessible_job(self, trans, decoded_job_id, stdout_position=-1, stdout_length=0, stderr_position=-1, stderr_length=0):
job = trans.sa_session.query(trans.app.model.Job).filter(trans.app.model.Job.id == decoded_job_id).first()
if job is None:
raise ObjectNotFound()
@@ -248,6 +248,30 @@ def get_accessible_job(self, trans, decoded_job_id):
if not self.dataset_manager.is_accessible(data_assoc.dataset.dataset, trans.user):
raise ItemAccessibilityException("You are not allowed to rerun this job.")
trans.sa_session.refresh(job)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a bunch of reasons this might not be possible I think. Older Pulsar, remote execution outside of Pulsar, a Pulsar modality that doesn't allow this, runners that respect the location of working directory stuff in this fashion. It feels like the right thing to do here is to throw a galaxy.exceptions.ConfigDoesNotAllowException and then catch this on the client and not poll. Does that make sense to you?

Ideally I think we would make all this opt-in based on the job destination until we've hardened it and seen it work well in production without performance issues or confusing user experience. The job object has a job.job_destination YAML thing we could use to configure things per environment/destination.

Copy link
Contributor Author

@gecage952 gecage952 May 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that exception definitely makes sense. The compatibility just isn't going to be there for some stuff, I guess. Opt-in is a good idea as well.

# If stdout_length and stdout_position are legitimate values, then return stdout with status.
if job.state == job.states.RUNNING:
working_directory = trans.app.object_store.get_filename(
job, base_dir="job_work", dir_only=True, obj_dir=True
)
if stdout_length > 0 and stdout_position > -1:
try:
stdout_path = Path(working_directory) / STDOUT_LOCATION
stdout_file = open(stdout_path, "r")
stdout_file.seek(stdout_position)
job.job_stdout = stdout_file.read(stdout_length)
job.tool_stdout = job.job_stdout
except Exception as e:
log.error("Could not read STDOUT: %s", e)
if stderr_length > 0 and stderr_position > -1:
try:
stderr_path = Path(working_directory) / STDERR_LOCATION
stderr_file = open(stderr_path, "r")
stderr_file.seek(stderr_position)
job.job_stderr = stderr_file.read(stderr_length)
job.tool_stderr = job.job_stderr
except Exception as e:
log.error("Could not read STDERR: %s", e)
return job

def stop(self, job, message=None):
6 changes: 5 additions & 1 deletion lib/galaxy/webapps/galaxy/api/job_files.py
Original file line number Diff line number Diff line change
@@ -104,7 +104,11 @@ def create(self, trans, job_id, payload, **kwargs):
target_dir = os.path.dirname(path)
util.safe_makedirs(target_dir)
try:
shutil.move(input_file.name, path)
if os.path.exists(path):
with open(path, "ab") as destination:
shutil.copyfileobj(open(input_file.name, "rb"), destination)
else:
shutil.move(input_file.name, path)
finally:
try:
input_file.close()
14 changes: 13 additions & 1 deletion lib/galaxy/webapps/galaxy/api/jobs.py
Original file line number Diff line number Diff line change
@@ -178,6 +178,10 @@ def show(
id: DecodedDatabaseIdField,
trans: ProvidesUserContext = DependsOnTrans,
full: Optional[bool] = False,
stdout_position: Optional[int] = None,
stdout_length: Optional[int] = None,
stderr_position: Optional[int] = None,
stderr_length: Optional[int] = None,
) -> Dict[str, Any]:
"""
Return dictionary containing description of job data
@@ -186,7 +190,15 @@ def show(
- id: ID of job to return
- full: Return extra information ?
"""
return self.service.show(trans, id, bool(full))
return self.service.show(
trans,
id,
bool(full),
int(stdout_position) if stdout_position else 0,
int(stdout_length) if stdout_length else 0,
int(stderr_position) if stderr_position else 0,
int(stderr_length) if stderr_length else 0,
)

@router.get("/api/jobs")
def index(
13 changes: 12 additions & 1 deletion lib/galaxy/webapps/galaxy/services/jobs.py
Original file line number Diff line number Diff line change
@@ -48,8 +48,19 @@ def show(
trans: ProvidesUserContext,
id: DecodedDatabaseIdField,
full: bool = False,
stdout_position: int = 0,
stdout_length: int = 0,
stderr_position: int = 0,
stderr_length: int = 0,
) -> Dict[str, Any]:
job = self.job_manager.get_accessible_job(trans, id)
job = self.job_manager.get_accessible_job(
trans,
id,
stdout_position,
stdout_length,
stderr_position,
stderr_length
)
return view_show_job(trans, job, bool(full))

def index(