Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate a part of the jobs API to Fast API #16778

Merged
merged 110 commits into from
Nov 22, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
110 commits
Select commit Hold shift + click to select a range
16febbd
Remove redundant endpoints that have already been migrated
heisner-tillman Oct 2, 2023
4b2293d
Move method get_job to JobsService
heisner-tillman Oct 3, 2023
94fa1e0
Refactor common_problems operation
heisner-tillman Oct 3, 2023
a039842
Remove mapping to the legacy route of common_problems operation
heisner-tillman Oct 3, 2023
e7e4fbe
Add pydantic model for common_problems operation
heisner-tillman Oct 3, 2023
defec91
Update client schema
heisner-tillman Oct 3, 2023
5555d1f
Create schema file for jobs API and move corresponding model to it
heisner-tillman Oct 5, 2023
c6dab1c
Move dictify methods to JobsService
heisner-tillman Oct 5, 2023
2975601
Refactor resume operation
heisner-tillman Oct 5, 2023
ea520df
Add pydantic model for dictify methods
heisner-tillman Oct 5, 2023
ebdd5c9
Remove mapping to the legacy route of resume operation
heisner-tillman Oct 5, 2023
347fdb5
Fix typing mistake
heisner-tillman Oct 5, 2023
ab971e2
Fix codestyle mistakes
heisner-tillman Oct 5, 2023
141b774
Update client schema
heisner-tillman Oct 5, 2023
7f48d75
Reorder endpoint operations
heisner-tillman Oct 9, 2023
c9cf40b
Refactor error operation
heisner-tillman Oct 9, 2023
5096cbe
Add pydantic models for error operation
heisner-tillman Oct 9, 2023
e964c9e
Remove mapping to the legacy route of the error operation
heisner-tillman Oct 9, 2023
14ba8b0
Update client schema
heisner-tillman Oct 9, 2023
7d78bf5
Refactor error test
heisner-tillman Oct 9, 2023
ee47316
Refactor search operation
heisner-tillman Oct 16, 2023
a7d755c
Add ServiceBase as base class to JobsService
heisner-tillman Oct 16, 2023
220bd84
Remove mapping to legacy route
heisner-tillman Oct 16, 2023
2cf5dbf
Add part of the pydantic models for search operation
heisner-tillman Oct 16, 2023
52cc418
Refactor search tests to specify payload as json
heisner-tillman Oct 16, 2023
9ec6b8c
Regenerate client schema
heisner-tillman Oct 16, 2023
876a2f0
Add descriptions to pydantic model
heisner-tillman Oct 18, 2023
ca28384
Add TODO/suggestion to the search operation
heisner-tillman Oct 18, 2023
d70865f
Refactor some JobModels to enable reuse
heisner-tillman Oct 20, 2023
dd63f5a
Add pydantic models for search operation
heisner-tillman Oct 20, 2023
58a59ef
Reformulate TODO
heisner-tillman Oct 20, 2023
4b37dd2
Regenerate client schema
heisner-tillman Oct 20, 2023
2e10db9
Refactor inputs operation
heisner-tillman Oct 23, 2023
e3d3e44
Remove legacy route
heisner-tillman Oct 23, 2023
0745696
Add test for inputs operation
heisner-tillman Oct 23, 2023
e786bd1
Regenerate the client schema
heisner-tillman Oct 23, 2023
3b34f9b
Refactor outputs operation
heisner-tillman Oct 23, 2023
e4edab8
Remove mapping to legacy route
heisner-tillman Oct 23, 2023
4df0aef
Refactor inputs test to include testing for outputs
heisner-tillman Oct 23, 2023
fb876e6
Regenrate client schema
heisner-tillman Oct 23, 2023
0fa51e7
Refactor pydantic model for search operation
heisner-tillman Oct 24, 2023
7a43500
Refactor search operation to return pydantic model
heisner-tillman Oct 24, 2023
d96673f
Regenerate client schema
heisner-tillman Oct 24, 2023
a7edbc9
Refactor delete operation
heisner-tillman Oct 24, 2023
eec9430
Add pydantic model for delete operation
heisner-tillman Oct 24, 2023
981489f
Create internal method for Job testing and add test for delete operation
heisner-tillman Oct 24, 2023
2dc4cb2
Regenerate client schema
heisner-tillman Oct 24, 2023
5387bc3
Add proper documentation to show operation and use Annotated on delet…
heisner-tillman Oct 24, 2023
e842c82
Refactor destination_params operation
heisner-tillman Oct 24, 2023
9e48fa0
Add pydantic model for destination_params operation
heisner-tillman Oct 24, 2023
479e673
Add test for destination_params operation
heisner-tillman Oct 24, 2023
9806bd0
Remove mapping to the legacy route
heisner-tillman Oct 24, 2023
a7c3dce
Regenerate the client schema
heisner-tillman Oct 24, 2023
f5690e9
Refactor part of the metrics operation
heisner-tillman Oct 30, 2023
813d846
Remove mapping to the legacy route
heisner-tillman Oct 30, 2023
2fb6875
Add test for part of the metrics operation
heisner-tillman Oct 30, 2023
14154a6
Regenerate the client schema
heisner-tillman Oct 30, 2023
24b7a68
Refactor part of the parameters_display operation
heisner-tillman Oct 30, 2023
c4d9cb7
Remove mapping to legacy route
heisner-tillman Oct 30, 2023
dd7aa02
Add tests for part of parameters_display operation
heisner-tillman Oct 30, 2023
e14f66c
Regenerate the client schema
heisner-tillman Oct 30, 2023
324db18
Refactor second part of the metrics operation
heisner-tillman Oct 30, 2023
063e221
Remove mapping to the legacy route
heisner-tillman Oct 30, 2023
4688dc5
Regenerate client schema
heisner-tillman Oct 30, 2023
efa7473
Refactor get_job method
heisner-tillman Oct 30, 2023
3077a06
Remove TODO from tests
heisner-tillman Oct 30, 2023
5b8a031
Refactor second part of parameters_display operation
heisner-tillman Oct 30, 2023
5ad4141
Remove mapping to legacy route
heisner-tillman Oct 30, 2023
f447988
Regenerate client schema
heisner-tillman Oct 30, 2023
a4ba468
Remove unused methods
heisner-tillman Oct 30, 2023
08010df
Remove unused dependencies of JobController
heisner-tillman Oct 30, 2023
0258a74
Add TODO
heisner-tillman Oct 30, 2023
053ef2b
Add pydantic model in metrics operations return
heisner-tillman Oct 30, 2023
53055d6
Remove outdated TODOs
heisner-tillman Oct 30, 2023
1d38b37
Regenerate the client schema
heisner-tillman Oct 30, 2023
baf5cdf
Access managers via service, remove uneccessary comments/TODOs and ad…
heisner-tillman Nov 4, 2023
47aa990
Add pydantic model for parameters_display_by_job_id operation
heisner-tillman Nov 4, 2023
b85994a
Regenerate client schema
heisner-tillman Nov 4, 2023
23c8f17
Remove uneccessary comments and add pydantic model to prarameters_dis…
heisner-tillman Nov 4, 2023
0ddc883
Extend JobDisplayParametersSummary to use it in parameters_display_by…
heisner-tillman Nov 4, 2023
604d360
Regenerate the client schema
heisner-tillman Nov 4, 2023
e167581
Regenerate the client schema
heisner-tillman Nov 4, 2023
fb0526d
Apply some suggestions
heisner-tillman Nov 10, 2023
dd410f4
Apply some suggestions on pydantic models
heisner-tillman Nov 10, 2023
b0852ae
Refined EncodedJobDetails model
heisner-tillman Nov 10, 2023
8c6a64e
Add description to JobDestinationParams model and remove TODOs
heisner-tillman Nov 10, 2023
3dd5cb7
Regenerate client schema
heisner-tillman Nov 10, 2023
9b107d7
Mark *_by_dataset operations as deprecated
heisner-tillman Nov 10, 2023
ecddea3
Mark some HdaLddaQueryParam's as deprecated
heisner-tillman Nov 11, 2023
a48ebf1
Follow up on previous deprecation and refactor create operation
heisner-tillman Nov 11, 2023
75def35
Remove mapping to some legacy routes
heisner-tillman Nov 11, 2023
92559ef
Regenerate client schema
heisner-tillman Nov 11, 2023
6571f08
Divide JobAssociation into Input and Output model
heisner-tillman Nov 12, 2023
543a7a4
Regenerate the client schema
heisner-tillman Nov 12, 2023
899608c
Use validator to load job search inputs
mvdbeek Nov 13, 2023
d38b8db
Remove accidental code insertion in test_jobs.py
heisner-tillman Nov 14, 2023
19c06a9
Set Ids of JobBaseModel to encoded and undo previous model split
heisner-tillman Nov 14, 2023
ec606f8
Refine EncodedHdacaSourceId model
heisner-tillman Nov 14, 2023
1405391
Regenerate client schema
heisner-tillman Nov 14, 2023
e4824b2
Fix codestyle issue
heisner-tillman Nov 14, 2023
66001b7
Remove completed TODOs and clarify some path parameters
heisner-tillman Nov 14, 2023
a8a4b91
Remove outdated TODO
heisner-tillman Nov 18, 2023
832ecd8
Add test for delete operation
heisner-tillman Nov 18, 2023
c2ca0a6
Regenerate the client schema
heisner-tillman Nov 18, 2023
d925527
Add expetion in case of missing parameter
heisner-tillman Nov 20, 2023
78fc912
Refine the JobDestinationParams model
heisner-tillman Nov 20, 2023
05edd51
Regenerate the client schema
heisner-tillman Nov 20, 2023
c0d43cf
Drop POST /api/jobs
mvdbeek Nov 21, 2023
f18a33b
Update description of '/api/jobs/{job_id}/parameters_display'
mvdbeek Nov 21, 2023
4c4f949
Drop BaseModel docstring
mvdbeek Nov 21, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,587 changes: 964 additions & 623 deletions client/src/api/schema/schema.ts

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions lib/galaxy/managers/jobs.py
Original file line number Diff line number Diff line change
Expand Up @@ -880,6 +880,8 @@ def summarize_job_parameters(trans, job):
Precondition: the caller has verified the job is accessible to the user
represented by the trans parameter.
"""
# More client logic here than is ideal but it is hard to reason about
# tool parameter types on the client relative to the server.

def inputs_recursive(input_params, param_values, depth=1, upgrade_messages=None):
if upgrade_messages is None:
Expand Down
233 changes: 233 additions & 0 deletions lib/galaxy/schema/jobs.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,233 @@
import json
from typing import (
Any,
Dict,
List,
Optional,
)

from pydantic import (
Extra,
Field,
Required,
UUID4,
validator,
)
from typing_extensions import Literal

from galaxy.schema.fields import (
DecodedDatabaseIdField,
EncodedDatabaseIdField,
)
from galaxy.schema.schema import (
DataItemSourceType,
EncodedDataItemSourceId,
EntityIdField,
JobState,
JobSummary,
Model,
)


class JobInputSummary(Model):
has_empty_inputs: bool = Field(
default=Required,
title="Empty inputs",
description="Job has empty inputs.",
)
has_duplicate_inputs: bool = Field(
default=Required,
title="Duplicate inputs",
description="Job has duplicate inputs.",
)


# TODO: Use Tuple again when `make update-client-api-schema` supports them
class JobErrorSummary(Model):
# messages: List[Union[Tuple[str, str], List[str]]]
messages: List[List[str]] = Field(
default=Required,
title="Error messages",
description="The error messages for the specified job.",
)


class JobAssociation(Model):
heisner-tillman marked this conversation as resolved.
Show resolved Hide resolved
name: str = Field(
default=Required,
title="name",
description="Name of the job parameter.",
)
dataset: EncodedDataItemSourceId = Field(
default=Required,
title="dataset",
description="Reference to the associated item.",
)


class JobInputAssociation(JobAssociation):
name: str = Field(
default=Required,
title="name",
description="Name of the job input parameter.",
)


class JobOutputAssociation(JobAssociation):
name: str = Field(
default=Required,
title="name",
description="Name of the job output parameter.",
)


class ReportJobErrorPayload(Model):
dataset_id: DecodedDatabaseIdField = Field(
default=Required,
title="History Dataset Association ID",
description="The History Dataset Association ID related to the error.",
)
email: Optional[str] = Field(
default=None,
title="Email",
description="Email address for communication with the user. Only required for anonymous users.",
)
message: Optional[str] = Field(
default=None,
title="Message",
description="The optional message sent with the error report.",
)


class SearchJobsPayload(Model):
tool_id: str = Field(
default=Required,
title="Tool ID",
description="The tool ID related to the job.",
)
inputs: Dict[str, Any] = Field(
default=Required,
title="Inputs",
description="The inputs of the job.",
)
state: JobState = Field(
default=Required,
nsoranzo marked this conversation as resolved.
Show resolved Hide resolved
title="State",
description="Current state of the job.",
)

@validator("inputs", pre=True)
def decode_json(cls, v):
if isinstance(v, str):
return json.loads(v)
return v

class Config:
extra = Extra.allow # This is used for items named file_ and __file_
mvdbeek marked this conversation as resolved.
Show resolved Hide resolved


class DeleteJobPayload(Model):
message: Optional[str] = Field(
default=None,
title="Job message",
description="Stop message",
)


class SrcItem(Model):
src: DataItemSourceType


class EncodedHdcaSourceId(SrcItem):
id: EncodedDatabaseIdField = EntityIdField
src: Literal[DataItemSourceType.hdca] = Field(
default=Required,
title="Source",
description="The source of this dataset, which in the case of the model can only be `hdca`.",
)


class EncodedDatasetJobInfo(EncodedDataItemSourceId):
uuid: Optional[UUID4] = Field(
default=None,
deprecated=True,
title="UUID",
description="Universal unique identifier for this dataset.",
)


class EncodedJobDetails(JobSummary):
command_version: str = Field(
...,
title="Command Version",
description="Tool version indicated during job execution.",
)
params: Any = Field(
...,
title="Parameters",
description=(
"Object containing all the parameters of the tool associated with this job. "
"The specific parameters depend on the tool itself."
),
)
inputs: Dict[str, EncodedDatasetJobInfo] = Field(
{},
title="Inputs",
description="Dictionary mapping all the tool inputs (by name) to the corresponding data references.",
)
outputs: Dict[str, EncodedDatasetJobInfo] = Field(
{},
title="Outputs",
description="Dictionary mapping all the tool outputs (by name) to the corresponding data references.",
)
copied_from_job_id: Optional[EncodedDatabaseIdField] = Field(
default=None, title="Copied from Job-ID", description="Reference to cached job if job execution was cached."
)
output_collections: Dict[str, EncodedHdcaSourceId] = Field(default={}, title="Output collections", description="")


class JobDestinationParams(Model):
mvdbeek marked this conversation as resolved.
Show resolved Hide resolved
runner: Optional[str] = Field(None, title="Runner", description="Job runner class", alias="Runner")
runner_job_id: Optional[str] = Field(
None,
title="Runner Job ID",
description="ID assigned to submitted job by external job running system",
alias="Runner Job ID",
)
handler: Optional[str] = Field(
None, title="Handler", description="Name of the process that handled the job.", alias="Handler"
)


class JobOutput(Model):
label: Any = Field(default=Required, title="Output label", description="The output label") # check if this is true
value: EncodedDataItemSourceId = Field(default=Required, title="Dataset", description="The associated dataset.")


class JobParameter(Model):
text: str = Field(
default=Required,
title="Text",
description="Text associated with the job parameter.",
)
depth: int = Field(
default=Required,
title="Depth",
description="The depth of the job parameter.",
)
value: Any = Field(default=Required, title="Value", description="The values of the job parameter")
notes: Optional[str] = Field(default=None, title="Notes", description="Notes associated with the job parameter.")


class JobDisplayParametersSummary(Model):
parameters: List[JobParameter] = Field(
default=Required, title="Parameters", description="The parameters of the job in a nested format."
)
has_parameter_errors: bool = Field(
default=Required, title="Has parameter errors", description="The job has parameter errors"
)
outputs: Dict[str, List[JobOutput]] = Field(
default=Required,
title="Outputs",
description="Dictionary mapping all the tool outputs (by name) with the corresponding dataset information in a nested format.",
)
31 changes: 23 additions & 8 deletions lib/galaxy/schema/schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -257,8 +257,6 @@ class DatasetCollectionPopulatedState(str, Enum):


class Model(BaseModel):
"""Base model definition with common configuration used by all derived models."""

class Config:
use_enum_values = True # when using .dict()
allow_population_by_field_name = True
Expand Down Expand Up @@ -475,6 +473,14 @@ class DatasetSourceType(str, Enum):
ldda = "ldda"


class DataItemSourceType(str, Enum):
hda = "hda"
ldda = "ldda"
hdca = "hdca"
dce = "dce"
dc = "dc"


class ColletionSourceType(str, Enum):
hda = "hda"
ldda = "ldda"
Expand Down Expand Up @@ -1821,18 +1827,18 @@ class JobIdResponse(Model):


class JobBaseModel(Model):
id: DecodedDatabaseIdField = EntityIdField
id: EncodedDatabaseIdField = EntityIdField
history_id: Optional[EncodedDatabaseIdField] = Field(
None,
title="History ID",
description="The encoded ID of the history associated with this item.",
)
model_class: JOB_MODEL_CLASS = ModelClassField(JOB_MODEL_CLASS)
tool_id: str = Field(
...,
title="Tool ID",
description="Identifier of the tool that generated this job.",
)
history_id: Optional[DecodedDatabaseIdField] = Field(
None,
title="History ID",
description="The encoded ID of the history associated with this item.",
)
state: JobState = Field(
...,
title="State",
Expand Down Expand Up @@ -1925,6 +1931,15 @@ class EncodedDatasetSourceId(DatasetSourceIdBase):
id: EncodedDatabaseIdField = EntityIdField


class EncodedDataItemSourceId(Model):
id: EncodedDatabaseIdField = EntityIdField
src: DataItemSourceType = Field(
...,
title="Source",
description="The source of this dataset, either `hda`, `ldda`, `hdca`, `dce` or `dc` depending of its origin.",
)


class DatasetJobInfo(DatasetSourceId):
uuid: UUID4 = UuidField

Expand Down
4 changes: 3 additions & 1 deletion lib/galaxy/security/idencoding.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,11 @@ def __init__(self, **config):
per_kind_id_secret_base = config.get("per_kind_id_secret_base", self.id_secret)
self.id_ciphers_for_kind = _cipher_cache(per_kind_id_secret_base)

def encode_id(self, obj_id, kind=None):
def encode_id(self, obj_id, kind=None, strict_integer=False):
if obj_id is None:
raise galaxy.exceptions.MalformedId("Attempted to encode None id")
if strict_integer and not isinstance(obj_id, int):
raise galaxy.exceptions.MalformedId("Attempted to encode id that is not an integer")
id_cipher = self.__id_cipher(kind)
# Convert to bytes
s = smart_str(obj_id)
Expand Down
Loading
Loading