Add API functionality #176
Conversation
|
@JBorrow the time allocated to the team at UCL to work on this has now ended so they're handing what they have (which is now fairly mature) over to us. I think that what we should do from here is:
If you agree with step 1, at least, then please go ahead and approve the merge to a branch and we can work from there :) |
|
@MatthieuSchaller you likely want to be aware of this PR, maybe you have input on the proposed workflow from here. |
|
Actually there's no reason not to merge this into a development branch, and I need to make a few commits to fix bugs due to the new |
| "cloudpickle>=2.2.1", | ||
| "numpy", | ||
| "h5py", | ||
| "unyt>=2.9.0", | ||
| "numba>=0.50.0", | ||
| "requests>=2.31.0", |
| class NumpyEncoder(json.JSONEncoder): | ||
| def default(self, obj): | ||
| if isinstance(obj, np.ndarray): | ||
| return obj.tolist() | ||
| return json.JSONEncoder.default(self, obj) |
There was a problem hiding this comment.
In general it would be great to have all the remote stuff in a separate file (e.g. remote.py)
| class RemoteSWIFTUnits: | ||
| def __init__(self, unit_dict=None): | ||
| excluded_fields = ["filename"] | ||
| if unit_dict is not None: | ||
| for key, value in unit_dict.items(): | ||
| if key not in excluded_fields and not isinstance( | ||
| value, unyt.unyt_quantity | ||
| ): | ||
| if isinstance(unit_dict[key], dict): | ||
| for nested_key, nested_value in unit_dict[key].items(): | ||
| setattr(self, nested_key, nested_value) | ||
| else: | ||
| setattr(self, key, unyt.unyt_quantity.from_string(value)) | ||
|
|
||
|
|
There was a problem hiding this comment.
What on earth does this do? Why does it exist? Needs to be documented.
| Args: | ||
| json_array (str): Numpy array as JSON | ||
| data_type (str): Data type of elements in the original array |
There was a problem hiding this comment.
Documentation is inconsistent with the rest of the codebase.
|
|
||
| def generate_getter_remote( | ||
| server_address: str, | ||
| session: requests.Session, |
| + SWIFTUnits, | ||
| + SWIFTMetadata, | ||
| + SWIFTParticleDataset |
There was a problem hiding this comment.
Seems to be for the remote versions of these.
|
Thanks @JBorrow. It's on my list to make a draft PR for this branch with a to-do list before it can be merged. Getting testing set up alone is going to be a big job. Ideally a lot of existing tests should be refactored to use fixtures so that we can test both the stand-alone and "client" SWIFTDatasets. Always hesitant to start making significant changes to tests, though, so will want to brainstorm that a bit before investing any effort. Figuring out how to host a server for the tests is also a |
As part of work between UCL and Durham, we have produced a prototype FastAPI app for sending numpy arrays derived from HDF5 files, as well as other SWIFT objects.
Here is the API