Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create pipeline for inference with gridded data to create map plots of ML prediction #91

Open
stevehadd opened this issue Aug 25, 2022 · 1 comment

Comments

@stevehadd
Copy link
Member

Currently we're focused on training and inference with the tabular dataset. Ultimately we want to display predictions as maps. We thus want to be able to load some gridded data (as would come directly out of the model and stage pipeline), and be able to easily do inference to create a ML prediction. This pipeline doesn't exiust yet so should be created in anticipation of further sharing of our results.

@stevehadd
Copy link
Member Author

it would be good to create a dataset class for gridded data, where like with tabular data you can get a memory object (possibly lazily loaded) back into memory. This could be backed by zarr and return an xarray object like one gets a pandas dataframe.

Would also be good to include more of lazy loading paradigm into the datasets concept (like maybe create a dask or ray or DVC cluster through the AzML compute GUI) to do you processing and then it returns your reduced in memory object.

Would be good to have different specialised datasets e.g. geospatial data, satellite images, 3D point cloud etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant