AzureML upload and prepare scripts/notebooks #86

stevehadd · 2022-08-18T09:20:48Z

This pull request includes the following changes:

A shell script to upload from the output directory of the scripts to a container on Azure Blob storage
A notebook to automatically create AzureML datasets from uploaded data
A revised version of the data verification to notebook that runs on AzureML using AzureML datasets
A version of the conda env requirements file for data prep for running on AzureML.
Some fixes to event configs

Closes #14

…datasets from files once they are uploaded. We may want to create some sort of catalog file to upload with the others to help with this process.

…ml script.

…he tabular datasets are currently messing up our data.

… (and other data prep stuff) on azure ML.

review-notebook-app · 2022-08-18T09:20:53Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

…le datasets for us to work with while the tabular datasets are being debugged.

#14 I'm adding some version pins to all requirements files, so updating the branch/PR to update the AzML specific data prep requirements file.

DPeterK

@stevehadd looks good 👍

data_prep/azureml/copy_to_azure_ml.sh

data_prep/azureml/prd_create_azml_datasets.ipynb

data_prep/azureml/prd_data_verification_azml_dataset.ipynb

stevehadd and others added 8 commits August 5, 2022 15:33

#14 Created the start of a notebook to create azureml datastores and …

15b1d7f

…datasets from files once they are uploaded. We may want to create some sort of catalog file to upload with the others to help with this process.

#11 notebook to cresate datasets from uploaded data.

97524f0

#14 changing location of stuff relating to azure ml data prep.

6697b34

#14: Adding ability to create the specified log directory if required.

ea38b14

#14 removed key from notebook.

d55a09c

#14 Fixed up event configs, and also did a test of te upload to azure…

197a42b

…ml script.

#14: Updating to use file datasets rather than tabular datasets, as t…

cfae41e

…he tabular datasets are currently messing up our data.

#14 Added a conda requirements file for running the data verification…

e5eee71

… (and other data prep stuff) on azure ML.

stevehadd added data_prep azure_ml labels Aug 18, 2022

stevehadd requested review from DPeterK, arh89 and hannahbrown7 August 18, 2022 09:20

stevehadd self-assigned this Aug 18, 2022

#14 Updated the automatic dataset creation notebook to also create fi…

1e33eed

…le datasets for us to work with while the tabular datasets are being debugged.

stevehadd marked this pull request as ready for review August 19, 2022 09:48

Adding some version pins

eec3d0b

#14 I'm adding some version pins to all requirements files, so updating the branch/PR to update the AzML specific data prep requirements file.

DPeterK reviewed Aug 19, 2022

View reviewed changes

stevehadd mentioned this pull request Aug 25, 2022

create setup.py for better imports #92

Open

#14 made changes suggested in code review.

5f68816

stevehadd merged commit c4d3f1c into main Aug 31, 2022

stevehadd deleted the prd11_azml_upload_prepare branch August 31, 2022 08:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AzureML upload and prepare scripts/notebooks #86

AzureML upload and prepare scripts/notebooks #86

stevehadd commented Aug 18, 2022

review-notebook-app bot commented Aug 18, 2022

DPeterK left a comment

AzureML upload and prepare scripts/notebooks #86

AzureML upload and prepare scripts/notebooks #86

Conversation

stevehadd commented Aug 18, 2022

review-notebook-app bot commented Aug 18, 2022

DPeterK left a comment

Choose a reason for hiding this comment