Skip to content

Latest commit

 

History

History
89 lines (54 loc) · 5.03 KB

commands.md

File metadata and controls

89 lines (54 loc) · 5.03 KB

set subscription

az account set --name balaraje az account set --subscription 4956d76e-866c-44cb-ae27-659157d2b925

check subscription

az account show

create job and run

az ml job create --file job.yml --resource-group scenera-demo --workspace-name scenera-demo-ml

add dataset to the azure ml workspace from your local file system

az ml data create --file prod-dataset.yml --resource-group scenera-demo --workspace-name scenera-demo-ml

Create service principle

az ad sp create-for-rbac --name "SCENERA-SP" --role contributor \
                              --scopes /subscriptions/4956d76e-866c-44cb-ae27-659157d2b925/resourceGroups/scenera-demo \
                              --sdk-auth

Register a model (https://learn.microsoft.com/en-us/azure/machine-learning/how-to-manage-models?view=azureml-api-2&tabs=cli%2Cuse-job-output#register-your-model-as-an-asset-in-machine-learning-by-using-the-cli)

az ml model create --workspace-name scenera-demo-ml --name scenera-demo-model --version 1 --path azureml://jobs/jolly_turnip_zv1k0xm78f/outputs/artifacts/paths/model/

Create an endpoint

az ml online-endpoint create --name scenera-mlflow-endpoint --file create-endpoint.yml --resource-group scenera-demo --workspace-name scenera-demo-ml

update an endpoint

az ml online-endpoint update --file create-endpoint.yml --resource-group scenera-demo --workspace-name scenera-demo-ml

Deploy the model

az ml online-deployment create --name scenera-mlflow-deployment --endpoint scenera-mlflow-endpoint --file deployment.yml --all-traffic --resource-group scenera-demo --workspace-name scenera-demo-ml

Delete deployment

az ml online-deployment delete --name scenera-mlflow-deployment --endpoint-name scenera-mlflow-endpoint --yes --resource-group scenera-demo --workspace-name scenera-demo-ml

Development Steps

  • Create Azure Machine Learning workspace and a compute instance/Cluster Show the compute cluster and instance created for this purpose

  • Create a registered data asset within Azure ML to be used in the training Jobs

  • Create Azure Machine Learning jobs to train the model with the registered dataset as input

  • Next step is to automate the process of running the training jobs using the Github Action so that it can be part of CICD Pipeline. In order to achieve this we needed to create Azure Service Principle to authenticate and run the job from the GitHub Action. Important learning is that the individual compute instances can only be assigned to a single human azure user. GitHub is authenticated to use your Azure Machine Learning workspace with a service principal. The service principal is only allowed to submit jobs that use a compute cluster, not a compute instance.

    Note: We added a linting and unit test job to validate the code quality before we can merge the pull request to the main branch.

    Note: We need to store the service principle credentials in Github secrets (Alternatively you could use azure key vault at this step) Explored both storing dev and prod secrets in Github Repo

  • We create

  • When you have separate environments for development, staging, and production, you can more easily control access to resources. We use environments to isolate workloads and control the training of the model. We set up a development and production environment. we will show you what this looks like later in the demo. And also took advantage of the distinct environment to added a required reviewer to the production jobs.

  • Register the model with GitHub Actions. The model's output was automatically generated by the MLflow auto log function in the training script. Register the model from the production job output in the Azure Machine Learning

az ml job download --name zen_leaf_jdgxnc94f7 --resource-group rg-scenera-001dev --workspace-name mlw-scenera-001dev

az ml job show --name train_model --resource-group rg-scenera-001dev --workspace-name mlw-scenera-001dev

  # chmod +x ./scripts/workflows/train_model.sh
      # ./scripts/workflows/train_model.sh

az ml job create --file ./mlops/azureml/train/pipeline.yml --resource-group rg-scenera-001dev --workspace-name mlw-scenera-001dev --query name -o tsv

az ml model register --name "scenera-model-v1" --run-id db5c7375-d984-427c-a3f9-5ce5b15da0f6 --resource-group rg-scenera-001dev --workspace-name mlw-scenera-001dev

az ml model create --name "scenera-model-v1" --version 1 --path runs:/db5c7375-d984-427c-a3f9-5ce5b15da0f6/yolov8-training-coco-dataset/ --type custom_model --resource-group rg-scenera-001dev --workspace-name mlw-scenera-001dev

az ml model create --name "scenera-model-v12" --version 1 --path azureml://jobs/7dee2a20-cc7a-40d2-9d8c-c4c65b7c5aeb/outputs/yolov8-training-coco-dataset --resource-group rg-scenera-001dev --workspace-name mlw-scenera-001dev

      az ml model create --name secenera-demo-2 --version 1 --path runs:/7dee2a20-cc7a-40d2-9d8c-c4c65b7c5aeb/yolov8-training-coco-dataset/ --type custom_model --resource-group rg-scenera-001dev --workspace-name mlw-scenera-001dev