diff --git a/AutoMLOps_User_Guide.pdf b/AutoMLOps_User_Guide.pdf index 2d1a6da..5f05b86 100644 Binary files a/AutoMLOps_User_Guide.pdf and b/AutoMLOps_User_Guide.pdf differ diff --git a/CHANGELOG.md b/CHANGELOG.md index 677dc6e..4186ddc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,28 @@ # Change Log All notable changes to this project will be documented in this file. +## [1.3.1] - 2024-10-30 + +### Added + +- Guide explaining how to set up Workload Identity Federation for AutoMLOps +- Created new docs/ directory +- Check that prohibits using Github and Cloud Build (this is currently not supported) + +### Changed + +- Made `github` and `github-actions` the default options for `source_repo_type` and `deployment_framework` respectively +- Updated example notebooks so that they are set to `use_ci=False` by default +- Provided samples in the example notebooks that demonstrate how to use the `github` and `github_actions` setup +- Added a guide for setting up Workload Identity Federation +- Updated github actions op versions (checkout and auth) + +### Fixed + +- Fixed broken default of now deprecated Cloud Source Repositories +- Updated sklearn version of training example00 to use v1.3 +- Removed `build_trigger_location` from resources manifest output + ## [1.3.0] - 2024-5-22 ### Added diff --git a/README.md b/README.md index 2180a15..eed2d19 100644 --- a/README.md +++ b/README.md @@ -64,8 +64,8 @@ Inferencing - Artifact Registry **Deployment Frameworks**: Builds component docker containers, compiles pipelines, and submits Pipeline Jobs -- Cloud Build - Github Actions +- Cloud Build - [coming soon] Gitlab CI - [coming soon] Bitbucket Pipelines - [coming soon] Jenkins @@ -87,10 +87,10 @@ Inferencing - [coming soon] pulumi **Source Code Repositories**: Repository for versioning generated MLOps code -- [deprecating soon] Cloud Source Repositories -- Bitbucket - Github +- Bitbucket - Gitlab +- [deprecated] Cloud Source Repositories # Prerequisites ### Generate @@ -122,11 +122,6 @@ In order to use `AutoMLOps.deploy(...)` with `use_ci=True`, the following are re git config --global user.name "Your Name" ``` - Registered and setup your SSH key if you are using Github, Gitlab, or Bitbucket -- [Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/provide-credentials-adc) are set up if you are using Cloud Source Repositories. This can be done through the following commands: -``` -gcloud auth application-default login -gcloud config set account -``` ### Monitor In order to use `AutoMLOps.monitor(...)`, the following are required: @@ -170,10 +165,7 @@ AutoMLOps will makes use of the following products based on user selected option 6. if `use_ci=True` and `schedule_pattern` is specified, AutoMLOps will use: - [Cloud Scheduler](https://cloud.google.com/scheduler/docs/overview) -7. if `use_ci=True` and `source_repo_type='cloud-source-repositories'`, AutoMLOps will use: -- [Cloud Source Repositories](https://cloud.google.com/source-repositories/docs) - -8. if `use_ci=True` and `setup_model_monitoring=True`, AutoMLOps will use: +7. if `use_ci=True` and `setup_model_monitoring=True`, AutoMLOps will use: - [Vertex AI Model Monitoring](https://cloud.google.com/vertex-ai/docs/model-monitoring/overview) - [Cloud Logging](https://cloud.google.com/logging/docs/overview) @@ -193,7 +185,6 @@ Based on the above user selection, AutoMLOps will enable up to the following API - [pubsub.googleapis.com](https://cloud.google.com/pubsub/docs/reference/rest) - [run.googleapis.com](https://cloud.google.com/run/docs/reference/rest) - [storage.googleapis.com](https://cloud.google.com/storage/docs/apis) -- [sourcerepo.googleapis.com](https://cloud.google.com/source-repositories/docs/reference/rest) AutoMLOps will create the following service account and update [IAM permissions](https://cloud.google.com/iam/docs/understanding-roles) during the provision step: @@ -232,7 +223,7 @@ Optional parameters (defaults shown): 5. `build_trigger_location: str = 'us-central1'` 6. `build_trigger_name: str = f'{naming_prefix}-build-trigger'` 7. `custom_training_job_specs: list[dict] = None` -8. `deployment_framework: str = 'cloud-build'` +8. `deployment_framework: str = 'github-actions'` 9. `naming_prefix: str = 'automlops-default-prefix'` 10. `orchestration_framework: str = 'kfp'` 11. `pipeline_job_runner_service_account: str = f'vertex-pipelines@{project_id}.iam.gserviceaccount.com'` @@ -249,7 +240,7 @@ Optional parameters (defaults shown): 22. `setup_model_monitoring: Optional[bool] = False` 23. `source_repo_branch: str = 'automlops'` 24. `source_repo_name: str = f'{naming_prefix}-repository'` -25. `source_repo_type: str = 'cloud-source-repositories'` +25. `source_repo_type: str = 'github'` 26. `storage_bucket_location: str = 'us-central1'` 27. `storage_bucket_name: str = f'{project_id}-{naming_prefix}-bucket'` 28. `use_ci: bool = False` @@ -262,8 +253,8 @@ Parameter Options: - `artifact_repo_type=`: - 'artifact-registry' (default) - `deployment_framework=`: - - 'cloud-build' (default) - - 'github-actions' + - 'github-actions' (default) + - 'cloud-build' - [coming soon] 'gitlab-ci' - [coming soon] 'bitbucket-pipelines' - [coming soon] 'jenkins' @@ -281,8 +272,7 @@ Parameter Options: - 'terraform' - [coming soon] 'pulumi' - `source_repo_type=`: - - 'cloud-source-repositories' (default) - - 'github' + - 'github' (default) - 'gitlab' - 'bitbucket' @@ -324,7 +314,7 @@ A description of the parameters is below: - `workload_identity_provider`: Provider for workload identity federation. - `workload_identity_service_account`: Service account for workload identity federation (specify the full string). -AutoMLOps will generate the resources specified by these parameters (e.g. Artifact Registry, Cloud Source Repo, etc.). If use_ci is set to True, AutoMLOps will turn the outputted AutoMLOps/ directory into a Git repo and use it for the source repo. If a cron formatted str is given as an arg for `schedule_pattern` then it will set up a Cloud Schedule to run accordingly. If `setup_model_monitoring` is set to true, a model_monitoring/ directory will be created and a monitoring section will be added to config/defaults.yaml with empty values. These values are then set by running `AutoMLOps.monitor()`. +AutoMLOps will generate the resources specified by these parameters (e.g. Artifact Registry, GCS bucket, etc.). If use_ci is set to True, AutoMLOps will turn the outputted AutoMLOps/ directory into a Git repo and use it for the source repo. If a cron formatted str is given as an arg for `schedule_pattern` then it will set up a Cloud Schedule to run accordingly. If `setup_model_monitoring` is set to true, a model_monitoring/ directory will be created and a monitoring section will be added to config/defaults.yaml with empty values. These values are then set by running `AutoMLOps.monitor()`. # Generating Code @@ -385,11 +375,6 @@ AutoMLOps currently provides 2 primary options for provisioning infrastructure: ### Cloud Continuous Integration and Continuous Deployment Workflow If `use_ci=True`, AutoMLOps will generate and use a fully featured CI/CD environment for the pipeline. Otherwise, it will use the local scripts to build and run the pipeline. In the diagrams below dashed boxes show areas users can select and customize their tooling. -**
Cloud Build option:
** -

- CICD -

- **
Github Actions option:
**

CICD @@ -478,6 +463,8 @@ AutoMLOps.generate(project_id=PROJECT_ID, workload_identity_service_account='workload_identity_sa') ``` +More specific details for setting up AutoMLOps to use Github and Github Actions can be found in [this doc](docs/Using%20Github%20With%20AMO.md). + **Set scheduled run:** Use the `schedule_pattern` parameter to specify a cron job schedule to run the pipeline job on a recurring basis. diff --git a/assets/deploy/github_actions/image1.png b/assets/deploy/github_actions/image1.png new file mode 100644 index 0000000..4ee11ff Binary files /dev/null and b/assets/deploy/github_actions/image1.png differ diff --git a/assets/deploy/github_actions/image2.png b/assets/deploy/github_actions/image2.png new file mode 100644 index 0000000..eb06cf5 Binary files /dev/null and b/assets/deploy/github_actions/image2.png differ diff --git a/assets/deploy/github_actions/image3.png b/assets/deploy/github_actions/image3.png new file mode 100644 index 0000000..6676946 Binary files /dev/null and b/assets/deploy/github_actions/image3.png differ diff --git a/assets/deploy/github_actions/image4.png b/assets/deploy/github_actions/image4.png new file mode 100644 index 0000000..e8b175a Binary files /dev/null and b/assets/deploy/github_actions/image4.png differ diff --git a/assets/deploy/github_actions/image5.png b/assets/deploy/github_actions/image5.png new file mode 100644 index 0000000..2c01b08 Binary files /dev/null and b/assets/deploy/github_actions/image5.png differ diff --git a/assets/deploy/github_actions/image6.png b/assets/deploy/github_actions/image6.png new file mode 100644 index 0000000..67a8338 Binary files /dev/null and b/assets/deploy/github_actions/image6.png differ diff --git a/docs/Using Github With AMO.md b/docs/Using Github With AMO.md new file mode 100644 index 0000000..57ee73f --- /dev/null +++ b/docs/Using Github With AMO.md @@ -0,0 +1,149 @@ +**Using Github With AMO** + +The steps below are required in order to deploy a model using the AutoMLOps package. The process below assumes that the [AutoMLOps](https://github.com/GoogleCloudPlatform/automlops) package is already installed in the environment where the AutoMLOps execution cells are being run. + +## **Create a Repository** + +Go to Github, and create a new repository leaving fields as default. Let’s call it automlops-test. + +

+ image1 +

+ +\[Optional\] Create repository via Terraform + +``` +resource "github_repository" "my_repo" { + name = "my_cool_repo" + description = "My awesome codebase" + visibility = "private" +} +``` + +## **Set up SSH connection** + +Follow [this guide](https://docs.github.com/en/authentication/connecting-to-github-with-ssh) on how to add your SSH keys to the environment from where you are running the AutoMLOps commands. + +## **Pre-requisites** + +Ensure that you have the following IAM Roles to provision an Identity Pool: + +* roles/iam.workloadIdentityPoolAdmin + +**Set up [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) (WIF) for GitHub Actions** + +1. Create an Identity Pool in GCP +

+ image2 +

+- Enter a pool name and click ‘Continue’. Example details: + - Name: automlops-1-github-pool + - Pool ID: automlops-1-github-pool + - \ + +2. Create an Identity Provider and add it to the pool: + +

+ image3 +

+ +

+ image4 +

+ +- Fill in the attributes and conditions for the provider with the following details: + + - #### Provider type: OpenID Connect (OIDC) + + - Provider Name: automlops-github-provider + - Provider ID: automlops-github-provider + - Issuer (URL): https://token.actions.githubusercontent.com/ + - JWK file (JSON): \ + - Audiences: \