Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[E2E Databricks] - Sprint 4 changes #1043

Merged
merged 46 commits into from
Jan 21, 2025
Merged

Conversation

ydaponte
Copy link
Collaborator

Type of PR

  • Documentation changes
  • Code changes
  • Test changes
  • CI-CD changes
  • GitHub Template changes

Purpose

  • This PR intent is to merge all the work done during Sprint 4 to the main branch

Does this introduce a breaking change? If yes, details on what can break

Author pre-publish checklist

  • Added test to prove my fix is effective or new feature works
  • No PII in logs
  • Made corresponding changes to the documentation

Validation steps

  • Deploy repository
  • Validate that bicep warnings are not appearing anymore
  • Validate that service connections in AzDO are now following the best practice using workload identities
  • Validate VM type with least resources for DBX cluster is automatically being selected aligned with the current deployment region
  • Test clean-up script to validate that all App registrations are being deleted
  • Revert bug - DBX cluster version from 15 to 14

Issues Closed or Referenced

ydaponte and others added 30 commits December 23, 2024 14:17
#988)

* automate region vm + dbx type cluster in the region

* Update e2e_samples/parking_sensors/scripts/configure_databricks.sh

Co-authored-by: Elena Terenzi <[email protected]>

* filtering photon support

---------

Co-authored-by: Elena Terenzi <[email protected]>
* fix breaking change from dbx cli output
* fix hyperlink check
Adding the Cleanup for federal credentials which follows a certain order:
Get the SP Objc ID of the Service Connection
Delete federated credentials from the SP Object ID
Delete the service connection
Service Connections are using the Workload Identity which comes from a configuration file.

The cleanup in case of re-start of this script also follows a peculiar order: 

Get SP object if according to the Service Connection
Get the federate credential according to the SP
Delete federated credentials
Delete the Service Connection
changing the docs with the required permission
rephrase the permissions needed for azdevops project
I adjust the order of the permissions
adding know issues how to delete federate credentials
identation was not correct
adding the following changes:
1) Function for cleanup
2)Function for sleep
3)Remove log file
Implement function for cleanup and function for sleep
Added cleanup_federate_credentials and wait_for_credentials used in the clean_up.sh and deploy_azdo_service_connection.sh
rename wait_for_cleanup to wait_for_process
renamed wait_for_cleanup to wait_for_process
typo:
comtain to contain fxed
added Elena's suggestion of rewrite

Co-authored-by: Elena Terenzi <[email protected]>
remove the log for SP
Clean up of the log
suppress update pipelines message
remove response from logging
changing the log for federate credentials removal
adding the SP in the log message
checking if the file exist before removal.
@ydaponte ydaponte added bug Something isn't working documentation Improvements or additions to documentation e2e: parking-sensors-databricks labels Jan 16, 2025
Copy link
Contributor

@bsherwin bsherwin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

…val (#1047)

* remove federated credentials only when needed

* fix typo
Copy link
Contributor

@elenaterenzi elenaterenzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM the clean up script is not deleting everything though, I will add a task for this we can fix it in the upcoming sprint

@ydaponte ydaponte merged commit 7597878 into main Jan 21, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working documentation Improvements or additions to documentation e2e: parking-sensors-databricks
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants