Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dagster-azure] add support for US Government Cloud tenant #25738

Open
wants to merge 24 commits into
base: master
Choose a base branch
from

Conversation

lilygrierjil
Copy link

Summary & Motivation

This PR addresses issue #7790 by removing the hard-coding to the Azure commercial cloud tenant and adding a cloud_type parameter that can accept either public or government, which can be specified in the ADLS2 configurations. This value defaults to public, so existing code calling these functions can run without breaking. The PR adds a primary_endpoint parameter to ADLS2FileHandle so the appropriate subdomain gets propagated from the ADLS2Resource.

How I Tested These Changes

  • added test_adls2_pickle_io_manager_uri to ensure the correct URI is generated
  • added test_adls_file_handle_adls2_path to ensure the file path changes based on the cloud_type parameter (and that the path reflects the public cloud tenant when no cloud_type is supplied)
  • added test_adls_file_manager_resource_cloud_type to ensure assets are materialized without error when specifying the government cloud type
  • ran pytest dagster_azure_tests/ to ensure all tests (old and new) run successfully

Changelog

You can now specify either public or government as cloud_type as a parameter to ADLS2Resource() and create_adls2_client(), and in the ADLS resource .yaml configuration. This parameter is optional and defaults to public.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant