Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixes 18156: upgrade pyiceberg; add optional OAuth2 scope #18669

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
+19 −2
Open

Fixes 18156: upgrade pyiceberg; add optional OAuth2 scope #18669

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d1924fc
upgrde pyiceberg; add optional OAuth2 scope
MChamberlin Nov 16, 2024
39c4db4
update tests
MChamberlin Nov 16, 2024
ae545b9
fix: remove oauth2 creds in tests
MChamberlin Nov 18, 2024
42973b6
Merge branch 'main' into FIX-18156-upgrade-pyiceberg-add-scope
MChamberlin Nov 20, 2024
c296a4e
Merge branch 'main' into FIX-18156-upgrade-pyiceberg-add-scope
MChamberlin Nov 25, 2024
c38097e
update dependencies
MChamberlin Nov 25, 2024
32f52b0
relax gcs version
MChamberlin Nov 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions ingestion/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
"boto3": "boto3>=1.20,<2.0", # No need to add botocore separately. It's a dep from boto3
"geoalchemy2": "GeoAlchemy2~=0.12",
"google-cloud-monitoring": "google-cloud-monitoring>=2.0.0",
"google-cloud-storage": "google-cloud-storage==1.43.0",
"google-cloud-storage": "google-cloud-storage>=1.43.0",
"gcsfs": "gcsfs>=2023.1.0",
"great-expectations": "great-expectations>=0.18.0,<0.18.14",
"grpc-tools": "grpcio-tools>=1.47.2",
Expand Down Expand Up @@ -212,6 +212,7 @@
"datalake-s3": {
# vendoring 'boto3' to keep all dependencies aligned (s3fs, boto3, botocore, aiobotocore)
"s3fs[boto3]",
VERSIONS["boto3"],
*COMMONS["datalake"],
},
"deltalake": {
Expand Down Expand Up @@ -240,12 +241,14 @@
"impyla~=0.18.0",
},
"iceberg": {
"pyiceberg==0.5.1",
"pyiceberg[hive,glue,dynamodb,s3fs,adlfs,gcsfs]==0.7.1",
# Forcing the version of a few packages so it plays nicely with other requirements.
VERSIONS["pydantic"],
VERSIONS["adlfs"],
VERSIONS["gcsfs"],
VERSIONS["pyarrow"],
VERSIONS["boto3"],
*COMMONS["hive"],
},
"impala": {
"presto-types-parser>=0.0.2",
Expand Down
1 change: 1 addition & 0 deletions ingestion/src/metadata/ingestion/source/database/iceberg/catalog/rest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ def get_catalog(cls, catalog: IcebergCatalog) -> Catalog:
"warehouse": catalog.warehouseLocation,
"uri": str(catalog.connection.uri),
"credential": credential,
"scope": catalog.connection.credential.scope,
"token": catalog.connection.token.get_secret_value()
if catalog.connection.token
else None,
Expand Down
5 changes: 5 additions & 0 deletions ...urces/json/schema/entity/services/connections/database/iceberg/restCatalogConnection.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,11 @@
"description": "OAuth2 Client Secret",
"type": "string",
"format": "password"
},
"scope": {
"title": "Scope",
"description": "OAuth2 Scope",
"type": "string"
}
},
"additionalProperties": false
Expand Down
8 changes: 8 additions & 0 deletions openmetadata-ui/src/main/resources/ui/public/locales/en-US/Database/Iceberg.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,14 @@ To get the client secret, follow these steps:
$$
$$section

### Scope $(id="scope")

#### REST Catalog
OAuth2 Scope to use for the Authentication Flow

$$
$$section

### Token $(id="token")

#### REST Catalog
Expand Down
Loading