Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(catalog/glue): add support for glue catalog namespace operations #173

Merged
merged 4 commits into from
Oct 24, 2024

Conversation

natusioe
Copy link
Contributor

Continuation from the previous PR to support Glue catalog. Add support for more namespace and table operations. In the next PR, remaining operations will be added to complete Glue catalog for tables (e.g. create table).

I mostly followed the same semantics as the pyiceberg Glue catalog for the implementation. This means that Glue database fields (description and location URI) are stored as props. Error handling are also done in a similar way.

Testing

  • Unit tests are added. I am not sure if the project has AWS Glue set up (or some sort of realistic mock) for integration testing but I did not see it run as part of the CI actions.

Updated cmd/iceberg/main.go and manually tested using my personal AWS account:

> go run ./cmd/iceberg create --catalog glue --description "my new catalog" --location-uri "s3://test-namespace/" namespace apache-iceberg-test-00006

> go run ./cmd/iceberg describe --catalog glue namespace apache-iceberg-test-00006
┌──────────────────────────────────────┐
| Key           | Value                |
| ------------------------------------ |
| database_type | ICEBERG              |
| Description   | my new catalog       |
| Location      | s3://test-namespace/ |
└──────────────────────────────────────┘

> go run ./cmd/iceberg list --catalog glue
┌───────────────────────────┐
| IDs                       |
| ------------------------- |
| apache-iceberg-test-00001 |
| apache-iceberg-test-00002 |
| apache-iceberg-test-00003 |
| apache-iceberg-test-00004 |
| apache-iceberg-test-00005 |
| apache-iceberg-test-00006 |
└───────────────────────────┘

> go run ./cmd/iceberg properties --catalog glue get namespace apache-iceberg-test-00006 Description
my new catalog

> go run ./cmd/iceberg properties --catalog glue set namespace apache-iceberg-test-00006 Description "updated description"
updated Description on apache-iceberg-test-00006

> go run ./cmd/iceberg properties --catalog glue get namespace apache-iceberg-test-00006 Description
updated description

> go run ./cmd/iceberg describe --catalog glue namespace apache-iceberg-test-00006
┌──────────────────────────────────────┐
| Key           | Value                |
| ------------------------------------ |
| Description   | updated description  |
| Location      | s3://test-namespace/ |
| database_type | ICEBERG              |
└──────────────────────────────────────┘

@natusioe natusioe force-pushed the glue-catalog-support branch 3 times, most recently from d9bd22b to 16bb119 Compare October 16, 2024 01:47
@nastra nastra self-requested a review October 16, 2024 14:59
Copy link
Member

@zeroshade zeroshade left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally LGTM just one question

catalog/glue.go Show resolved Hide resolved
@nastra nastra merged commit 97c04be into apache:main Oct 24, 2024
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants