[BUGFIX] Added get_partition pagination #82
Conversation
| ) | ||
| p = re.compile('s3://([^/]*)/(.*)') | ||
| for partition in partitions["Partitions"]: | ||
| paginator = glue_client.get_paginator("get_partitions") |
There was a problem hiding this comment.
if you have a chance, can you check what would happen if the current user doesn't have enough permission to fetch the partitions?
| ) | ||
| p = re.compile('s3://([^/]*)/(.*)') | ||
| for partition in partitions["Partitions"]: | ||
| paginator = glue_client.get_paginator("get_partitions") |
There was a problem hiding this comment.
if you have a chance, can you check what would happen if the current user doesn't have enough permission to fetch the partitions?
There was a problem hiding this comment.
You bet. I should have some time to look into it tomorrow to check the permission case and run through the proper dbt-athena testing.
There was a problem hiding this comment.
I have validated that the dbt run fails before doing any work when permission to access the permissions is not available:
14:48:34 Runtime Error in model epic_835 (models/test.sql)
14:48:34 Insufficient permissions to execute the query. User: dustin.verzal is not authorized to perform: glue:GetPartitions on resource: arn:aws:glue:us-east-2:********:catalog with an explicit deny in an identity-based policy . You may need to manually clean the data at location 'PATH' before retrying. Athena will not delete data in your account.
There was a problem hiding this comment.
@DVerzal this is an issue due to missing permissions. You are missing a glue:GetPartitions on the table.
…nto bugfix/81_get_partition_pagination
| "Expression": where_condition, | ||
| "ExcludeColumnSchema": True, | ||
| } | ||
| partition_pg = paginator.paginate(**partition_params) |
There was a problem hiding this comment.
alternatively you could do this
partitions = partition_pg.build_full_result().get('Partitions')
Doing so you avoid a for loop, that will happen automatically when calling build_full_result method.
Addressed get_partition pagination to handle multiple pages of partitions.