Parameter validation failed CatalogId

[jessedobbelaere]

Since #370 I'm running into this error on dbt-athena 1.6.1:

13:04:23  2 of 13 START sql table model dev_branch_data_3570_xxx.test  [RUN]
13:04:26  Unhandled error while executing 
Parameter validation failed:
Invalid type for parameter CatalogId, value: None, type: <class 'NoneType'>, valid types: <class 'str'>

I looked at the glue client method: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue/client/get_table.html

I don't think it supports None. Either don't add the CatalogId option or use a string. I can reproduce it with the following code:

import boto3
from threading import Lock

boto3_client_lock = Lock()
with boto3_client_lock:
    glue_client = boto3.client('glue', region_name='eu-central-1')
    # Passing in `None` will make it fail, it only supports string types
    table = glue_client.get_tables(CatalogId=None, DatabaseName='mydb')

  File "/Users/xxx/xxx/local-development/data-mesh/xxx/test.py", line 7, in <module>
    table = glue_client.get_tables(CatalogId=None, DatabaseName='stag_silver_xxx_pii')
  File "/Users/xxx/xxx/local-development/data-mesh/xxx/.venv/lib/python3.10/site-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/xxx/xxx/local-development/data-mesh/xxx/.venv/lib/python3.10/site-packages/botocore/client.py", line 919, in _make_api_call
    request_dict = self._convert_to_request_dict(
  File "/Users/xxx/xxx/local-development/data-mesh/xxx/.venv/lib/python3.10/site-packages/botocore/client.py", line 990, in _convert_to_request_dict
    request_dict = self._serializer.serialize_to_request(
  File "/Users/xxx/xxx/local-development/data-mesh/xxx/.venv/lib/python3.10/site-packages/botocore/validate.py", line 381, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter CatalogId, value: None, type: <class 'NoneType'>, valid types: <class 'str'>

I'll see if I can prepare a fix

[jessedobbelaere]

As found in #411, the reason that Glue client gets called with a CatalogId=None is because the data catalog (=database) is not detected.

In a custom macro (insert-by-period) we are creating a tmp relation from scratch but that did not specify the database key:

-- Should always mention both database and schema! Using: database=target_relation.database
{%- set tmp_relation = api.Relation.create(schema=schema, identifier=tmp_identifier, type='table') -%}

The issue was:

# relation.database returns None because it was not specified in the macro
data_catalog = self._get_data_catalog(relation.database)
catalog_id = get_catalog_id(data_catalog)