AttributeError: 'list' object has no attribute 'upper'

[git243]

I am going tap-hubspot to target-bigquery, and after everything set-up and run the tap and target in separate virtual environment, the stated error above is prompted. After reviewing, I believe this is because array of array is extracted. Below is the example of data I extracted into csv.

Any solution to solve this problem?

[{'deleted-changed-timestamp': '1970-01-01T00:00:00.000000Z', 'saved-at-timestamp': '2020-03-26T06:05:56.130000Z', 'vid': 301, 'identities': [{'type': 'EMAIL', 'value': '[email protected]', 'timestamp': '2020-03-26T06:05:56.104000Z'}, {'type': 'LEAD_GUID', 'value': 'c985e285-2c5a-40c7-a81f-bd4679bcc881', 'timestamp': '2020-03-26T06:05:56.125000Z'}]}]

[doody-doody]

Yes. Same here. Same independent diagnosis.

The BQ target doesn't like "properties_versions" because properties is already an object (STRUCT) and then versions comes along.

I think it might be a problem with target-bigquery implementation from @saidtezel, but I am still poking around in the code that decomposes a struct to a struct_gut-of-struct in target_bigquery.py.

[saidtezel]

Yes. Same here. Same independent diagnosis.

The BQ target doesn't like "properties_versions" because properties is already an object (STRUCT) and then versions comes along.

I think it might be a problem with target-bigquery implementation from @saidtezel, but I am still poking around in the code that decomposes a struct to a struct_gut-of-struct in target_bigquery.py.

@doody-doody Could you paste the error code you receive while using my BQ repo? I'll have a look.

[doody-doody]

Of course (and thank you for your BigQuery target! And your awesome blog posts on using this target in k8s - https://saidtezel.com/ga-bigquery-replication-part-1).

Error is:

Traceback (most recent call last): File "target-bigquery/bin/target-bigquery", line 11, in <module> sys.exit(main()) File "/home/spaccavd/code/singer/target-bigquery/lib/python3.6/site-packages/target_bigquery.py", line 300, in main state = persist_lines_stream(config['project_id'], config['dataset_id'], input, validate_records=validate_records) File "/home/spaccavd/code/singer/target-bigquery/lib/python3.6/site-packages/target_bigquery.py", line 239, in persist_lines_stream tables[table] = bigquery.Table(dataset.table(table), schema=build_schema(schemas[table])) File "/home/spaccavd/code/singer/target-bigquery/lib/python3.6/site-packages/google/cloud/bigquery/table.py", line 335, in __init__ self.schema = schema File "/home/spaccavd/code/singer/target-bigquery/lib/python3.6/site-packages/google/cloud/bigquery/table.py", line 401, in schema self._properties["schema"] = {"fields": _build_schema_resource(value)} File "/home/spaccavd/code/singer/target-bigquery/lib/python3.6/site-packages/google/cloud/bigquery/schema.py", line 260, in _build_schema_resource return [field.to_api_repr() for field in fields] File "/home/spaccavd/code/singer/target-bigquery/lib/python3.6/site-packages/google/cloud/bigquery/schema.py", line 260, in <listcomp> return [field.to_api_repr() for field in fields] File "/home/spaccavd/code/singer/target-bigquery/lib/python3.6/site-packages/google/cloud/bigquery/schema.py", line 149, in to_api_repr "type": self.field_type.upper(), AttributeError: 'list' object has no attribute 'upper'
Invocation is:
target-bigquery/bin/target-bigquery -c target-bigquery-config.json < hs-stream-short.json

I have attached the two files in the invocation:
target-bigquery-error.zip

System is Ubuntu Linux + Python 3.6 (but I suspect this is a code issue).

Thanks again for looking at this.

[tonyrb]

@git243 & @doody-doody did you manage to find a solution?
I am having the same situation.

[git243]

@git243 & @doody-doody did you manage to find a solution?
I am having the same situation.

Yes, I fixed it by writing another Python script to flatten the elements in the nested arrays.

[tonyrb]

@git243 is it possible to share your script and method with us?
Is it in the process of creating the catalog file or i am wrong all over?

I have tried to use target-csv instead of target-bigquery and have the same problem (csv file is corrupt with empty columns and data in other columns) so probably due to the same kind of problem.

[doody-doody]

@git243 Good idea! Why did I not think of that!? That is much simpler than trying to fix it in @saidtezel's code.

If you are happy to share, I also would appreciate your script (thank you).

[YingWang-Clare]

Of course (and thank you for your BigQuery target! And your awesome blog posts on using this target in k8s - https://saidtezel.com/ga-bigquery-replication-part-1).

Error is:

Traceback (most recent call last): File "target-bigquery/bin/target-bigquery", line 11, in <module> sys.exit(main()) File "/home/spaccavd/code/singer/target-bigquery/lib/python3.6/site-packages/target_bigquery.py", line 300, in main state = persist_lines_stream(config['project_id'], config['dataset_id'], input, validate_records=validate_records) File "/home/spaccavd/code/singer/target-bigquery/lib/python3.6/site-packages/target_bigquery.py", line 239, in persist_lines_stream tables[table] = bigquery.Table(dataset.table(table), schema=build_schema(schemas[table])) File "/home/spaccavd/code/singer/target-bigquery/lib/python3.6/site-packages/google/cloud/bigquery/table.py", line 335, in __init__ self.schema = schema File "/home/spaccavd/code/singer/target-bigquery/lib/python3.6/site-packages/google/cloud/bigquery/table.py", line 401, in schema self._properties["schema"] = {"fields": _build_schema_resource(value)} File "/home/spaccavd/code/singer/target-bigquery/lib/python3.6/site-packages/google/cloud/bigquery/schema.py", line 260, in _build_schema_resource return [field.to_api_repr() for field in fields] File "/home/spaccavd/code/singer/target-bigquery/lib/python3.6/site-packages/google/cloud/bigquery/schema.py", line 260, in <listcomp> return [field.to_api_repr() for field in fields] File "/home/spaccavd/code/singer/target-bigquery/lib/python3.6/site-packages/google/cloud/bigquery/schema.py", line 149, in to_api_repr "type": self.field_type.upper(), AttributeError: 'list' object has no attribute 'upper'
Invocation is:
target-bigquery/bin/target-bigquery -c target-bigquery-config.json < hs-stream-short.json

I have attached the two files in the invocation:
target-bigquery-error.zip

System is Ubuntu Linux + Python 3.6 (but I suspect this is a code issue).

Thanks again for looking at this.

Hello @doody-doody , have you got any response from @saidtezel to resolve this issue? Or have you figured it out? I met exactly the same question as yours. I used tap-facebook and target-bigquery.