🐛 DB to DW: Large numerical fields are exported as Float like scientific numbers.

[auyer]

Environment

• Airbyte version: 0.36.0
• OS Version / Instance: Linux EKS Kubernetes
• Deployment: Helm Chart
• Source Connector and version: Postgres 0.4.11, 0.4.10, Mysql 0.5.9, 0.5.6, Mongo V2 0.1.11, probably others
• Destination Connector and version: S3 0.3.1, BigQuery 0.6.3, probably others
• Severity: Critical
• Step where error happened: Sync job

Current Behavior

Large numerical fields like BIGINT, BIG SERIAL and probably others are converted to float and presented in scientific notation. This makes the IDs unusable. My guess is that this happens during Avro conversions.

Expected Behavior

Numbers should keep their original value. If they are float, they should keep the same precision. In this case, they are integers, and its event more critical that they keep their values.

Steps to Reproduce

1. Get a large number in any of the mentioned sources, but probably many more.
2. Sync it through Airbyte to S3 (parquet) or BigQuery. I did not test other formats.
3. See results as scientific float like value

Example:

[auyer]

Related to #9590.
Since this is not restricted to Mongo / BigQuery, I opened a new issue.

[auyer]

It still happens with Postgres Source connector version 0.4.12 and Aribyte version v0.36.4-alpha

[grishick]

Hey team! Please add your planning poker estimate with ZenHub @tuliren @edgao @noahkawasaki-airbyte

[misteryeo]

Issue was linked to Harvestr Discovery: Improve DB/DW data mappings

[edgao]

should be fixed as of #14362 - could you give it a try?

[joelluijmes]

@edgao unfortunately the issue is still present for mongo -> bigquery (#9590 (comment)).