New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to specify schema while reading file from S3 bucket #72

Open

dkommineni opened this issue May 17, 2021 · 2 comments

dkommineni commented May 17, 2021

How to specify schema while reading file from S3 bucket? I have S3 object which contains one of the column as DATE, when I read without schema, I get the DATE as integer value?

{
VERSION_KEY: 2,
VERSION_DESCRIPTION: 'XXX',
EFFECTIVE_DATE: '43200000000000'
}

I am using the method
const reader = await parquet.ParquetReader.openS3(client, params);

nikolaip commented Nov 12, 2021

@dkommineni ... Did you find a workaround for this? The number isn't a timestamp so unless I am missing something, there is no way to turn the number into a date.

nikolaip commented Nov 15, 2021

Just in case someone else comes upon this .. my issue stemmed from the field being an int96 which is no longer officially part of the Parquet spec and so, not supported by this library which makes sense.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment