Not getting all columns while streaming parquet file from S3 or locally downloaded file reading

[rajan596]

Hi,

I am using standard code mentioned below. I am not getting desired columns in cursor. Can anyone tell where is the issue in library ?

Desired columns in S3 parquet is A,B,C while I am not getting column A most of the records.
While validating same parquet file downloading local and converting it to CSV A column value is present for all the dataset.
Please help where can this go wrong ?

Version: "parquetjs-lite": "0.8.0",
NodeJs version: v8.0.0

import parquet from 'parquetjs-lite/parquet'
let reader = await parquet.ParquetReader.openS3(s3Client, params);
let cursor = reader.getCursor();
while (record = await cursor.next()) {
 console.log(record)
}

[rajan596]

Update: same is completely readable by using the python panda module.
The parquet file present in s3 was originated by spark/python.

[entitycs]

Have you found any patterns in which rows have 'column A' data missing?  Are they strings or numbers (eg. BigInt?) Regards, Dustin

…

-----Original message----- From: Rajan Kasodariya Sent: Sunday, January 24 2021, 6:24 am To: ZJONSSON/parquetjs Cc: Dustin Charles; Comment Subject: Re: [ZJONSSON/parquetjs] Not getting all columns while streaming parquet file from S3 or locally downloaded file reading (#60)     Update: same is completely readable by using the python panda module. The parquet file present in s3 was originated by spark/python. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[rajan596]

@entitycs I can not share the parquerjs file but there was one pattern for all the parquet files which I tried to read. It was giving this field missing after 70k lines of data read.
Not sure about the data type but it was in form of number.

[entitycs]

@rajan596 When you get a chance, can you try reverting to this commit, and attempt to read past the 70K mark again?

5277eb8

Without being able to see the data, my hunch is that either the numbers in the field grow, and a compression algorithm not in the lite version takes on a different path, or there is an issue in lib/shred.js, which has differing iteration methods between HEAD and the above commit.