Skip to content

Commit

Permalink
Add openBuffer
Browse files Browse the repository at this point in the history
  • Loading branch information
ZJONSSON committed Feb 1, 2018
1 parent 57c03bb commit bdb76c1
Show file tree
Hide file tree
Showing 2 changed files with 19 additions and 0 deletions.
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,14 @@ const params = {
let reader = await parquet.ParquetReader.openS3(client,params);
```

### Reading data from a buffer

If the complete parquet file is in buffer it can be read directly from memory without incurring any additional I/O.

``` js
const file = fs.readFileSync('fruits.parquet');
let reader = await parquet.ParquetReader.openBuffer(file);
```

Encodings
---------
Expand Down
11 changes: 11 additions & 0 deletions lib/reader.js
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,11 @@ class ParquetReader {
return this.openEnvelopeReader(envelopeReader);
}

static async openBuffer(buffer) {
let envelopeReader = await ParquetEnvelopeReader.openBuffer(Buffer);
return this.openEnvelopeReader(envelopeReader);
}

/**
* Open the parquet file from S3 using the supplied aws client and params
* The params have to include `Bucket` and `Key` to the file requested
Expand Down Expand Up @@ -224,6 +229,12 @@ class ParquetEnvelopeReader {
return new ParquetEnvelopeReader(readFn, closeFn, fileStat.size);
}

static async openBuffer(buffer) {
let readFn = (offset, length) => buffer.slice(offset,offset+length);
let closeFn = () => ({});
return new ParquetEnvelopeReader(readFn, closeFn, buffer.length);
}

static async openS3(client, params) {
let fileStat = await client.headObject(params).promise();

Expand Down

0 comments on commit bdb76c1

Please sign in to comment.