Skip to content

Commit

Permalink
HPCC-31571 Parquet Plugin supports invalid compression options
Browse files Browse the repository at this point in the history
  • Loading branch information
jackdelv committed Apr 5, 2024
1 parent 0c66669 commit bfc1389
Show file tree
Hide file tree
Showing 2 changed files with 1 addition and 5 deletions.
2 changes: 1 addition & 1 deletion plugins/parquet/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ dataset := ParquetIO.Read(layout, '/source/directory/data.parquet');

The Write function empowers ECL programmers to write ECL datasets to Parquet files. By leveraging the Parquet format's columnar storage capabilities, this function provides efficient compression and optimized storage for data. There is an optional argument that sets the overwrite behavior of the plugin. The default value is false meaning it will throw an error if the target file already exists. If overwrite is set to true the plugin will check for files that match the target path passed in and delete them first before writing new files.

The Parquet Plugin supports all available Arrow compression types. Specifying the compression when writing is optional and defaults to Uncompressed. The options for compressing your files are Snappy, GZip, Brotli, LZ4, LZ4Frame, LZ4Hadoop, ZSTD, Uncompressed.
The Parquet Plugin supports all available Arrow compression types. Specifying the compression when writing is optional and defaults to Uncompressed. The options for compressing your files are Snappy, GZip, Brotli, LZ4, ZSTD, Uncompressed.

```
overwriteOption := TRUE;
Expand Down
4 changes: 0 additions & 4 deletions plugins/parquet/parquetembed.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1874,10 +1874,6 @@ ParquetEmbedFunctionContext::ParquetEmbedFunctionContext(const IContextLogger &_
compressionOption = arrow::Compression::BROTLI;
else if (strieq(val, "lz4"))
compressionOption = arrow::Compression::LZ4;
else if (strieq(val, "lz4frame"))
compressionOption = arrow::Compression::LZ4_FRAME;
else if (strieq(val, "lz4hadoop"))
compressionOption = arrow::Compression::LZ4_HADOOP;
else if (strieq(val, "zstd"))
compressionOption = arrow::Compression::ZSTD;
else if (strieq(val, "uncompressed"))
Expand Down

0 comments on commit bfc1389

Please sign in to comment.