Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SNOW-983635 Allow ZSTD compression algorithm #654

Merged
merged 30 commits into from
Feb 1, 2024
Merged
Changes from 1 commit
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
a7927d5
SNOW-983635 Set ZSTD as default compression algorithm
sfc-gh-lthiede Dec 5, 2023
5db8db3
SNOW-983635 Print supported Compression algos
sfc-gh-lthiede Dec 21, 2023
326f9b4
SNOW-983635 Test creating a variable of type RecyclingBufferPool
sfc-gh-lthiede Dec 21, 2023
fe06a7f
SNOW-983635 Add ZSTD dependency
sfc-gh-lthiede Dec 21, 2023
189e894
SNOW-983635 Add version number to zstd dependency
sfc-gh-lthiede Dec 21, 2023
c95d2fe
SNOW-983635 Sort pom
sfc-gh-lthiede Dec 21, 2023
86c5dc5
SNOW-983635 Use same version of zstd dependency as parquet
sfc-gh-lthiede Dec 21, 2023
efaf824
SNOW-983635 Reexecute probably failing static member initialization
sfc-gh-lthiede Dec 21, 2023
4cee250
SNOW-983635 Print UnsatisfiedLinkError
sfc-gh-lthiede Dec 22, 2023
01f7337
SNOW-983635 Don't relocated com.github.luben.zstd
sfc-gh-lthiede Dec 22, 2023
df2938f
SNOW-983635 Don't use luben.zstd directly
sfc-gh-lthiede Dec 22, 2023
cb52f29
SNOW-983635 Hopefully correctly exclude zstd library from shading
sfc-gh-lthiede Dec 22, 2023
c996081
SNOW-983635 Explicitly exclude zstd from shading in another part of p…
sfc-gh-lthiede Jan 4, 2024
6f8053b
SNOW-983635 Add tests for ZSTD compression
sfc-gh-lthiede Jan 9, 2024
4b48a85
SNOW-983635 Remove test that only made sure that I configured the com…
sfc-gh-lthiede Jan 9, 2024
67c3a69
SNOW-983635 Add zstd-jni to dependencies in pom.xml
sfc-gh-lthiede Jan 15, 2024
7f09641
SNOW-983635 Sort pom
sfc-gh-lthiede Jan 16, 2024
24148e3
SNOW-983635 Remove direct zstd dependency
sfc-gh-lthiede Jan 16, 2024
7143018
SNOW-983635 Add zstd-jni as explicit dependency but exclude it from c…
sfc-gh-lthiede Jan 16, 2024
ab25c68
Delete dependency-reduced-pom.xml
sfc-gh-lthiede Jan 22, 2024
601d01b
SNOW-983635 Add zstd-jni dependency to README.md and make ZSTD non de…
sfc-gh-lthiede Jan 22, 2024
a8cdc30
SNOW-983635 Remove unnecessary new line
sfc-gh-lthiede Jan 23, 2024
ff91de2
SNOW-983635 Parameterise StreamingIngestIT.java to run with GZIP and …
sfc-gh-lthiede Jan 23, 2024
0e743c8
SNOW-983635 Add even more datatype tests with ZSTD
sfc-gh-lthiede Jan 29, 2024
17c4b25
SNOW-983635 consistently use compile in public pom
sfc-gh-lthiede Jan 29, 2024
94c62bf
SNOW-983635 Fix AbstractDataTypeTest and remove zstd-jni from depende…
sfc-gh-lthiede Jan 29, 2024
584e75c
SNOW-983635 Make zstd-jni a runtime dependency
sfc-gh-lthiede Jan 29, 2024
4574c8e
SNOW-983635 Sort dependencies is pom
sfc-gh-lthiede Jan 29, 2024
fa71d94
SNOW-983635 Temporarily set ZSTD as default again to test recent changes
sfc-gh-lthiede Jan 29, 2024
a95b87f
SNOW-983635 Revert setting ZSTD as default compression algorithm
sfc-gh-lthiede Jan 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
SNOW-983635 Temporarily set ZSTD as default again to test recent changes
Description

Testing
sfc-gh-lthiede committed Jan 29, 2024
commit fa71d9435ccc95cd1c2a61f90ca6fb496817f5bb
Original file line number Diff line number Diff line change
@@ -62,7 +62,7 @@ public class ParameterProvider {
public static final int MAX_CHUNKS_IN_BLOB_AND_REGISTRATION_REQUEST_DEFAULT = 100;

public static final Constants.BdecParquetCompression BDEC_PARQUET_COMPRESSION_ALGORITHM_DEFAULT =
Constants.BdecParquetCompression.GZIP;
Constants.BdecParquetCompression.ZSTD;

/* Parameter that enables using internal Parquet buffers for buffering of rows before serializing.
It reduces memory consumption compared to using Java Objects for buffering.*/