Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SNOW-1501662 Allow JDBC to handle ZSTD compressed http response streams #1802

Conversation

sfc-gh-dbouassida
Copy link
Contributor

@sfc-gh-dbouassida sfc-gh-dbouassida commented Jun 24, 2024

Overview

SNOW-1501662

Summary

This PR introduces the integration of the zstd-jni library into Snowflake JDBC.

Key Changes

  1. Dependency Addition:

    • Added zstd-jni version 1.5.6-5 as a dependency to support Zstandard compression.
    • Updated relevant POM files (pom.xml, FIPS/pom.xml, thin_public_pom.xml, parent-pom.xml) to include the zstd-jni dependency.
  2. Shading and Relocation:

    • Updated the Maven Shade Plugin configuration to relocate com.github.luben.zstd into ${shadeBase}.com.github.luben.zstd.
    • Modified FIPS/pom.xml and pom.xml to include relocation rules for the zstd-jni package.
  3. Code Integration:

    • Modified DefaultResultStreamProvider.java to handle streams encoded with Zstandard. The code now checks for the zstd encoding and uses ZstdInputStream for decompression.
  4. Script Updates:

    • Updated ci/scripts/check_content.sh and FIPS/scripts/check_content.sh to ignore native library files (.so, .dylib, .dll) located in OS-specific directories (aix, darwin, freebsd, linux, win). This prevents errors during the dependency shading check.
  5. Linkage Checker Updates:

    • Removed unnecessary exclusions related to com.github.luben.zstd in linkage-checker-exclusion-rules.xml, as the dependencies are now properly shaded and relocated.

Pre-review self checklist

  • PR branch is updated with all the changes from master branch
  • The code is correctly formatted (run mvn -P check-style validate)
  • New public API is not unnecessary exposed (run mvn verify and inspect target/japicmp/japicmp.html)
  • The pull request name is prefixed with SNOW-XXXX:
  • Code is in compliance with internal logging requirements

External contributors - please answer these questions before submitting a pull request. Thanks!

  1. What GitHub issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

    Fixes #NNNN

  2. Fill out the following pre-review checklist:

    • I am adding a new automated test(s) to verify correctness of my new code
    • I am adding new logging messages
    • I am modifying authorization mechanisms
    • I am adding new credentials
    • I am modifying OCSP code
    • I am adding a new dependency or upgrading an existing one
    • I am adding new public/protected component not marked with @SnowflakeJdbcInternalApi (note that public/protected methods/fields in classes marked with this annotation are already internal)
  3. Please describe how your code solves the related issue.

    Please write a short description of how your code change solves the related issue.

Copy link

github-actions bot commented Jun 24, 2024

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@sfc-gh-dbouassida sfc-gh-dbouassida changed the title SNOW-1452649 update dependency to httpclient library to support zstd SNOW-1452649 Allow JDBC to handle ZSTD compressed query results Jun 24, 2024
@sfc-gh-dbouassida sfc-gh-dbouassida force-pushed the dbouassida-SNOW-1501662-allow-jdbc-to-decompress-zstd-compressed-query-results branch from b2a52a0 to b789b62 Compare September 3, 2024 12:26
@sfc-gh-dbouassida sfc-gh-dbouassida marked this pull request as ready for review September 3, 2024 15:26
@sfc-gh-dbouassida sfc-gh-dbouassida requested a review from a team as a code owner September 3, 2024 15:26
@sfc-gh-dbouassida sfc-gh-dbouassida marked this pull request as draft September 4, 2024 09:49
thin_public_pom.xml Outdated Show resolved Hide resolved
parent-pom.xml Outdated Show resolved Hide resolved
@sfc-gh-dbouassida sfc-gh-dbouassida force-pushed the dbouassida-SNOW-1501662-allow-jdbc-to-decompress-zstd-compressed-query-results branch from 2d50eb2 to 0748b77 Compare September 5, 2024 13:08
@sfc-gh-dbouassida sfc-gh-dbouassida force-pushed the dbouassida-SNOW-1501662-allow-jdbc-to-decompress-zstd-compressed-query-results branch from 0748b77 to 7ed1497 Compare September 5, 2024 15:14
@sfc-gh-dbouassida sfc-gh-dbouassida marked this pull request as ready for review September 5, 2024 15:15
@sfc-gh-dbouassida
Copy link
Contributor Author

I have read the CLA Document and I hereby sign the CLA

@sfc-gh-dbouassida sfc-gh-dbouassida changed the title SNOW-1452649 Allow JDBC to handle ZSTD compressed query results SNOW-1501662 Allow JDBC to handle ZSTD compressed query results Sep 6, 2024
Copy link

@sfc-gh-ioukid sfc-gh-ioukid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@sfc-gh-dbouassida sfc-gh-dbouassida changed the title SNOW-1501662 Allow JDBC to handle ZSTD compressed query results SNOW-1501662 Allow JDBC to handle ZSTD compressed http response streams Sep 9, 2024
@sfc-gh-dbouassida sfc-gh-dbouassida merged commit d29dea9 into master Sep 10, 2024
142 checks passed
@sfc-gh-dbouassida sfc-gh-dbouassida deleted the dbouassida-SNOW-1501662-allow-jdbc-to-decompress-zstd-compressed-query-results branch September 10, 2024 08:14
@github-actions github-actions bot locked and limited conversation to collaborators Sep 10, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants