Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SNOW-1651983: Upgrade azure storage sdk v8 to v12, fix authenticated proxy issue #1895

Draft
wants to merge 6 commits into
base: master
Choose a base branch
from
+517 −248
Draft

SNOW-1651983: Upgrade azure storage sdk v8 to v12, fix authenticated proxy issue #1895

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
20242c6
SNOW-1651983 authenticated proxy on azure
sfc-gh-mhofman Sep 5, 2024
0e8de1a
Formatting
sfc-gh-astachowski Sep 6, 2024
8656d34
Initial azure-storage-blob sdk bump
sfc-gh-mkubik Sep 16, 2024
f4ae228
fix merge conflicts - apply master changes in pom, use case insensiti…
sfc-gh-mkubik Oct 28, 2024
d045bee
fix merge conficts
sfc-gh-mkubik Nov 4, 2024
d117a6b
add old api with storage sdk v8 to prevent BCR
sfc-gh-mkubik Nov 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion parent-pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,9 @@
<avro.version>1.8.1</avro.version>
<awaitility.version>4.2.0</awaitility.version>
<awssdk.version>1.12.655</awssdk.version>
<azure.storage.version>5.0.0</azure.storage.version>
<azure.storage.blob.version>12.26.1</azure.storage.blob.version>
<azure.storage.version>8.0.0</azure.storage.version>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the newest is 8.6.6 - can we use it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, I haven't noticed the newer version but it'll be rather no-op anyway

<azuresdk.version>1.2.29</azuresdk.version>
<bouncycastle.version>1.78.1</bouncycastle.version>
<bouncycastle.bcfips.version>1.0.2.5</bouncycastle.bcfips.version>
<bouncycastle.bcpkixfips.version>1.0.7</bouncycastle.bcpkixfips.version>
Expand Down Expand Up @@ -115,6 +117,13 @@
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-sdk-bom</artifactId>
<version>${azuresdk.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson</groupId>
<artifactId>jackson-bom</artifactId>
Expand Down Expand Up @@ -586,6 +595,18 @@
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client</artifactId>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-storage-blob</artifactId>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-core</artifactId>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-storage-common</artifactId>
</dependency>
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-storage</artifactId>
Expand Down
12 changes: 12 additions & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -811,6 +811,10 @@
<pattern>software.amazon.ion</pattern>
<shadedPattern>${shadeBase}.software.amazon.ion</shadedPattern>
</relocation>
<relocation>
<pattern>com.azure</pattern>
<shadedPattern>${shadeBase}.azure</shadedPattern>
</relocation>
<relocation>
<pattern>com.microsoft.azure</pattern>
<shadedPattern>${shadeBase}.microsoft.azure</shadedPattern>
Expand Down Expand Up @@ -875,6 +879,14 @@
<pattern>io.netty</pattern>
<shadedPattern>${shadeBase}.io.netty</shadedPattern>
</relocation>
<relocation>
<pattern>io.projectreactor</pattern>
<shadedPattern>${shadeBase}.io.projectreactor</shadedPattern>
</relocation>
<relocation>
<pattern>org.reactivestreams</pattern>
<shadedPattern>${shadeBase}.org.reactivestreams</shadedPattern>
</relocation>
<relocation>
<pattern>com.carrotsearch</pattern>
<shadedPattern>${shadeBase}.com.carrotsearch</shadedPattern>
Expand Down
142 changes: 130 additions & 12 deletions src/main/java/net/snowflake/client/core/HttpUtil.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,10 @@
import static org.apache.http.client.config.CookieSpecs.IGNORE_COOKIES;

import com.amazonaws.ClientConfiguration;
import com.azure.core.http.ProxyOptions;
import com.azure.core.util.HttpClientOptions;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Strings;
import com.microsoft.azure.storage.OperationContext;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
Expand All @@ -30,6 +31,8 @@
import java.util.concurrent.atomic.AtomicBoolean;
import javax.annotation.Nullable;
import javax.net.ssl.TrustManager;

import com.microsoft.azure.storage.OperationContext;
import net.snowflake.client.jdbc.ErrorCode;
import net.snowflake.client.jdbc.RestRequest;
import net.snowflake.client.jdbc.SnowflakeDriver;
Expand Down Expand Up @@ -178,33 +181,38 @@ public static void setSessionlessProxyForS3(
S3HttpUtil.setSessionlessProxyForS3(proxyProperties, clientConfig);
}


/**
* A static function to set Azure proxy params for sessionless connections using the proxy params
* from the StageInfo
*
* @param proxyProperties proxy properties
* @param opContext the configuration needed by Azure to set the proxy
* @throws SnowflakeSQLException
*
* @deprecated, please use {@link HttpUtil#setSessionlessProxyForAzure(Properties, HttpClientOptions)} as
* it supports the up-to-date azure storage client.
*/
@Deprecated
public static void setSessionlessProxyForAzure(
Properties proxyProperties, OperationContext opContext) throws SnowflakeSQLException {
Properties proxyProperties, OperationContext opContext) throws SnowflakeSQLException {
if (proxyProperties != null
&& proxyProperties.size() > 0
&& proxyProperties.getProperty(SFSessionProperty.USE_PROXY.getPropertyKey()) != null) {
&& proxyProperties.size() > 0
&& proxyProperties.getProperty(SFSessionProperty.USE_PROXY.getPropertyKey()) != null) {
Boolean useProxy =
Boolean.valueOf(
proxyProperties.getProperty(SFSessionProperty.USE_PROXY.getPropertyKey()));
Boolean.valueOf(
proxyProperties.getProperty(SFSessionProperty.USE_PROXY.getPropertyKey()));
if (useProxy) {
String proxyHost =
proxyProperties.getProperty(SFSessionProperty.PROXY_HOST.getPropertyKey());
proxyProperties.getProperty(SFSessionProperty.PROXY_HOST.getPropertyKey());
int proxyPort;
try {
proxyPort =
Integer.parseInt(
proxyProperties.getProperty(SFSessionProperty.PROXY_PORT.getPropertyKey()));
Integer.parseInt(
proxyProperties.getProperty(SFSessionProperty.PROXY_PORT.getPropertyKey()));
} catch (NumberFormatException | NullPointerException e) {
throw new SnowflakeSQLException(
ErrorCode.INVALID_PROXY_PROPERTIES, "Could not parse port number");
ErrorCode.INVALID_PROXY_PROPERTIES, "Could not parse port number");
}
Proxy azProxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort));
logger.debug("Setting sessionless Azure proxy. Host: {}, port: {}", proxyHost, proxyPort);
Expand All @@ -217,24 +225,117 @@ public static void setSessionlessProxyForAzure(
}
}

/**
* A static function to set Azure proxy params for sessionless connections using the proxy params
* from the StageInfo
*
* @param proxyProperties proxy properties
* @param httpClientOptions the configuration needed by Azure to set the proxy
* @throws SnowflakeSQLException
*/
public static void setSessionlessProxyForAzure(
Properties proxyProperties, HttpClientOptions httpClientOptions) throws SnowflakeSQLException {
if (proxyProperties == null
|| proxyProperties.isEmpty()
|| proxyProperties.getProperty(SFSessionProperty.USE_PROXY.getPropertyKey()) == null) {
logger.debug("Omitting sessionless Azure proxy setup");
return;
}

boolean useProxy =
Boolean.parseBoolean(proxyProperties.getProperty(SFSessionProperty.USE_PROXY.getPropertyKey()));
if (!useProxy) {
logger.debug("Omitting sessionless Azure proxy setup as proxy is disabled");
}

// TODO: extract common logic
String proxyHost =
proxyProperties.getProperty(SFSessionProperty.PROXY_HOST.getPropertyKey());
int proxyPort;
try {
proxyPort =
Integer.parseInt(
proxyProperties.getProperty(SFSessionProperty.PROXY_PORT.getPropertyKey()));
} catch (NumberFormatException | NullPointerException e) {
throw new SnowflakeSQLException(
ErrorCode.INVALID_PROXY_PROPERTIES, "Could not parse port number");
}

String proxyUser = proxyProperties.getProperty(SFSessionProperty.PROXY_USER.getPropertyKey());
String proxyPassword = proxyProperties.getProperty(SFSessionProperty.PROXY_PASSWORD.getPropertyKey());

boolean setProxyUser =
!Strings.isNullOrEmpty(proxyUser);
// Isn't blank password allowed? Also, the userid I guess. Source: RFC 2617
// && !Strings.isNullOrEmpty(proxyPassword);

ProxyOptions proxyOptions = new ProxyOptions(ProxyOptions.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort));
if (setProxyUser) {
proxyOptions.setCredentials(proxyUser, proxyPassword);
}

logger.info(
"Setting Azure proxy {} user. Host: {}, port: {}",
setProxyUser ? "with" : "without",
proxyHost,
proxyPort);

httpClientOptions.setProxyOptions(proxyOptions);
}

/**
* A static function to set Azure proxy params when there is a valid session
*
* @param key key to HttpClient map containing OCSP and proxy info
* @param opContext the configuration needed by Azure to set the proxy
*
* @deprecated Use {@link HttpUtil#setProxyForAzure(HttpClientSettingsKey, OperationContext)} as it supports
* the most recent azure storage client
*/
@Deprecated
public static void setProxyForAzure(HttpClientSettingsKey key, OperationContext opContext) {
if (key != null && key.usesProxy()) {
Proxy azProxy =
new Proxy(Proxy.Type.HTTP, new InetSocketAddress(key.getProxyHost(), key.getProxyPort()));
new Proxy(Proxy.Type.HTTP, new InetSocketAddress(key.getProxyHost(), key.getProxyPort()));
logger.debug(
"Setting Azure proxy. Host: {}, port: {}", key.getProxyHost(), key.getProxyPort());
"Setting Azure proxy. Host: {}, port: {}", key.getProxyHost(), key.getProxyPort());
opContext.setProxy(azProxy);
} else {
logger.debug("Omitting Azure proxy setup");
}
}

/**
* A static function to set Azure proxy params when there is a valid session
*
* @param key key to HttpClient map containing OCSP and proxy info
* @param httpClientOptions the configuration needed by Azure to set the proxy
*/
public static void setProxyForAzure(HttpClientSettingsKey key, HttpClientOptions httpClientOptions) {
if (key != null && key.usesProxy()) {
ProxyOptions proxyOptions =
new ProxyOptions(ProxyOptions.Type.HTTP, new InetSocketAddress(key.getProxyHost(), key.getProxyPort()));
boolean setProxyUser =
!Strings.isNullOrEmpty(key.getProxyUser());
// Isn't blank password allowed? Also, the userid I guess. Source: RFC 2617
// && !Strings.isNullOrEmpty(key.getProxyPassword());
logger.info(
"Setting Azure proxy {} user. Host: {}, port: {}",
setProxyUser ? "with" : "without",
key.getProxyHost(),
key.getProxyPort());
if (setProxyUser) {
proxyOptions.setCredentials(key.getProxyUser(), key.getProxyPassword());
logger.info("Azure proxy user: {}, azure proxy password: {}", key.getProxyUser(), key.getProxyPassword());
}

httpClientOptions.setProxyOptions(proxyOptions);

} else {
logger.debug("Omitting Azure proxy setup");
}
}

/**
* Constructs a user-agent header with the following pattern: connector_name/connector_version
* (os-platform_info) language_implementation/language_version
Expand Down Expand Up @@ -873,6 +974,23 @@ private static String executeRequestInternal(
return theString;
}

// TODO: do we need this?
public static void configureHttpClientTimeouts(HttpClientOptions httpClientOptions) {
// Set the connection timeout for a request to be sent.
httpClientOptions.setConnectTimeout(Duration.ofMillis(DEFAULT_HTTP_CLIENT_CONNECTION_TIMEOUT_IN_MS));
// Set the duration of time before an idle connection.
httpClientOptions.setConnectionIdleTimeout(Duration.ofSeconds(DEFAULT_IDLE_CONNECTION_TIMEOUT));
//// Set the maximum connection pool size used by the underlying HTTP client.
// httpClientOptions.setMaximumConnectionPoolSize(SystemUtil.convertSystemPropertyToIntValue(
// JDBC_MAX_CONNECTIONS_PROPERTY, DEFAULT_MAX_CONNECTIONS));
//// Sets the read timeout duration used when reading the server response.
// httpClientOptions.setReadTimeout(Duration readTimeout);
//// Sets the response timeout duration used when waiting for a server to reply.
// httpClientOptions.setResponseTimeout(Duration responseTimeout);
//// Sets the writing timeout for a request to be sent.
// httpClientOptions.setWriteTimeout(Duration writeTimeout);
}

// This is a workaround for JDK-7036144.
//
// The GZIPInputStream prematurely closes its input if a) it finds
Expand Down
10 changes: 5 additions & 5 deletions src/main/java/net/snowflake/client/jdbc/RestRequest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -368,19 +368,19 @@ public static CloseableHttpResponse execute(
break;
} else {
if (response != null) {
logger.debug(
logger.info(
"{}HTTP response not ok: status code: {}, request: {}",
requestIdStr,
response.getStatusLine().getStatusCode(),
requestInfoScrubbed);
} else if (savedEx != null) {
logger.debug(
logger.info(
"{}Null response for cause: {}, request: {}",
requestIdStr,
getRootCause(savedEx).getMessage(),
requestInfoScrubbed);
} else {
logger.debug("{}Null response for request: {}", requestIdStr, requestInfoScrubbed);
logger.info("{}Null response for request: {}", requestIdStr, requestInfoScrubbed);
}

// get the elapsed time for the last request
Expand Down Expand Up @@ -497,14 +497,14 @@ public static CloseableHttpResponse execute(
// sleep for backoff - elapsed amount of time
if (backoffInMilli > elapsedMilliForLastCall) {
try {
logger.debug(
logger.info(
"{}Retry request {}: sleeping for {} ms",
requestIdStr,
requestInfoScrubbed,
backoffInMilli);
Thread.sleep(backoffInMilli);
} catch (InterruptedException ex1) {
logger.debug("{}Backoff sleep before retrying login got interrupted", requestIdStr);
logger.info("{}Backoff sleep before retrying login got interrupted", requestIdStr);
}
elapsedMilliForTransientIssues += backoffInMilli;
backoffInMilli =
Expand Down
5 changes: 3 additions & 2 deletions src/main/java/net/snowflake/client/jdbc/SnowflakeFileTransferAgent.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1579,9 +1579,10 @@ public boolean execute() throws SQLException {
private void uploadStream() throws SnowflakeSQLException {
try {
FileMetadata fileMetadata = fileMetadataMap.get(SRC_FILE_NAME_FOR_STREAM);
logger.info("Start uploading stream {}", SRC_FILE_NAME_FOR_STREAM);

if (fileMetadata.resultStatus == ResultStatus.SKIPPED) {
logger.debug(
logger.info(
"Skipping {}, status: {}, details: {}",
SRC_FILE_NAME_FOR_STREAM,
fileMetadata.resultStatus,
Expand Down Expand Up @@ -2102,7 +2103,7 @@ private static void pushFileToRemoteStore(
remoteLocation.path + (!remoteLocation.path.endsWith("/") ? "/" : "") + destFileName;
}

logger.debug(
logger.info(
"Upload object. Location: {}, key: {}, srcFile: {}, encryption: {}",
remoteLocation.location,
destFileName,
Expand Down
Loading