SNOW-974631: Socket timeout errors from 3.14.x version onwards #1565
Comments
github-actions
bot
changed the title
|
Can you pls help us with how to get past this issue with the 3.14.x version pls? We don't want our customer integrations to fail due to this upgrade. |
|
Hi @Syed-SnapLogic, Have you noticed this happening only with PUT statements? We're going to need debugging logs from your environment to help us understand the nature of the issue. Typically, when you see this type of error: That's an issue occurring at the network layer. In this error stack, it suggests that the client attempted to initiate a TLS connection by starting a handshake, and then the remote side forcefully closed the connection by sending a RST packet. I'm not sure why you would only see this in the 3.14.x versions in comparison to the 3.13.x versions, but that's something that would require a deeper dive in your environment if it's not anything we're able to reproduce. Can I ask you to open a support ticket and reference this GitHub issue, please? Can you be sure to enable JDBC debugging when reproducing this issue and attach the log when opening the ticket, please? |
|
@sfc-gh-wfateem The issue is occuring only when we enable the proxy, and i see there is a fix made related to some proxy in 3.14.0 https://github.com/snowflakedb/snowflake-jdbc/pull/1475/files this might be cause of regression? |
|
@muskaan62 the changes you're referring to is specifc to the connections made later on to the cloud provider's storage server when you get to the point where you need to download resultsets. For that to happen, your first connection in order to login has to work, is that happening? Based on the following in the stack, it would seem that this is occurring when the driver is trying to download data: I recall there was an issue where the proxy settings passed to the JDBC driver weren't take affect when connecting to the S3 service and that's because the S3 client was picking up environment variables. You can refer to the details of how the AWS S3 SDK library determines which proxy settings to use in PR #1475. How are you passing your proxy settings to the JDBC driver, and which parameters are you using? |
|
|
Right, so if you're able to login with those proxy settings but it's failing to download data then it sounds like the proxy settings are not taking effect for S3 connections. I did a quick test by using: That's a non-existing proxy server, so any connections going through the proxy server should fail. I bypassed only connections going to *.snowflakecomputing.com, so if the S3 connections were in fact picking up the proxy configurations then I would expect those to fail. However, all connections were working, which pretty much confirms that these proxy settings are ignored by the S3 client. As a workaround, just add the following JVM argument: |
|
@sfc-gh-wfateem I tried the workaround but its not working(below i s the stack trace with the workaround) |
|
@Syed-SnapLogic It might have just been a coincidence that this occurred after you tried the proposed workaround, but that's a different issue. This is a plain DNS lookup issue, and shouldn't be related to the JDBC driver upgrade: Are you consistently getting a java.net.UnknownHostException? If so, what is the output of the following command from that same machine hosting your JDBC driver: |
|
@Syed-SnapLogic have you been able to figure out that DNS issue you were facing? |
|
@sfc-gh-wfateem Give us some time,to respond to your query. |
sfc-gh-wfateem
added
enhancement
|
@muskaan62 just checking if you still needed help with this. |
|
@sfc-gh-wfateem That unknown host issue is not yet resolve from my end..i'll let you know by tomorrow if the provided workaround work for us or not. Thanks for your quick responses. |
|
@sfc-gh-wfateem https://github.com/snowflakedb/snowflake-jdbc/pull/1475/files
|
|
@Syed-SnapLogic I think the connection reset was actually caused by the fact that your S3 connections were not going through your proxy server in your environment, which resulted in that connection reset. We use the AWS S3 client library for data transfers to and from the S3 service. The issue that we discovered that needed to be addressed with that change you're referring to is that the S3 client (only) was picking up proxy settings from environment variables (HTTP_PROXY, HTTPS_PROXY, NO_PROXY). That's counter to what all the other components do, where none of them consider environment variables. That change addresses that particular issue, which might have not been a problem for you, however, with that change it became necessary to add the JVM argument You could consider it a "regression," but if you think about it, simply reverting that change would be yet another regression, but more severe than the one you're experiencing here. The issue you're facing is easily addressed using that JVM argument. The I've been able to reproduce your issue and confirm everything I'm explaining here, including the workaround I proposed. The workaround does not result in an If you're saying that you're consistently getting an You might have a DNS record for one of those but not the other two. In any other environment all hosts will resolve correctly, like in my case, which would be the only reason in my mind why I'm not seeing the Are you consistently getting an You can capture the TCP/IP packets to analyze the DNS queries sent by the client using the old and new JDBC driver releases to compare those and see which S3 host is the driver trying to establish a connection with. |
|
@sfc-gh-wfateem Thanks for your valuable time and detailed explanation, I really appreciate it. we are facing some challenges to reproduce this and seeing some different behaviour..If its possible for you I would like to connect and have a short call with you..please let me know if it is feasible for you to connect. |
|
@muskaan62 Sure thing. |
|
@sfc-gh-wfateem , we are not able to open a support case. Is it fine if we can proceed with a short call without any support ticket? Let me know few of your convenient time slots (along with your time zone) so I can pick the one that works best for both of us. |
|
@muskaan62 you can use this link to book a time for us. |
|
@sfc-gh-wfateem I have schedule a call on Friday(15 Dec) at 9 AM CST..Looking forward to see you there, Thanks. |
|
@Syed-SnapLogic I met with @muskaan62 and it's what I was suspecting might have been happening in your environment. There are two things that are happening here: 1.) In order to get the S3 connections to go through the proxy server, you have to add the JVM argument That addressed your customer's particular problem with the connection reset error, and in their environment they're able to resolve the S3 hostnames without any issues. 2.) In your environment, however, you're getting an Which basically means, there's some other change that happened between the 3.13.x and 3.14.x JDBC driver where the S3 host format used has changed and it's specifically a problem in your environment. That's something I'll need to look into a bit closer. |
|
I compared the hostname format for the S3 hosts when connecting using the JDBC driver version 3.13.34 and 3.14.3. There weren't any differences, so the issue seems to be specific to your environment. As I pointed out earlier, you simply need to add the I'll go ahead and close this off since you have confirmed in the support ticket with us that adding the JVM argument |
|
@sfc-gh-wfateem I have one query is this property "-Dhttp.useProxy=true" exclusive to snowflake?? |
|
@muskaan62 yes |
Please answer these questions before submitting your issue.
In order to accurately debug the issue this information is required. Thanks!
What version of JDBC driver are you using?
3.14.x
What operating system and processor architecture are you using?
Linux, 3.10.0-1160.102.1.el7.x86_64
What version of Java are you using?
11.0.12
What did you do?
Earlier, we were using 3.13.x version of the snowflake JDBC JAR. Things were all good. The moment we switch to 3.14.0 then we started seeing this failure:
com.snaplogic.api.ExecutionException: Error while waiting for PUT command to finish at com.snaplogic.snaps.snowflake.BulkLoad.execute(BulkLoad.java:549) at com.snaplogic.cc.snap.common.SnapRunnableImpl.executeSnap(SnapRunnableImpl.java:812) at com.snaplogic.cc.snap.common.SnapRunnableImpl.execute(SnapRunnableImpl.java:586) at com.snaplogic.cc.snap.common.SnapRunnableImpl.doRun(SnapRunnableImpl.java:877) at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:436) at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:120) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: net.snowflake.client.jdbc.SnowflakeSQLLoggedException: AWS operation failed: Operation=listObjects, Error message=Unable to execute HTTP request: Connection reset at java.base/java.util.concurrent.FutureTask.report(Unknown Source) at java.base/java.util.concurrent.FutureTask.get(Unknown Source) at com.snaplogic.snaps.snowflake.AbstractBulkSnap.waitForPutChunksCompletion(AbstractBulkSnap.java:978) at com.snaplogic.snaps.snowflake.BulkLoad.execute(BulkLoad.java:547) ... 11 more Caused by: net.snowflake.client.jdbc.SnowflakeSQLLoggedException: AWS operation failed: Operation=listObjects, Error message=Unable to execute HTTP request: Connection reset at net.snowflake.client.jdbc.cloud.storage.SnowflakeS3Client.handleS3Exception(SnowflakeS3Client.java:740) at net.snowflake.client.jdbc.cloud.storage.SnowflakeS3Client.handleStorageException(SnowflakeS3Client.java:676) at net.snowflake.client.jdbc.SnowflakeFileTransferAgent.filterExistingFiles(SnowflakeFileTransferAgent.java:2327) at net.snowflake.client.jdbc.SnowflakeFileTransferAgent.execute(SnowflakeFileTransferAgent.java:1326) at net.snowflake.client.core.SFStatement.executeFileTransfer(SFStatement.java:783) at net.snowflake.client.core.SFStatement.executeQuery(SFStatement.java:129) at net.snowflake.client.core.SFStatement.execute(SFStatement.java:769) at net.snowflake.client.core.SFStatement.execute(SFStatement.java:677) at net.snowflake.client.jdbc.SnowflakeStatementV1.executeInternal(SnowflakeStatementV1.java:320) at net.snowflake.client.jdbc.SnowflakeStatementV1.execute(SnowflakeStatementV1.java:392) at com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:94) at com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java) at com.snaplogic.snaps.snowflake.AbstractBulkSnap.putChunks(AbstractBulkSnap.java:944) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at com.snaplogic.cc.snap.common.SnapHolder.lambda$ensureThreadGroup$3(SnapHolder.java:428) ... 1 more Caused by: net.snowflake.client.jdbc.internal.amazonaws.SdkClientException: Unable to execute HTTP request: Connection reset at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219) at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1165) at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5397) at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:928) at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:902) at net.snowflake.client.jdbc.cloud.storage.SnowflakeS3Client.listObjects(SnowflakeS3Client.java:274) at net.snowflake.client.jdbc.SnowflakeFileTransferAgent.filterExistingFiles(SnowflakeFileTransferAgent.java:2313) ... 15 more Caused by: java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at net.snowflake.client.jdbc.internal.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436) at net.snowflake.client.jdbc.internal.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384) at net.snowflake.client.jdbc.internal.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) at net.snowflake.client.jdbc.internal.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at jdk.internal.reflect.GeneratedMethodAccessor103.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at net.snowflake.client.jdbc.internal.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) at net.snowflake.client.jdbc.internal.amazonaws.http.conn.$Proxy163.connect(Unknown Source) at net.snowflake.client.jdbc.internal.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at net.snowflake.client.jdbc.internal.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at net.snowflake.client.jdbc.internal.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at net.snowflake.client.jdbc.internal.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at net.snowflake.client.jdbc.internal.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at net.snowflake.client.jdbc.internal.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at net.snowflake.client.jdbc.internal.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1346) at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) ... 29 moreWhat did you expect to see?
3.14.0 version should also work like earlier versions.
Can you set logging to DEBUG and collect the logs?
Not now as it happens in customer's environment.
What is your Snowflake account identifier, if any? (Optional)
Snaplogic
Note: We tried using all the released 3.14.x versions as of today so far but still the same problem keeps occuring. We even tried setting maxHttpRetries to zero in the URL properties of the jdbc string but still the same issue persists.
The text was updated successfully, but these errors were encountered: