Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SNOW-977565: Added start and end symbol to match full regex to bypass proxy server. #899

Merged
merged 10 commits into from
Apr 4, 2024
18 changes: 17 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ The following table lists all valid connection properties:
| PROXYPORT | Depends | The port number of the proxy server. <br/> <br/> If USEPROXY is set to `true`, you must set this parameter. <br/> <br/> This parameter was introduced in v2.0.4. |
| PROXYUSER | No | The username for authenticating to the proxy server. <br/> <br/> This parameter was introduced in v2.0.4. |
| PROXYPASSWORD | Depends | The password for authenticating to the proxy server. <br/> <br/> If USEPROXY is `true` and PROXYUSER is set, you must set this parameter. <br/> <br/> This parameter was introduced in v2.0.4. |
| NONPROXYHOSTS | No | The list of hosts that the driver should connect to directly, bypassing the proxy server. Separate the hostnames with a pipe symbol (\|). You can also use an asterisk (`*`) as a wildcard. <br/> <br/> This parameter was introduced in v2.0.4. |
| NONPROXYHOSTS | No | The list of hosts that the driver should connect to directly, bypassing the proxy server. Separate the hostnames with a pipe symbol (\|). You can also use an asterisk (`*`) as a wildcard. <br/> The host target value should fully match with any item from the proxy host list to bypass the proxy server. <br/> <br/> This parameter was introduced in v2.0.4. |
| FILE_TRANSFER_MEMORY_THRESHOLD | No | The maximum number of bytes to store in memory used in order to provide a file encryption. If encrypting/decrypting file size exceeds provided value a temporary file will be created and the work will be continued in the temporary file instead of memory. <br/> If no value provided 1MB will be used as a default value (that is 1048576 bytes). <br/> It is possible to configure any integer value bigger than zero representing maximal number of bytes to reside in memory. |
| CLIENT_CONFIG_FILE | No | The location of the client configuration json file. In this file you can configure easy logging feature. |
| ALLOWUNDERSCORESINHOST | No | Specifies whether to allow underscores in account names. This impacts PrivateLink customers whose account names contain underscores. In this situation, you must override the default value by setting allowUnderscoresInHost to true. |
Expand Down Expand Up @@ -377,6 +377,22 @@ using (IDbConnection conn = new SnowflakeDbConnection())
}
```

The NONPROXYHOSTS property could be set to specify if the server proxy should be bypassed by an specified host. This should be defined using the full host url or including the url + `*` wilcard symbol.

Examples:

- `*` (Bypassed all hosts from the proxy server)
- `*.snowflakecomputing.com` ('Bypass all host that ends with `snowflakecomputing.com`')
- `https:\\testaccount.snowflakecomputing.com` (Bypass proxy server using full host url).
- `*.myserver.com | *testaccount*` (You can specify multiple regex for the property divided by `|`)


> Note: The nonproxyhost value should match the full url including the http or https section. The '*' wilcard could be added to bypass the hostname successfully.

- `myaccount.snowflakecomputing.com` (Not bypassed).
- `*myaccount.snowflakecomputing.com` (Bypassed).


## Using Connection Pools

Instead of creating a connection each time your client application needs to access Snowflake, you can define a cache of Snowflake connections that can be reused as needed. Connection pooling usually reduces the lag time to make a connection. However, it can slow down client failover to an alternative DNS when a DNS problem occurs.
Expand Down
57 changes: 56 additions & 1 deletion Snowflake.Data.Tests/IntegrationTests/SFConnectionIT.cs
Original file line number Diff line number Diff line change
Expand Up @@ -1540,6 +1540,60 @@ public void TestInvalidProxySettingFromConnectionString()
}
}

[Test]
[TestCase("*")]
[TestCase("*{0}*")]
[TestCase("^*{0}*")]
[TestCase("*{0}*$")]
[TestCase("^*{0}*$")]
sfc-gh-mhofman marked this conversation as resolved.
Show resolved Hide resolved
[TestCase("^nonmatch*{0}$|*")]
[TestCase("*a*", "a")]
[TestCase("*la*", "la")]
public void TestNonProxyHostShouldBypassProxyServer(string regexHost, string proxyHost = "proxyserverhost")
{
using (var conn = new SnowflakeDbConnection())
{
// Arrange
var host = ResolveHost();
var nonProxyHosts = string.Format(regexHost, $"{host}");
conn.ConnectionString =
$"{ConnectionString}USEPROXY=true;PROXYHOST={proxyHost};NONPROXYHOSTS={nonProxyHosts};PROXYPORT=3128;";

// Act
conn.Open();
sfc-gh-knozderko marked this conversation as resolved.
Show resolved Hide resolved

// Assert
// The connection would fail to open if the web proxy would be used because the proxy is configured to a non-existent host.
Assert.AreEqual(ConnectionState.Open, conn.State);
sfc-gh-knozderko marked this conversation as resolved.
Show resolved Hide resolved
}
}

[Test]
[TestCase("invalid{0}")]
[TestCase("*invalid{0}*")]
[TestCase("^invalid{0}$")]
[TestCase("*a.b")]
[TestCase("a", "a")]
[TestCase("la", "la")]
public void TestNonProxyHostShouldNotBypassProxyServer(string regexHost, string proxyHost = "proxyserverhost")
{
using (var conn = new SnowflakeDbConnection())
{
// Arrange
var nonProxyHosts = string.Format(regexHost, $"{testConfig.host}");
conn.ConnectionString =
$"{ConnectionString}connection_timeout=5;USEPROXY=true;PROXYHOST={proxyHost};NONPROXYHOSTS={nonProxyHosts};PROXYPORT=3128;";

// Act/Assert
// The connection would fail to open if the web proxy would be used because the proxy is configured to a non-existent host.
var exception = Assert.Throws<SnowflakeDbException>(() => conn.Open());
sfc-gh-knozderko marked this conversation as resolved.
Show resolved Hide resolved
sfc-gh-knozderko marked this conversation as resolved.
Show resolved Hide resolved

// Assert
Assert.AreEqual(270001, exception.ErrorCode);
AssertIsConnectionFailure(exception);
}
}

[Test]
public void TestUseProxyFalseWithInvalidProxyConnectionString()
{
Expand All @@ -1561,7 +1615,7 @@ public void TestInvalidProxySettingWithByPassListFromConnectionString()
= ConnectionString
+ String.Format(
";useProxy=true;proxyHost=Invalid;proxyPort=8080;nonProxyHosts={0}",
"*.foo.com %7C" + testConfig.account + ".snowflakecomputing.com|" + testConfig.host);
$"*.foo.com %7C{testConfig.account}.snowflakecomputing.com|*{testConfig.host}*");
conn.Open();
// Because testConfig.host is in the bypass list, the proxy should not be used
}
Expand Down Expand Up @@ -2169,6 +2223,7 @@ public void TestNativeOktaSuccess()
Assert.AreEqual(ConnectionState.Open, conn.State);
}
}

}
}

Expand Down
5 changes: 5 additions & 0 deletions Snowflake.Data.Tests/SFBaseTest.cs
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,11 @@ public SFBaseTestAsync()
testConfig.password);

protected TestConfig testConfig { get; }

protected string ResolveHost()
{
return testConfig.host ?? $"{testConfig.account}.snowflakecomputing.com";
}
}

[SetUpFixture]
Expand Down
7 changes: 5 additions & 2 deletions Snowflake.Data/Core/HttpUtil.cs
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
using System.Collections.Specialized;
using System.Web;
using System.Security.Authentication;
using System.Runtime.InteropServices;
using System.Linq;
using Snowflake.Data.Core.Authenticator;

Expand Down Expand Up @@ -186,7 +185,11 @@
entry = entry.Replace(".", "[.]");
// * -> .* because * is a quantifier and need a char or group to apply to
entry = entry.Replace("*", ".*");


entry = entry.StartsWith("^") ? entry : $"^{entry}";
sfc-gh-knozderko marked this conversation as resolved.
Show resolved Hide resolved

entry = entry.EndsWith("$") ? entry : $"{entry}$";

// Replace with the valid entry syntax
bypassList[i] = entry;

Expand Down Expand Up @@ -349,13 +352,13 @@
int backOffInSec = s_baseBackOffTime;
int totalRetryTime = 0;

ServicePoint p = ServicePointManager.FindServicePoint(requestMessage.RequestUri);

Check warning on line 355 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on Linux (net6.0, AWS)

'ServicePointManager.FindServicePoint(Uri)' is obsolete: 'WebRequest, HttpWebRequest, ServicePoint, and WebClient are obsolete. Use HttpClient instead.' (https://aka.ms/dotnet-warnings/SYSLIB0014)

Check warning on line 355 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on Linux (net6.0, GCP)

'ServicePointManager.FindServicePoint(Uri)' is obsolete: 'WebRequest, HttpWebRequest, ServicePoint, and WebClient are obsolete. Use HttpClient instead.' (https://aka.ms/dotnet-warnings/SYSLIB0014)

Check warning on line 355 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on MAC (net6.0, AWS)

'ServicePointManager.FindServicePoint(Uri)' is obsolete: 'WebRequest, HttpWebRequest, ServicePoint, and WebClient are obsolete. Use HttpClient instead.' (https://aka.ms/dotnet-warnings/SYSLIB0014)

Check warning on line 355 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on Linux (net6.0, AZURE)

'ServicePointManager.FindServicePoint(Uri)' is obsolete: 'WebRequest, HttpWebRequest, ServicePoint, and WebClient are obsolete. Use HttpClient instead.' (https://aka.ms/dotnet-warnings/SYSLIB0014)

Check warning on line 355 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on MAC (net6.0, AZURE)

'ServicePointManager.FindServicePoint(Uri)' is obsolete: 'WebRequest, HttpWebRequest, ServicePoint, and WebClient are obsolete. Use HttpClient instead.' (https://aka.ms/dotnet-warnings/SYSLIB0014)

Check warning on line 355 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on Windows (net6.0, AZURE)

'ServicePointManager.FindServicePoint(Uri)' is obsolete: 'WebRequest, HttpWebRequest, ServicePoint, and WebClient are obsolete. Use HttpClient instead.' (https://aka.ms/dotnet-warnings/SYSLIB0014)

Check warning on line 355 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on MAC (net6.0, GCP)

'ServicePointManager.FindServicePoint(Uri)' is obsolete: 'WebRequest, HttpWebRequest, ServicePoint, and WebClient are obsolete. Use HttpClient instead.' (https://aka.ms/dotnet-warnings/SYSLIB0014)

Check warning on line 355 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on Windows (net6.0, GCP)

'ServicePointManager.FindServicePoint(Uri)' is obsolete: 'WebRequest, HttpWebRequest, ServicePoint, and WebClient are obsolete. Use HttpClient instead.' (https://aka.ms/dotnet-warnings/SYSLIB0014)

Check warning on line 355 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on Windows (net6.0, AWS)

'ServicePointManager.FindServicePoint(Uri)' is obsolete: 'WebRequest, HttpWebRequest, ServicePoint, and WebClient are obsolete. Use HttpClient instead.' (https://aka.ms/dotnet-warnings/SYSLIB0014)
p.Expect100Continue = false; // Saves about 100 ms per request
p.UseNagleAlgorithm = false; // Saves about 200 ms per request
p.ConnectionLimit = 20; // Default value is 2, we need more connections for performing multiple parallel queries

TimeSpan httpTimeout = (TimeSpan)requestMessage.Properties[SFRestRequest.HTTP_REQUEST_TIMEOUT_KEY];

Check warning on line 360 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on Linux (net6.0, AWS)

'HttpRequestMessage.Properties' is obsolete: 'HttpRequestMessage.Properties has been deprecated. Use Options instead.'

Check warning on line 360 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on Linux (net6.0, GCP)

'HttpRequestMessage.Properties' is obsolete: 'HttpRequestMessage.Properties has been deprecated. Use Options instead.'

Check warning on line 360 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on MAC (net6.0, AWS)

'HttpRequestMessage.Properties' is obsolete: 'HttpRequestMessage.Properties has been deprecated. Use Options instead.'

Check warning on line 360 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on Linux (net6.0, AZURE)

'HttpRequestMessage.Properties' is obsolete: 'HttpRequestMessage.Properties has been deprecated. Use Options instead.'

Check warning on line 360 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on MAC (net6.0, AZURE)

'HttpRequestMessage.Properties' is obsolete: 'HttpRequestMessage.Properties has been deprecated. Use Options instead.'

Check warning on line 360 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on Windows (net6.0, AZURE)

'HttpRequestMessage.Properties' is obsolete: 'HttpRequestMessage.Properties has been deprecated. Use Options instead.'

Check warning on line 360 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on MAC (net6.0, GCP)

'HttpRequestMessage.Properties' is obsolete: 'HttpRequestMessage.Properties has been deprecated. Use Options instead.'

Check warning on line 360 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on Windows (net6.0, GCP)

'HttpRequestMessage.Properties' is obsolete: 'HttpRequestMessage.Properties has been deprecated. Use Options instead.'

Check warning on line 360 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on Windows (net6.0, AWS)

'HttpRequestMessage.Properties' is obsolete: 'HttpRequestMessage.Properties has been deprecated. Use Options instead.'
TimeSpan restTimeout = (TimeSpan)requestMessage.Properties[SFRestRequest.REST_REQUEST_TIMEOUT_KEY];

Check warning on line 361 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on Linux (net6.0, AWS)

'HttpRequestMessage.Properties' is obsolete: 'HttpRequestMessage.Properties has been deprecated. Use Options instead.'

Check warning on line 361 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on Linux (net6.0, GCP)

'HttpRequestMessage.Properties' is obsolete: 'HttpRequestMessage.Properties has been deprecated. Use Options instead.'

Check warning on line 361 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on MAC (net6.0, AWS)

'HttpRequestMessage.Properties' is obsolete: 'HttpRequestMessage.Properties has been deprecated. Use Options instead.'

Check warning on line 361 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on Linux (net6.0, AZURE)

'HttpRequestMessage.Properties' is obsolete: 'HttpRequestMessage.Properties has been deprecated. Use Options instead.'

Check warning on line 361 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on MAC (net6.0, AZURE)

'HttpRequestMessage.Properties' is obsolete: 'HttpRequestMessage.Properties has been deprecated. Use Options instead.'

Check warning on line 361 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on Windows (net6.0, AZURE)

'HttpRequestMessage.Properties' is obsolete: 'HttpRequestMessage.Properties has been deprecated. Use Options instead.'

Check warning on line 361 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on MAC (net6.0, GCP)

'HttpRequestMessage.Properties' is obsolete: 'HttpRequestMessage.Properties has been deprecated. Use Options instead.'

Check warning on line 361 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on Windows (net6.0, GCP)

'HttpRequestMessage.Properties' is obsolete: 'HttpRequestMessage.Properties has been deprecated. Use Options instead.'

Check warning on line 361 in Snowflake.Data/Core/HttpUtil.cs

View workflow job for this annotation

GitHub Actions / Tests on Windows (net6.0, AWS)

'HttpRequestMessage.Properties' is obsolete: 'HttpRequestMessage.Properties has been deprecated. Use Options instead.'

if (logger.IsDebugEnabled())
{
Expand Down
Loading