Decouple HTTP retries and error mapping mechanics from DatabricksClient into httpclient.ApiClient

[nfx]

Changes

This PR decouples resiliency of HTTP client (rate limiting, retries, timeouts, debug logging) from client.DatabricksClient and moves it to a dedicated class, allowing to later use the same robust mechanism to refresh AAD tokens.

What's changed in httpclient.ApiClient compared from client.DatabricksClient (originated in Databricks Terraform Provider back in the early 2020):

• added native integration with golang.org/x/oauth2 package for both using token sources in the request as well as injecting a custom *http.Client the context for token refreshes.
• error mapper is now a pluggable function, default implementation available
• error retry checking is now pluggable function, default implementation checks for HTTP 429 and 504
• transient error messages are now configurable per client instance
• HTTP request visitors are now configurable per client instance
• Explicit and coupled Do(ctx context.Context, method, path string, headers map[string]string, request, response any, visitors ...func(*http.Request) error) error became concise and flexible func (c *ApiClient) Do(ctx context.Context, method, path string, opts ...DoOption) error. Headers can now be specified with httpclient.WithHeaders(headers), JSON deserialisation can now be specified with httpclient.WithUnmarshal(&entity).
• added httpclient.WithCaptureHeader(key, &value) to assign headers to a string pointer. This is required to get a proper integration with Azure Container Registry Managed Identity passwordless setup.
• body can be specified as httpclient.WithBody(any), that internally sets a special data field. Technically, this could have been a request visitor, but we have special logic for "debug bytes" logging, which is out of scope of this PR change.
• removed flawed trailing double-forward-slash trimming (//), as it was breaking requests containing full URLs.
• Once we merge the Allow injection of HTTP transport to enable HTTP replayer pattern #697, we can properly unit-test the implementations of Azure Client Secret and MSI flows

This is required for downstream tasks like:

• Better handling of AAD errors #240
• [FEATURE] Support for Azure AD Workload Identities #566
• Add ARM_USE_OIDC auth type #495
• proper solution to Retry token refresh for Azure SPN in case of Temporarily throttled, too many requests #321 instead of hack in d5e57f7

Tests

• Only one test was failing in TF provider, fixed in Fixed TestWorkspace_WaitForResolve_Failure by checking the unwrapped *net.DNSError terraform-provider-databricks#2957

[codecov-commenter]

Codecov Report

Attention: 60 lines in your changes are missing coverage. Please review.

Comparison is base (fe6b768) 15.77% compared to head (c3188f4) 16.33%.

Files	Patch %	Lines
httpclient/api_client.go	89.71%	12 Missing and 6 partials ⚠️
httpclient/response.go	73.80%	7 Missing and 4 partials ⚠️
httpclient/request.go	89.15%	7 Missing and 2 partials ⚠️
client/client.go	82.05%	4 Missing and 3 partials ⚠️
httpclient/errors.go	65.00%	5 Missing and 2 partials ⚠️
config/azure.go	0.00%	6 Missing ⚠️
config/auth_azure_client_secret.go	0.00%	1 Missing ⚠️
config/auth_azure_msi.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #699      +/-   ##
==========================================
+ Coverage   15.77%   16.33%   +0.55%     
==========================================
  Files          94       98       +4     
  Lines       13880    13995     +115     
==========================================
+ Hits         2190     2286      +96     
- Misses      11506    11518      +12     
- Partials      184      191       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[nkuln]

(drive-by) Just did a quick skim through. Although looks like most of the corner case handlings (that I have questions about) are moved from a different package that is time-tested in TF provider so that should be good.

[tanmay-db]

LGTM

[mgyucht]

Overall LGTM, thanks for doing this! Couple nits and suggestions.

[nfx]

testing TF provider with c3188f4:

 go get github.com/databricks/databricks-sdk-go@c3188f42c1913f1f1fc30d8bac82fc9944418809
go: upgraded github.com/databricks/databricks-sdk-go v0.25.0 => v0.25.1-0.20231124124904-c3188f42c191
go: upgraded google.golang.org/api v0.150.0 => v0.151.0
➜  terraform-provider-databricks git:(test/go-sdk-http-client) ✗ go mod tidy
go: downloading github.com/jhump/protoreflect v1.15.1
go: downloading github.com/bufbuild/protocompile v0.4.0
➜  terraform-provider-databricks git:(test/go-sdk-http-client) ✗ make test                                                                              
✓ Filling vendor folder with library code ...
✓ Linting source code with https://staticcheck.io/ ...

...

=== RUN   TestWorkspace_WaitForResolve_Failure
    resource_mws_workspaces_test.go:1035: 
                Error Trace:    /Users/serge.smertin/git/databricks/terraform-provider-databricks/mws/resource_mws_workspaces_test.go:1035
                                                        /Users/serge.smertin/git/databricks/terraform-provider-databricks/qa/testing.go:534
                                                        /Users/serge.smertin/git/databricks/terraform-provider-databricks/mws/resource_mws_workspaces_test.go:1029
                Error:          Expected value not to be nil.
                Test:           TestWorkspace_WaitForResolve_Failure

Only TestWorkspace_WaitForResolve_Failure was failing, which is fixed by databricks/terraform-provider-databricks#2957