Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Local network communications #276

Open
roquie opened this issue Dec 15, 2023 · 6 comments
Open

[Bug] Local network communications #276

roquie opened this issue Dec 15, 2023 · 6 comments
Assignees
@eliobischof
Labels
devops

Comments

@roquie
Copy link

roquie commented Dec 15, 2023

I want to do a simple thing. Close my development environment behind Cloudflare so no one can access, including the Zitadel test instance. But here's the trouble, by closing IP access to WAF I get 403 errors in the logs of my service which is integrated with Zitadel via this SDK.

In order to be able to use Zitadel within the local network (Kubernetes), I wrote a local GRPC endpoint address for the Zitadel service and at the Istio level, in sidecar, made a host replacement (to pass the security check).

This allowed the zitadel:80/.well-known/openid-configuration endpoint to converge successfully, similarly configured h2c app protocol in Kubernetes Service. Everything is done, for successful local operation.

My settings for the go-client:
issuer: https://sso.example.com
grpc_endpoint: zitadel:80
option: WithInsecure()

Log when trying to get a user profile:

  {"level":"error","timestamp":"2023-12-15T15:10:54Z","component":"usecase.getUserProfile","msg":"STOP. getZitadelUserProfile error","userID":"123131313312313123","error":"client.GetHumanProfile error: http status not ok: 403 Forbidden <!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 8]>    <html class=\"no-js ie8 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if gt IE 8]><!--> <html class=\"no-js\" lang=\"en-US\"> <!--<![endif]-->\n<head>\n<title>Attention Required! | Cloudflare</title>\n<meta charset=\"UTF-8\" />\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" />\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\" />\n<meta name=\"robots\" content=\"noindex, nofollow\" />\n<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\" />\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/cf.errors.css\" />\n<!--[if lt IE 9]><link rel=\"stylesheet\" id='cf_styles-ie-css' href=\"/cdn-cgi/styles/cf.errors.ie.css\" /><![endif]-->\n<style>body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!-->\n<script>\n  if (!navigator.cookieEnabled) {\n    window.addEventListener('DOMContentLoaded', function () {\n      var cookieEl = document.getElementById('cookie-alert');\n      cookieEl.style.display = 'block';\n    })\n  }\n</script>\n<!--<![endif]-->\n\n\n</head>\n<body>\n  <div id=\"cf-wrapper\">\n    <div class=\"cf-alert cf-alert-error cf-cookie-error\" id=\"cookie-alert\" data-translate=\"enable_cookies\">Please enable cookies.</div>\n    <div id=\"cf-error-details\" class=\"cf-error-details-wrapper\">\n      <div class=\"cf-wrapper cf-header cf-error-overview\">\n        <h1 data-translate=\"block_headline\">Sorry, you have been blocked</h1>\n        <h2 class=\"cf-subheadline\"><span data-translate=\"unable_to_access\">You are unable to access</span> example.com</h2>\n      </div><!-- /.header -->\n\n      <div class=\"cf-section cf-highlight\">\n        <div class=\"cf-wrapper\">\n          <div class=\"cf-screenshot-container cf-screenshot-full\">\n            \n              <span class=\"cf-no-screenshot error\"></span>\n            \n          </div>\n        </div>\n      </div><!-- /.captcha-container -->\n\n      <div class=\"cf-section cf-wrapper\">\n        <div class=\"cf-columns two\">\n          <div class=\"cf-column\">\n            <h2 data-translate=\"blocked_why_headline\">Why have I been blocked?</h2>\n\n            <p data-translate=\"blocked_why_detail\">This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.</p>\n          </div>\n\n          <div class=\"cf-column\">\n            <h2 data-translate=\"blocked_resolve_headline\">What can I do to resolve this?</h2>\n\n            <p data-translate=\"blocked_resolve_detail\">You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.</p>\n          </div>\n        </div>\n      </div><!-- /.section -->\n\n      <div class=\"cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300\">\n  <p class=\"text-13\">\n    <span class=\"cf-footer-item sm:block sm:mb-1\">Cloudflare Ray ID: <strong class=\"font-semibold\">835f9db30d35250c</strong></span>\n    <span class=\"cf-footer-separator sm:hidden\">&bull;</span>\n    <span id=\"cf-footer-item-ip\" class=\"cf-footer-item hidden sm:block sm:mb-1\">\n      Your IP:\n      <button type=\"button\" id=\"cf-footer-ip-reveal\" class=\"cf-footer-ip-reveal-btn\">Click to reveal</button>\n      <span class=\"hidden\" id=\"cf-footer-ip\">209.38.252.7</span>\n      <span class=\"cf-footer-separator sm:hidden\">&bull;</span>\n    </span>\n    <span class=\"cf-footer-item sm:block sm:mb-1\"><span>Performance &amp; security by</span> <a rel=\"noopener noreferrer\" href=\"https://www.cloudflare.com/5xx-error-landing\" id=\"brand_link\" target=\"_blank\">Cloudflare</a></span>\n    \n  </p>\n  <script>(function(){function d(){var b=a.getElementById(\"cf-footer-item-ip\"),c=a.getElementById(\"cf-footer-ip-reveal\");b&&\"classList\"in b&&(b.classList.remove(\"hidden\"),c.addEventListener(\"click\",function(){c.classList.add(\"hidden\");a.getElementById(\"cf-footer-ip\").classList.remove(\"hidden\")}))}var a=document;document.addEventListener&&a.addEventListener(\"DOMContentLoaded\",d)})();</script>\n</div><!-- /.error-footer -->\n\n\n    </div><!-- /#cf-error-details -->\n  </div><!-- /#cf-wrapper -->\n\n  <script>\n  window._cf_translation = {};\n  \n  \n</script>\n\n</body>\n</html>\n","stacktrace":"gitlab.example2.com/common/go-libs/pkg/log.(*zapLogger).Error\n\t/build/vendor/gitlab.example2.com/common/go-libs/pkg/log/zap_logger.go:202\ngitlab.example2.com/pp/backend/internal/application/usecases/users/getUserProfile.(*UseCase).Run\n\t/build/internal/application/usecases/users/getUserProfile/usecase.go:76\ngitlab.example2.com/pp/backend/internal/application/gateways/openapi.(*Server).GetProfile\n\t/build/internal/application/gateways/openapi/getProfile.go:39\ngitlab.example2.com/pp/backend/internal/pkg/server/openapi/application.(*ServerInterfaceWrapper).GetProfile\n\t/build/internal/pkg/server/openapi/application/server.gen.go:666\ngithub.com/labstack/echo/v4.(*Echo).add.func1\n\t/build/vendor/github.com/labstack/echo/v4/echo.go:582\ngitlab.example2.com/common/go-libs/pkg/http/middlewares/recovery.EchoRecovery.func1.1\n\t/build/vendor/gitlab.example2.com/common/go-libs/pkg/http/middlewares/recovery/recovery.go:84\ngitlab.example2.com/common/go-libs/pkg/http/middlewares.Logging.EchoLogger.func2.1\n\t/build/vendor/gitlab.example2.com/common/go-libs/pkg/http/middlewares/logging.go:73\ngitlab.example2.com/common/go-libs/pkg/http/middlewares/metrics.Handler.EchoMetrics.func2.1.1\n\t/build/vendor/gitlab.example2.com/common/go-libs/pkg/http/middlewares/metrics/metrics.go:54\ngithub.com/slok/go-http-metrics/middleware.Middleware.Measure\n\t/build/vendor/github.com/slok/go-http-metrics/middleware/middleware.go:117\ngitlab.example2.com/common/go-libs/pkg/http/middlewares/metrics.Handler.EchoMetrics.func2.1\n\t/build/vendor/gitlab.example2.com/common/go-libs/pkg/http/middlewares/metrics/metrics.go:53\ngitlab.example2.com/common/go-libs/pkg/http/middlewares.ApiGatewayAuthMiddleware.func1.1\n\t/build/vendor/gitlab.example2.com/common/go-libs/pkg/http/middlewares/auth.go:137\ngithub.com/labstack/echo/v4.(*Echo).ServeHTTP\n\t/build/vendor/github.com/labstack/echo/v4/echo.go:669\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}

Now the question is why it ignores the local grpc endpoint and uses issuer to perform the request?
@fforootd
Copy link
Member

I am not fully sure I can follow 😁 so let me try and ask some questions.

  • External access to zitadel works, right? (but you are fencing of with CF)
  • Accessing zitadel directly through the internal k8s server does return an error (should be instance not found)

@roquie
Copy link
Author

roquie commented Dec 20, 2023

External access to zitadel works, right? (but you are fencing with CF)

It works, but we want to close it to the test environment behind an IP whitelist using WAF.

Accessing zitadel directly through the internal k8s server does return an error (should be instance not found)

It used to return an error, but now it doesn't, because I'm substituting the Host header at the sidecar (istio) level. By doing so, I just bypassed the zitadel check to try to use the service entirely inside the Kubernetes network.

The problem is that by closing the zitadel service behind CF, my service's integration with the zitadel service breaks. And this is despite the fact that zitadel's GPRC address is listed as local (service name in kubernetes)!

@HaimKortovich
Copy link
Contributor

Hi @roquie, did you find a workaround? I'm having the same issue

@roquie
Copy link
Author

roquie commented Mar 27, 2024

No. @fforootd any updates?

@HaimKortovich
Copy link
Contributor

ok so I managed to do it by creating my own (and probably not very good solution)

func Discover(key []byte, discoverUrl string) func(issuer string, scopes []string) (oauth2.TokenSource, error) {
	return func(issuer string, scopes []string) (oauth2.TokenSource, error) {
		var machineKeyData MachineKey
		if err := json.Unmarshal(key, &machineKeyData); err != nil {
			return nil, err
		}
		signer, err := client.NewSignerFromPrivateKeyByte([]byte(machineKeyData.Key), machineKeyData.KeyID)
		if err != nil {
			return nil, err
		}
		source := &jwtProfileTokenSource{
			clientID:   machineKeyData.UserID,
			audience:   []string{issuer},
			signer:     signer,
			scopes:     scopes,
			httpClient: http.DefaultClient,
		}
		config, err := client.Discover(discoverUrl, http.DefaultClient)
		if err != nil {
			return nil, err
		}
		source.tokenEndpoint = config.TokenEndpoint
		return source, nil
	}
}

func GetDiscoveryConfig(issuer string, httpClient *http.Client, wellKnownUrl ...string) (*oidc.DiscoveryConfiguration, error) {
	wellKnown := strings.TrimSuffix(issuer, "/") + oidc.DiscoveryEndpoint
	if len(wellKnownUrl) == 1 && wellKnownUrl[0] != "" {
		wellKnown = wellKnownUrl[0]
	}
	req, err := http.NewRequest("GET", wellKnown, nil)
	if err != nil {
		return nil, err
	}
	discoveryConfig := new(oidc.DiscoveryConfiguration)
	err = httphelper.HttpRequest(httpClient, req, &discoveryConfig)
	if err != nil {
		return nil, err
	}
	return discoveryConfig, nil
}

and passing it to the client

		zitadel.WithJWTProfileTokenSource(Discover([]byte(machineKeyData), GetAPIUrl(zitadelCluster))),

@hifabienne hifabienne moved this to 🧐 Investigating in Product Management Mar 28, 2024
@HaimKortovich
Copy link
Contributor

@fforootd

@adlerhurst adlerhurst added devops and removed devx labels Apr 25, 2024
@eliobischof eliobischof self-assigned this Nov 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@eliobischof eliobischof

Labels
devops
Projects
Product Management
Status: 🧐 Investigating
Milestone
No milestone
Development

No branches or pull requests
6 participants
@roquie @fforootd @eliobischof @adlerhurst @hifabienne @HaimKortovich