Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request unauthorized for SQL search-attributes (claims are not passed) #6664

Open
TheGeniesis opened this issue Oct 16, 2024 · 3 comments
Open

Comments

@TheGeniesis
Copy link

TheGeniesis commented Oct 16, 2024

Expected Behavior

I can execute CRUD commands with admin and non admin x-search-attributes without errors.

Actual Behavior

I have devl:admin and temporal-system:admin roles assigned. I use only sql, postgres12 driver for "default" and "visibility" dbs.

When I execute one of x-search-attributes command inside admintools pod it fails with error:

I use devl namespace as an example of not working scenario

tctl --namespace=devl admin cluster get-search-attributes
Error: Unable to get search attributes.
Error Details: rpc error: code = PermissionDenied desc = Request unauthorized.

When I run this command with not existing namespace I get different error:

Error Details: rpc error: code = NotFound desc = Namespace searchtest2 is not found.

Cluster is working:

tctl cluster health
temporal.api.workflowservice.v1.WorkflowService: SERVING

I tested other admin commands (provided auth token as an env variable) related to cluster info (tctl --namespace=devl admin cluster describe, tctl --namespace=devl admin cluster list ), namespaces (tctl --namespace=devl namespace list, tctl --namespace=devl namespace describe), etc. and they are working. Only search-attributes has this problem. Non-admin command (tctl --namespace=devl cluster get-search-attributes) works as expected. Unfortunately only admin command allows to add attributes.
Update: The newest Temporal version allows to add new search attributes, admin commands are marked as deprecated.

Log from frontend pod:

 {"level":"error","ts":"2024-10-15T13:59:29.365Z","msg":"service failures","operation":"AdminGetSearchAttributes","wf-namespace":"devl","error":"Unable to get namespace devl info with error: Request unauthorized.","logging-call-at":"telemetry.go:411","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).handleError\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/telemetry.go:411\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).UnaryIntercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/telemetry.go:202\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo.temporal.io/server/service/frontend.(*RedirectionInterceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/frontend/redirection_interceptor.go:187\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo.temporal.io/server/common/authorization.(*Interceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/authorization/interceptor.go:181\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo.temporal.io/server/service/frontend.GrpcServerOptionsProvider.NewServerMetricsContextInjectorInterceptor.func2\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:315\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/namespace_logger.go:85\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).NamespaceValidateIntercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/namespace_validator.go:135\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo.temporal.io/server/common/utf8validator.(*Validator).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/utf8validator/validate.go:182\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo.temporal.io/server/service/frontend.GrpcServerOptionsProvider.NewServiceErrorInterceptor.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/grpc.go:178\ngoogle.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1177\ngo.temporal.io/server/api/adminservice/v1._AdminService_GetSearchAttributes_Handler\n\t/home/runner/work/docker-builds/docker-builds/temporal/api/adminservice/v1/service_grpc.pb.go:1062\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1369\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1780\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1019"}

Stacktrace from command:

Error: Unable to get search attributes.
Error Details: rpc error: code = Unavailable desc = Unable to get namespace devl info with error: Request unauthorized.
Stack trace:
goroutine 1 [running]:
runtime/debug.Stack()
	/opt/hostedtoolcache/go/1.21.11/x64/src/runtime/debug/stack.go:24 +0x5e
runtime/debug.PrintStack()
	/opt/hostedtoolcache/go/1.21.11/x64/src/runtime/debug/stack.go:16 +0x13
github.com/temporalio/tctl/cli_curr.printError({0x1990e48, 0x20}, {0x1dd3940, 0xc00051c000})
	/home/runner/work/docker-builds/docker-builds/tctl/cli_curr/util.go:393 +0x218
github.com/temporalio/tctl/cli_curr.ErrorAndExit({0x1990e48?, 0x1dfaf50?}, {0x1dd3940?, 0xc00051c000?})
	/home/runner/work/docker-builds/docker-builds/tctl/cli_curr/util.go:404 +0x25
github.com/temporalio/tctl/cli_curr.AdminGetSearchAttributes(0xc0005371e0)
	/home/runner/work/docker-builds/docker-builds/tctl/cli_curr/admin_cluster_search_attributes_commands.go:157 +0x89
github.com/temporalio/tctl/cli_curr.newAdminClusterCommands.func3(0xc0005371e0?)
	/home/runner/work/docker-builds/docker-builds/tctl/cli_curr/admin.go:496 +0x13
github.com/urfave/cli.HandleAction({0x16a4dc0?, 0x1a3db98?}, 0x15?)
	/home/runner/go/pkg/mod/github.com/urfave/[email protected]/app.go:526 +0x75
github.com/urfave/cli.Command.Run({{0x1978c96, 0x15}, {0x0, 0x0}, {0xc0006cb080, 0x1, 0x1}, {0x198e717, 0x1f}, {0x0, ...}, ...}, ...)
	/home/runner/go/pkg/mod/github.com/urfave/[email protected]/command.go:173 +0x63e
github.com/urfave/cli.(*App).RunAsSubcommand(0xc000239880, 0xc000536f20)
	/home/runner/go/pkg/mod/github.com/urfave/[email protected]/app.go:405 +0xe07
github.com/urfave/cli.Command.startApp({{0x195d3e0, 0x7}, {0x0, 0x0}, {0xc0006cb220, 0x1, 0x1}, {0x198c21d, 0x1e}, {0x0, ...}, ...}, ...)
	/home/runner/go/pkg/mod/github.com/urfave/[email protected]/command.go:378 +0xb58
github.com/urfave/cli.Command.Run({{0x195d3e0, 0x7}, {0x0, 0x0}, {0xc0006cb220, 0x1, 0x1}, {0x198c21d, 0x1e}, {0x0, ...}, ...}, ...)
	/home/runner/go/pkg/mod/github.com/urfave/[email protected]/command.go:102 +0x7e5
github.com/urfave/cli.(*App).RunAsSubcommand(0xc0002396c0, 0xc000536dc0)
	/home/runner/go/pkg/mod/github.com/urfave/[email protected]/app.go:405 +0xe07
github.com/urfave/cli.Command.startApp({{0x19598b4, 0x5}, {0x0, 0x0}, {0xc0006cb1d0, 0x1, 0x1}, {0x1974c0a, 0x13}, {0x0, ...}, ...}, ...)
	/home/runner/go/pkg/mod/github.com/urfave/[email protected]/command.go:378 +0xb58
github.com/urfave/cli.Command.Run({{0x19598b4, 0x5}, {0x0, 0x0}, {0xc0006cb1d0, 0x1, 0x1}, {0x1974c0a, 0x13}, {0x0, ...}, ...}, ...)
	/home/runner/go/pkg/mod/github.com/urfave/[email protected]/command.go:102 +0x7e5
github.com/urfave/cli.(*App).Run(0xc000239340, {0xc00003e0a0, 0x5, 0x5})
	/home/runner/go/pkg/mod/github.com/urfave/[email protected]/app.go:277 +0xb27
main.main()
	/home/runner/work/docker-builds/docker-builds/tctl/cmd/tctl/main.go:47 +0xa5

Steps to Reproduce the Problem

  1. Deploy temporal on AKS cluster using modified chart
  2. Create proper groups, users and service principals (I followed medium article with small tweaks, since it wasn't 100% up-to-date)
  3. Login into admintools pod
  4. Create a new namespace using UI session temporal operator namespace create --namespace <namespace> --grpc-meta=Authorization='Bearer <token_from_ui>' (works)
  5. Run any of admin x-search-attributes command tctl --namespace=devl cluster get-search-attributes --auth='Bearer <token_from_ui>' or temporal operator search-attribute create --name email --type Keyword --grpc-meta=Authorization='Bearer <token_from_ui>'

Specifications

  • Version: 1.24.2 - after upgrade to 1.25.1 problem still exists
  • Chart version: 0.44.0 (modified to enable internal frontend and oAuth) - after upgrade to 0.50.0 problem still exists

Question

Do I miss something related to role assignment? I searched frontend, admintools and worker logs, but I couldn't find anything which might help me to debug this problem.

Update: See my comments below. Looks like a bug. Claims are not passed for DescribeNamespace and UpdateNamespace endpoints which are executed by *SearchAttributesSQL functions. This is the source of the error.

@TheGeniesis TheGeniesis changed the title Request unauthorized for admin earch-attributes commands Request unauthorized for admin search-attributes commands Oct 16, 2024
@TheGeniesis
Copy link
Author

I did one more test - I enabled ES (changed flag elasticsearch.enabled: true in a chart and the feature works.

I checked Temporal documentation and I shouldn't have any problems with using search attributes without ES.

If you use Elasticsearch as your Visibility store, your custom Search Attributes apply globally and can be used across Namespaces. However, if using any of the supported SQL databases with Temporal Server v1.20 and later, your custom Search Attributes are associated with a specific Namespace and can be used for Workflow Executions in that Namespace.

I just wonder if this is the reason of the problem. For PSQL it should be namespace specific, but command was created for admin.

I also checked UI and https://<host>/api/v1/namespaces/devl/search-attributes? returns 503 in the UI when PSQL is enabled (no problem with ES).

@TheGeniesis
Copy link
Author

TheGeniesis commented Oct 21, 2024

For 503 error I added additional logs into the code and found that claims are not send for DescribeNamespace:

Changes in default_authorizer.go

func (a *defaultAuthorizer) Authorize(_ context.Context, claims *Claims, target *CallTarget) (Result, error) {
	// APIs that are essentially read-only health checks with no sensitive information are
	// always allowed
	if IsHealthCheckAPI(target.APIName) {
		return resultAllow, nil
	}
	fmt.Println("Check claims for ", target.APIName)
	dbg2(claims)
	if claims == nil {
		return resultDeny, nil
	}
	metadata := api.GetMethodMetadata(target.APIName)
	fmt.Println("API nAME", target.APIName)

Logs from frontend service:

Check claims for  /temporal.api.operatorservice.v1.OperatorService/ListSearchAttributes
{"Subject":"DmXTqIs843B3i6f-AnAcr2cakEpztzAmMrCuG-d8DXM","System":9,"Namespaces":{"devl":9,"intg":9,"uacc":9},"Extensions":null}
API nAME /temporal.api.operatorservice.v1.OperatorService/ListSearchAttributes
hasRole 9

Check claims for  /temporal.api.workflowservice.v1.WorkflowService/DescribeNamespace
null

In docker-builds/temporal/service/frontend/admin_handler.go I see function getSearchAttributesSQL which executes the DescribeNamespace function.

Is there any easy way to add claims to the function execution?

Edit:

To confirm I added a condition to bypass claims check for DescribeNamespace endpoint and UI started working.

if (strings.Contains(target.APIName, "DescribeNamespace")) {
  return resultAllow, nil
}

Edit 2:
After upgrade to the newest version I see commands to add search-attributes on operator level. I added the same bypass for UpdateNamespace and executed command - it works.

@TheGeniesis TheGeniesis changed the title Request unauthorized for admin search-attributes commands Request unauthorized for SQL search-attributes (claims are not passed) Oct 22, 2024
@bergundy
Copy link
Member

Thanks for the report. While these admin APIs are deprecated and so is tctl there may still be an actual issue here and we'll look into it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants