This document details how to setup a local reference architecture, and design and deploy an API. This will show the following API management features in a kube native environment using Kuadrant and other open source tools:
-
-
API design
-
API security and access control
-
API monitoring
-
Traffic management and scalability
-
-
The sections in this document are grouped by the persona that is typically associated with the steps in that section. The 3 personas are:
-
-
The platform engineer, who provides and maintains a platform for application developers,
-
the application developer, who designs, builds and maintains applications and APIs,
-
and the api consumer, who makes API calls to the API
This will take several minutes as 3 local kind clusters are started and configured in a hub and spoke architecture.
-The following components will be installed on the clusters:
View the ManagedZone, Gateway and TLSPolicy. The ManagedZone and TLSPolicy should have a Ready status of true. The Gateway should have a Programmed status of True.
To get a top level view of the constraints in violation, the Stitch: Platform Engineer Dashboard can be used. This can be accessed by at https://grafana.172.31.0.2.nip.io
-
Grafana has a default username and password of admin.
-You can find the Stitch: Platform Engineer Dashboard dashboard in the Default folder.
Since we have created all the policies that Gatekeeper had the guardrails around, you should no longer see any constraints in violation. This can be seen back in the Stitch: Platform Engineer Dashboard in Grafana at https://grafana.172.31.0.2.nip.io
Fork and/or clone the Petstore App at https://github.com/Kuadrant/api-petstore
-
gitclonegit@github.com:kuadrant/api-petstore&&cdapi-petstore
-# Or if you forked the repository:
-# git clone git@github.com:<your_github_username>/api-petstore && cd api-petstore
-
When the DNS Policy has been created, and the previously created HTTPRoute has been attached, a DNS record custom resource will also be created in the cluster resulting in records being created in your AWS Route53. Navigate to Route53 and you should see some new records in the zone.
In /user/login, we have a Gateway API backendRef set and a rate_limit set. The rate limit policy for this endpoint restricts usage of this endpoint to 2 requests in a 10 second window:
-
In /store/inventory, we have also have a Gateway API backendRefset and a rate_limit set. The rate limit policy for the endpoint restricts usage of this endpoint to 10 requests in a 10 second window:
-
kuadrantctl is a cli that supports the generation of various Kubernetes resources via OAS specs.
-Let's run some commands to generate some of these resources.
-If you forked the api-pestore repo, you can check them in also.
-Let's apply these to our running workload to implement rate limiting and auth.
We'll generate an AuthPolicy to implement API key auth, per the securityScheme in our OAS spec:
-
# Generate this resource and save:
-kuadrantctlgeneratekuadrantauthpolicy--oasopenapi.yaml|yq-P|teeresources/authpolicy.yaml
-
-# Apply this resource to our cluster:
-kubectl--contextkind-api-workload-1apply-f./resources/authpolicy.yaml
-
-
Next we'll generate a RateLimitPolicy, to protect our APIs with the limits we have setup in our OAS spec:
-
# Generate this resource and save:
-kuadrantctlgeneratekuadrantratelimitpolicy--oasopenapi.yaml|yq-P|teeresources/ratelimitpolicy.yaml
-
-# Apply this resource to our cluster:
-kubectl--contextkind-api-workload-1apply-f./resources/ratelimitpolicy.yaml
-
-
Lastly, we'll generate a Gateway API HTTPRoute to service our APIs:
-
# Generate this resource and save:
-kuadrantctlgenerategatewayapihttproute--oasopenapi.yaml|yq-P|teeresources/httproute.yaml
-
-# Apply this resource to our cluster, setting the hostname in via the KUADRANT_ZONE_ROOT_DOMAIN env var:
-kustomizebuild./resources/|envsubst|kubectl--contextkind-api-workload-1apply-f-
-
Let's check that our RateLimitPolicy for the /store/inventory has been applied and works correctly. Recall, our OAS spec had the following limits applied:
-Navigate to the /store/inventory API, click Try it out, and Execute.
-
You'll see a response similar to:
-
{
-"available":10,
-"pending":5,
-"sold":3
-}
-
-
This API has a rate limit applied, so if you send more than 10 requests in a 10 second window, you will see a 429 HTTP Status code from responses, and a "Too Many Requests" message in the response body. Click Execute quickly in succession to see your RateLimitPolicy in action.
Let's check that our AuthPolicy for the /store/admin endpoint has been applied and works correctly. Recall, our OAS spec had the following securitySchemes applied:
Navigate to the /store/admin API, click Try it out, and Execute.
-You'll get a 401 response.
-
You can set a value for the api_key header by clicking Authorize at the top of the page. Set a value of secret.
-This api key value is stored in the petstore-api-key Secret in the petstore namespace.
-Try the /store/admin endpoint again and you should get a 200 response with the following:
You should be able to access the Swagger Editor at http://localhost:8080.
-Our /store/inventory API needs some additonal rate limiting. This is one of our slowest, most expensive services, so we'd like to rate limit it further.
-
In your openapi.yaml, navigate to the /store/inventory endpoint in the paths block. Modify the rate_limit block to further restrict the amount of requests this endpoint can serve to 2 requests per 10 seconds:
Save your updated spec - File > Save as YAML > and update your existing openapi.yaml. You may need to copy the file from your Downloads folder to the location of the petstore repository.
-
Next we'll re-generate our RateLimitPolicy with kuadrantctl:
-
# Generate this resource and save:
-kuadrantctlgeneratekuadrantratelimitpolicy--oasopenapi.yaml|yq-P|teeresources/ratelimitpolicy.yaml
-
-# Apply this resource to our cluster:
-kubectl--contextkind-api-workload-1apply-f./resources/ratelimitpolicy.yaml
-
-
At this stage you can optionally check in all the changes to the repo if you forked it.
-
# Optionally add, commit & push the changes to your fork
-gitaddresources
-gitcommit-am"Generated AuthPolicy,RateLimitPolicy & HTTPRoute"
-gitpush# You may need to set an upstream as well
-
Label the 1st cluster as being in the 'EU' region,
-and the 2nd cluster as being in the 'US' region.
-These labels are used by the DNSPolicy for configuring geo DNS.
python3 and pip3: these are required for this part of the walkthrough
-
-
To demonstrate traffic management by geographical region, we'll use a tool called 'geosight'. This tool resolves hostnames from different regions, fetches a website from the resulting DNS record address and takes a screenshot. The petstore app has been configured to serve a flag image based on which region it is running in. In the 1st cluster, the EU flag is used. In the 2nd cluster, the US flag is used.
-
To install 'geosight', run the following commands:
Now that the app developer has deployed their app, new metrics and data is now available in the platform engineer dashboard seen in the previous step https://grafana.172.31.0.2.nip.io:
-
-
Gateways, routes and policies
-
Constraints & Violations (there should be no violations present)
You now have a local environment with a reference architecture to design and deploy an API in a kube native way, using Kuadrant and other open source tools.
Kuadrant provides connectivity, security and service protection capabilities in both a single and multi-cluster environment. It exposes these capabilities in the form of Kubernetes CRDs that implement the Gateway API concept of policy attachment. These policy APIs can target specific Gateway API resources such as Gateways and HTTPRoutes to extend their capabilities and configuration. They enable platform engineers to secure, protect and connect their infrastructure and allow application developers to self service and refine policies to their specific needs in order to protect exposed endpoints.
Kuadrant architecture is defined and implemented with both control plane and data plane components.
-
The control plane is where policies are exposed and expressed as Kubernetes APIs and reconciled by a policy controller.
-
The data plane is where Kuadrant's "policy enforcement" components exist. These components are configured by the control plane and integrate either directly with the Gateway provider or via external integrations.
The control plane is a set of controllers and operators that are responsible for for installation and configuration of other components such as the data plane enforcement components and configuration of the Gateway to enable the data plane components to interact with incoming requests. The control plane also owns and reconciles the policy CRD APIs into more complex and specific configuration objects that the policy enforcement components consume in order to know the rules to apply to incoming requests or the configuration to apply to external integrations such as DNS and ACME providers.
Installs and configures the Limitador data plane component based on the Limitador CR. Limits specified in the limitador CR are mountd via configmap into the limitador component.
Manages TLS certificates for our components and for the Gateways. Consumes Certificate resources created by Kuadrant operator in response to the TLSPolicy.
DNS operator consumes DNSRecord resources that are configured via the DNSPolicy api and applies them into the targeted cloud DNS provider
-AWS, Azure and Google DNS are our main targets
The data plane components sit in the request flow and are responsible for enforcing configuration defined by policy and providing service protection capabilities based on configuration managed and created by the control plane.
Complies with the with Envoy rate limiting API to provide rate limiting to the gateway. Consumes limits from a configmap created based on the RateLimitPolicy API.
Complies with the Envoy external auth API to provide auth integration to the gateway. It provides both Authn and Authz. Consumes AuthConfigs created by the kuadrant operator based on the defined AuthPolicy API.
Uses the Proxy WASM ABI Spec to integrate with Envoy and provide filtering and connectivity to Limitador for request time enforcement of and rate limiting.
In a single cluster, you have the Kuadrant control plane and data plane sitting together. It is configured to integrate with Gateways on the same cluster and configure a DNS zone via a DNS provider secret (configured alongside a DNSPolicy). Storage of rate limit counters is possible but not required as they are not being shared.
In the default multi-cluster setup. Each individual cluster has Kuadrant installed. Each of these clusters are unaware of the other. They are effectively operating as single clusters. The multi-cluster aspect is created by sharing access with the DNS zone, using a shared host across the clusters and leveraging shared counter storage.
-The zone is operated on independently by each of DNS operator on both clusters to form a single cohesive record set. More details on this can be found in the following RFC document: TODO add link.
-The rate limit counters can also be shared and used by different clusters in order to provide global rate limiting. This is achieved by connecting each instance of Limitador to a shared data store that uses the Redis protocol. There is another option available for achieving multi-cluster connectivity (see intgrations below) that requires the use of a "hub" cluster and integration with OCM (open cluster management).
-
-
Shown above is a multi-cluster multi ingress gateway topology. This might be used to support a geographically distributed system for example. However, it is also possible to leverage overlay networking tools such as Skupper that integrate at the Kubernetes service level to have a single gateway cluster that then integrates with multiple backends (on different clusters or in custom infrastructure).
Provides a multi-cluster control plane to enable the defining and distributing of Gateways across multiple clusters.
-
-
While the default setup is to leverage a distributed configuration for DNS and rate limiting. There is also a component that offers experimental integration with Open Cluster Management.
-
In this setup, the OCM integration controller is installed into the HUB alongside the DNS Operator and the cert-manager. This integration allows you to define gateways in the Hub and distribute them to "spoke" clusters. The addresses of these gateways are gathered from the spokes and aggregated back to the hub. The Kuadrant operator and DNS operator then act on this information as though it were a single cluster gateway with multiple addresses. The DNS zone in the configured DNS provider is managed centrally by one DNS operator instance.
It is important to note that Kuadrant is not in itself a gateway provider. Kuadrant provides a set of valuable policy APIs that enhance Gateway API via its defined policy attachment extension point. The policy APIs are reconciled by a set of policy controllers and enforced via integration at different points to configure, enhance and secure the application connectivity provided via Gateway API and the underlying gateway provider.
-These policy extensions are focused around areas such as DNS management supporting global load balancing and health checks, alongside service protection specific APIs such as rate limiting and auth. Kuadrant also integrates with Open Cluster Management as a multi-cluster control plane to enable defining and distributing Gateways across multiple clusters, providing load balancing and tls management for these distributed gateways. These integrations and features can be managed centrally in a declarative way from the Open Cluster Management Hub using Kubernetes resources.
The Kuadrant architecture is spread across a control plane and also a data plane. Kuadrant can work in both a single and multi-cluster context.
-Currently in order for all APIs to work in a single or multi-cluster context you need to have Open Cluster Management installed. While this may change in the future, this approach allows us to start with a single cluster and seamlessly scale as more clusters are added.
-
The control plane is where policies are exposed and expressed as kubernetes APIs and reconciled by the Kuadrant policy controllers.
-
The data plane is where Kuadrant's service protection components, configured by the control plane policies, are enforced within the gateway instance as part of the request flow.
A control plane component is something responsible for accepting instruction via a CRD based API and ensuring that configuration is manifested into state that can be acted on.
Configures DNS providers (e.g AWS Route 53) and TLS providers
-
Focused around use cases involving distributed gateways (for example across clouds or geographic regions)
-
Integrates with Open Cluster Management as the multi-cluster management hub to distribute and observe gateway status based on the clusters they are deployed to. Works directly with Open Cluster Management APIs such PlacementDecision and ManifestWork.
A data plane component sits in the request flow and is responsible for enforcing policy and providing service protection capabilities based on configuration managed and created by the control plane.
Uses the Proxy WASM ABI Spec to integrate with Envoy and provide filtering and connectivity to Limitador for request time enforcement of and rate limiting
In order to provide its full suite of functionality, Kuadrant has several dependencies. Some of these are optional depending on the functionality needed.
Rather than providing any Kuadrant specific observability tooling, we instead look to leverage existing tools and technologies to provide observability capabilities for ingress.
Kuadrant has a multi-cluster gateway controller that is intended to run in a Open Cluster Management provided "Hub" cluster. This cluster is effectively a central management cluster where policy and gateways along with all that Open Cluster Management offers can be defined and distributed to the managed "spoke" clusters.
In a single cluster context, the overall architecture remains the same as above, the key difference is that the Hub and Spoke cluster are now a single cluster rather than multiple clusters. This is how we are initially supporting single cluster.
-
How does Kuadrant leverage Open Cluster Management?¶
-
Kuadrant deploys a multi-cluster gateway controller into the Open Cluster Management hub (a control plane that manages a set of "spoke" clusters where workloads are executed). This controller offers its own APIs but also integrates with hub CRD based APIs (such as the placement API) along with the Gateway API CRD based APIs in order to provide multi-cluster Gateway capabilities to the hub and distribute actual gateway instances to the spokes. See the Open Cluster Management docs for further details on the hub spoke architecture.
-
As part of installing Kuadrant, the Gateway API CRDs are also installed into the hub cluster and Kuadrant defines a standard Gateway API GatewayClass resource that the multi-cluster gateway controller is the chosen controller for.
-
Once installed, an Open Cluster Management user can then (with the correct RBAC in place) define in the standard way a Gateway resource that inherits from the Kuadrant configured GatewayClass in the hub. There is nothing unique about this Gateway definition, the difference is what it represents and how it is used. This Gateway is used to represent a "multi-cluster" distributed gateway. As such there are no pods running behind this Gateway instance in the hub cluster, instead it serves as a template that the Kuadrant multi-cluster gateway controller reconciles and distributes to targeted spoke clusters. It leverages the Open Cluster Management APIs to distribute these gateways (more info below) and aggregates the status information from each spoke cluster instance of this gateway back to this central definition, in doing this it can represent the status of the gateway across multiple clusters but also use that information to integrate with DNS providers etc.
In order for a multi-cluster gateway to be truly useful, it needs to be distributed or "placed" on a specific set of hub managed spoke clusters. Open Cluster Management is responsible for a set of placement and replication APIs. Kuadrant is aware of these APIs, and so when a given gateway is chosen to be placed on a set of managed clusters, Kuadrant multi-cluster gateway controller will ensure the right resources (ManifestWork) are created in the correct namespaces in the hub. Open Cluster Management then is responsible for syncing these to the actual spoke cluster and reporting back the status of these resources to the Hub. A user would indicate which clusters they want a gateway placed on by using a Placement and then labeling the gateway using the cluster.open-cluster-management.io/placement label.
-
In order for the Gateway to be instantiated, we need to know what underlying gateway provider is being used on the spoke clusters. Admins can then set this provider in the hub via the GatewayClass params. In the hub, Kuadrant will then apply a transformation to the gateway to ensure when synced it references this spoke gateway provider (Istio for example).
-
It is the Open Cluster Management workagent that is responsible for syncing down and applying the resources into the managed spoke cluster. It is also responsible for syncing status information back to the hub. It is the multi-cluster gateway controller that is responsible for aggregating this status.
-
The status information reported back to the Hub is used by the multi-cluster gateway controller to know what LB hosts / IPAddresses to use for DNSRecords that it creates and manages.
-
-
More info on the Open Cluster Management hub and spoke architecture can be found here
-
How does Kuadrant integrate with Gateway Providers?¶
-
Currently the Kuadrant data plane only integrates with an Istio based gateway provider:
-
-
It registers Authorino with the IstioOperator as an auth provider so that Authorino can be used as an external auth provider.
-
It leverages an EnvoyFilter to register the rate limiting service as an upstream service.
-
Based on the Kuadrant AuthPolicy, it leverages Istio's AuthorizationPolicy resource to configure when a request should trigger Authorino to be called for a given host, path and method etc.
-
It provides a WebAssembly (WASM) Plugin that conforms to the Proxy WASM ABI (application binary interface). This WASM Plugin is loaded into the underlying Envoy based gateway provider and configured via the Kuadrant Operator based on defined RateLimitPolicy resources. This binary is executed in response to a HTTP request being accepted by the gateway via the underlying Envoy instance that provides the proxy layer for the Gateway (IE Envoy). This plugin is configured with the correct upstream rate limit service name and when it sees a request, based on the provided configuration, it will trigger a call to the installed Limitador that is providing the rate limit capabilities and either allow the request to continue or trigger a response to the client with a 429 (too many requests) HTTP code.
The initial creation of these APIs (gateways, policies etc) is done by the relevant persona in the control plane just as they would any other k8s resource. We use the term cluster admin or gateway admin as the operations type persona configuring, and placing gateways.
-As shown above, in a multi-cluster configuration. API definitions are pulled from the Hub and "manifested" into the spokes. The Status of those synced resources are reported back to the Hub. The same happens for a single cluster, the only difference being the work agent hub controllers are all installed on one cluster.
In order to enforce the policy configuration, components in the control plane and data plane can reach out to configured 3rd parties such as cloud based DNS provider, TLS providers and Auth providers.
Requests coming through the gateway instance can be sent to Limitador based on configuration of the WASM plugin installed into the Envoy based gateway provider or to Authorino based configuration provided by the Istio AuthorizationPolicy.
-Each of these components have the capability to see the request and need to in order to make the required decision. Each of these components can also prevent the request from reaching its intended backend destination based on user configuration.
As all of the APIs are CRDs, auth around creating these resources is handled in the standard way IE by the kubernetes cluster and RBAC. There is no relationship by default between the Auth features provided by Authorino to application developers and the auth requirements of the cluster API server.
-
For Auth between Spoke and Hub see Open Cluster Management docs
Kuadrant doesn't provide any specific observability components, but rather provides a reference setup using well known and established components along with some useful dashboards to help observe key things around the Gateways. The focus of this setup, is in the context of a multi-cluster setup where Open Cluster Management is installed and gateways are being defined and distributed from that hub.
This section is here to provide some insight into architectural changes that may be seen in the near future:
-
What is in this doc represents the architecture at point our MVP release. Below are some areas that we have identified that are likely to change in the coming releases. As these happen, this doc will also evolve.
-
-
We want to separate out the ocm integration into its own controller so that policies can evolve without a coupling to any one multi-cluster management solution
-
We want to separate the policies into their own controller that is capable of supporting both single (without Open Cluster Management) and multi-cluster (with Open Cluster Management enabled) contexts, so that the barrier to entry is reduced for those starting with a single cluster
-
We want to allow for an on cluster DNS Provider such as CoreDNS so that we can provide an implementation that is disconnected from any cloud provider and provides more flexible DNS setups.
-
We will look to reduce our integration with Istio and want to provide integration with additional gateway providers such as EnvoyGateway
Kuadrant is developing a set of loosely coupled functionalities built directly on top of Kubernetes.
-Kuadrant aims to allow customers to just install, use and understand those functionalities they need.
Currently, the installation tool of kuadrant, the kuadrantctl CLI,
-installs all or nothing. Installing more than the customer needs adds unneeded complexity and operational effort.
-For example, if a customer is looking for rate limiting and not interested in authentication functionality,
-then the customer should be able to just install and run that part of Kuadrant.
Install only required components. Operate only required components.
-
-
Reduce system complexity and operational effort to the minimum required.
-Components in this context make reference to deployments and running instances.
-
-
Expose only the activated functionalities
-
-
A user of a partial Kuadrant install should not be confronted with data in custom resources that
-has no meaning or is not accessible in their partial Kuadrant install. The design of the kuadrant
-API should have this goal into account.
The kuadrant installation mechanism should offer modular installation to enable/disable loosely coupled pieces of kuadrant.
-Modular installation options should be feature oriented rather than deployment component oriented.
-Then, it is up to the installation tool to decide what components need to be deployed and how to
-configure it.
-
Each feature, or part of it, is eligible to be included or excluded when installing kuadrant.
-
Some profiles can be defined to group set of commonly required features. Naming the profiles
-allows the customer to easily express wanted installation configuration. Furthermore, profiles
-not only can be used to group a set of features, profiles can be used to define deployment options.
-
-
-
-
Name
-
Description
-
-
-
-
-
Minimal
-
Minimal installation required to run an API without any protection, analytics or API management. Default deployment option
-
-
-
AuthZ
-
Authentication and authorization mechanisms activated
-
-
-
RateLimit
-
Basic rate limit (only pre-auth rate limit) features
-
-
-
Full
-
Full featured kuadrant installation
-
-
-
-
A kuadrant operator, together with a design of a kuadrant CRD is desired.
-Not only for kuadrant installation, but also for lifecycle management.
-Additionally, the kuadrantctl CLI tool can also
-be useful to either deploy kuadrant components and manifests or just deploy the kuadrant operator.
-
The kuadrant control plane should be aware of the installed profile via env vars or command line params
-in the control plane running components. With that information, the control plane can decide to
-enable or disable CRD watching, label and annotation monitoring and ultimately reject any configuration
-object that relies on disabled functionality. The least a customer can expect from kuadrant is to be
-consistent and reject any functionality request that cannot provide.
The RateLimitPolicy API (v1beta1), particularly its RateLimit type used in ratelimitpolicy.spec.rateLimits, designed in part to fit the underlying implementation based on the Envoy Rate limit filter, has been proven to be complex, as well as somewhat limiting for the extension of the API for other platforms and/or for supporting use cases of not contemplated in the original design.
-
Users of the RateLimitPolicy will immediately recognize elements of Envoy's Rate limit API in the definitions of the RateLimit type, with almost 1:1 correspondence between the Configuration type and its counterpart in the Envoy configuration. Although compatibility between those continue to be desired, leaking such implementation details to the level of the API can be avoided to provide a better abstraction for activators ("matchers") and payload ("descriptors"), stated by users in a seamless way.
-
Furthermore, the Limit type – used as well in the RLP's RateLimit type – implies presently a logical relationship between its inner concepts – i.e. conditions and variables on one side, and limits themselves on the other – that otherwise could be shaped in a different manner, to provide clearer understanding of the meaning of these concepts by the user and avoid repetition. I.e., one limit definition contains multiple rate limits, and not the other way around.
spec.rateLimits[] replaced with spec.limits{<limit-name>: <limit-definition>}
-
spec.rateLimits.limits replaced with spec.limits.<limit-name>.rates
-
spec.rateLimits.limits.maxValue replaced with spec.limits.<limit-name>.rates.limit
-
spec.rateLimits.limits.seconds replaced with spec.limits.<limit-name>.rates.duration + spec.limits.<limit-name>.rates.unit
-
spec.rateLimits.limits.conditions replaced with spec.limits.<limit-name>.when, structured field based on well-known selectors, mainly for expressing conditions not related to the HTTP route (although not exclusively)
-
spec.rateLimits.limits.variables replaced with spec.limits.<limit-name>.counters, based on well-known selectors
-
spec.rateLimits.rules replaced with spec.limits.<limit-name>.routeSelectors, for selecting (or "sub-targeting") HTTPRouteRules that trigger the limit
-
new matcher spec.limits.<limit-name>.routeSelectors.hostnames[]
-
spec.rateLimits.configurations removed – descriptor actions configuration (previously spec.rateLimits.configurations.actions) generated from spec.limits.<limit-name>.when.selector ∪ spec.limits.<limit-name>.counters and unique identifier of the limit (associated with spec.limits.<limit-name>.routeSelectors)
-
Limitador conditions composed of "soft" spec.limits.<limit-name>.when conditions + a "hard" condition that binds the limit to its trigger HTTPRouteRules
The following are examples of RLPs targeting the route and the gateway. Each example is independent from the other.
-
Example 1. Minimal example - network resource targeted entirely without filtering, unconditional and unqualified rate limiting¶
-
In this example, all traffic to *.toystore.acme.com will be limited to 5rps, regardless of any other attribute of the HTTP request (method, path, headers, etc), without any extra "soft" conditions (conditions non-related to the HTTP route), across all consumers of the API (unqualified rate limiting).
-
apiVersion:kuadrant.io/v2beta1
-kind:RateLimitPolicy
-metadata:
-name:toystore-infra-rl
-namespace:toystore
-spec:
-targetRef:
-group:gateway.networking.k8s.io
-kind:HTTPRoute
-name:toystore
-limits:
-base:# user-defined name of the limit definition - future use for handling hierarchical policy attachment
--rates:# at least one rate limit required
--limit:5
-unit:second
-
-
- How is this RLP implemented under the hood?
-
-
Example 2. Targeting specific route rules, with counter qualifiers, multiple rates per limit definition and "soft" conditions¶
-
In this example, a distinct limit will be associated ("bound") to each individual HTTPRouteRule of the targeted HTTPRoute, by using the routeSelectors field for selecting (or "sub-targeting") the HTTPRouteRule.
-
The following limit definitions will be bound to each HTTPRouteRule:
-- /toys* → 50rpm, enforced per username (counter qualifier) and only in case the user is not an admin ("soft" condition).
-- /assets/* → 5rpm / 100rp12h
-
Each set of trigger matches in the RLP will be matched to all HTTPRouteRules whose HTTPRouteMatches is a superset of the set of trigger matches in the RLP. For every HTTPRouteRule matched, the HTTPRouteRule will be bound to the corresponding limit definition that specifies that trigger. In case no HTTPRouteRule is found containing at least one HTTPRouteMatch that is identical to some set of matching rules of a particular limit definition, the limit definition is considered invalid and reported as such in the status of RLP.
-
apiVersion:kuadrant.io/v2beta1
-kind:RateLimitPolicy
-metadata:
-name:toystore-per-endpoint
-namespace:toystore
-spec:
-targetRef:
-group:gateway.networking.k8s.io
-kind:HTTPRoute
-name:toystore
-limits:
-toys:
-rates:
--limit:50
-duration:1
-unit:minute
-counters:
--auth.identity.username
-routeSelectors:
--matches:# matches the 1st HTTPRouteRule (i.e. GET or POST to /toys*)
--path:
-type:PathPrefix
-value:"/toys"
-when:
--selector:auth.identity.group
-operator:neq
-value:admin
-
-assets:
-rates:
--limit:5
-duration:1
-unit:minute
--limit:100
-duration:12
-unit:hour
-routeSelectors:
--matches:# matches the 2nd HTTPRouteRule (i.e. /assets/*)
--path:
-type:PathPrefix
-value:"/assets/"
-
-
- How is this RLP implemented under the hood?
-
-
Example 3. Targeting a subset of an HTTPRouteRule - HTTPRouteMatch missing¶
-
Consider a 150rps rate limit set on requests to GET /toys/special. Such specific application endpoint is covered by the first HTTPRouteRule in the HTTPRoute (as a subset of GET or POST to any path that starts with /toys). However, to avoid binding limits to HTTPRouteRules that are more permissive than the actual intended scope of the limit, the RateLimitPolicy controller requires trigger matches to find identical matching rules explicitly defined amongst the sets of HTTPRouteMatches of the HTTPRouteRules potentially targeted.
-
As a consequence, by simply defining a trigger match for GET /toys/special in the RLP, the GET|POST /toys* HTTPRouteRule will NOT be bound to the limit definition. In order to ensure the limit definition is properly bound to a routing rule that strictly covers the GET /toys/special application endpoint, first the user has to modify the spec of the HTTPRoute by adding an explicit HTTPRouteRule for this case:
Example 4. Targeting a subset of an HTTPRouteRule - HTTPRouteMatch found¶
-
This example is similar to Example 3. Consider the use case of setting a 150rpm rate limit on requests to GET /toys*.
-
The targeted application endpoint is covered by the first HTTPRouteRule in the HTTPRoute (as a subset of GET or POST to any path that starts with /toys). However, unlike in the previous example where, at first, no HTTPRouteRule included an explicit HTTPRouteMatch for GET /toys/special, in this example the HTTPRouteMatch for the targeted application endpoint GET /toys* does exist explicitly in one of the HTTPRouteRules, thus the RateLimitPolicy controller would find no problem to bind the limit definition to the HTTPRouteRule. That would nonetheless cause a unexpected behavior of the limit triggered not strictly for GET /toys*, but also for POST /toys*.
-
To avoid extending the scope of the limit beyond desired, with no extra "soft" conditions, again the user must modify the spec of the HTTPRoute, so an exclusive HTTPRouteRule exists for the GET /toys* application endpoint:
-
apiVersion:gateway.networking.k8s.io/v1alpha2
-kind:HTTPRoute
-metadata:
-name:toystore
-namespace:toystore
-spec:
-parentRefs:
--name:istio-ingressgateway
-namespace:istio-system
-hostnames:
--"*.toystore.acme.com"
-rules:
--matches:# first HTTPRouteRule split into two – one for GET /toys*, other for POST /toys*
--path:
-type:PathPrefix
-value:"/toys"
-method:GET
-backendRefs:
--name:toystore
-port:80
--matches:
--path:
-type:PathPrefix
-value:"/toys"
-method:POST
-backendRefs:
--name:toystore
-port:80
--matches:
--path:
-type:PathPrefix
-value:"/assets/"
-backendRefs:
--name:toystore
-port:80
-filters:
--type:ResponseHeaderModifier
-responseHeaderModifier:
-set:
--name:Cache-Control
-value:"max-age=31536000,immutable"
-
-
The RLP can then target the new HTTPRouteRule strictly:
-
apiVersion:kuadrant.io/v2beta1
-kind:RateLimitPolicy
-metadata:
-name:toy-readers
-namespace:toystore
-spec:
-targetRef:
-group:gateway.networking.k8s.io
-kind:HTTPRoute
-name:toystore
-limits:
-toyReaders:
-rates:
--limit:150
-unit:second
-routeSelectors:
--matches:# matches the new more specific HTTPRouteRule (i.e. GET /toys*)
--path:
-type:PathPrefix
-value:"/toys"
-method:GET
-
-
- How is this RLP implemented under the hood?
-
-
Example 5. One limit triggered by multiple HTTPRouteRules¶
-
In this example, both HTTPRouteRules, i.e. GET|POST /toys* and /assets/*, are targeted by the same limit of 50rpm per username.
-
Because the HTTPRoute has no other rule, this is technically equivalent to targeting the entire HTTPRoute and therefore similar to Example 1. However, if the HTTPRoute had other rules or got other rules added afterwards, this would ensure the limit applies only to the two original route rules.
Example 6. Multiple limit definitions targeting the same HTTPRouteRule¶
-
In case multiple limit definitions target a same HTTPRouteRule, all those limit definitions will be bound to the HTTPRouteRule. No limit "shadowing" will be be enforced by the RLP controller. Due to how things work as of today in Limitador nonetheless (i.e. the rule of the most restrictive limit wins), in some cases, across multiple limits triggered, one limit ends up "shadowing" others, depending on further qualification of the counters and the actual RL values.
-
E.g., the following RLP intends to set 50rps per username on GET /toys*, and 100rps on POST /toys* or /assets/*:
-
apiVersion:kuadrant.io/v2beta1
-kind:RateLimitPolicy
-metadata:
-name:toystore-per-endpoint
-namespace:toystore
-spec:
-targetRef:
-group:gateway.networking.k8s.io
-kind:HTTPRoute
-name:toystore
-limits:
-readToys:
-rates:
--limit:50
-unit:second
-counters:
--auth.identity.username
-routeSelectors:
--matches:# matches the 1st HTTPRouteRule (i.e. GET or POST to /toys*)
--path:
-type:PathPrefix
-value:"/toys"
-method:GET
-
-postToysOrAssets:
-rates:
--limit:100
-unit:second
-routeSelectors:
--matches:# matches the 1st HTTPRouteRule (i.e. GET or POST to /toys*)
--path:
-type:PathPrefix
-value:"/toys"
-method:POST
--matches:# matches the 2nd HTTPRouteRule (i.e. /assets/*)
--path:
-type:PathPrefix
-value:"/assets/"
-
-
- How is this RLP implemented under the hood?
-
-
limits:
--conditions:# actually applies to GET|POST /toys*
--toystore/toystore-per-endpoint/readToys == "1"
-variables:
--auth.identity.username
-max_value:50
-seconds:1
-namespace:kuadrant
--conditions:# actually applies to GET|POST /toys* and /assets/*
--toystore/toystore-per-endpoint/postToysOrAssets == "1"
-max_value:100
-seconds:1
-namespace:kuadrant
-
-
-
-
This example was only written in this way to highlight that it is possible that multiple limit definitions select a same HTTPRouteRule. To avoid over-limiting between GET|POST /toys* and thus ensure the originally intended limit definitions for each of these routes apply, the HTTPRouteRule should be split into two, like done in Example 4.
-
Example 7. Limits triggered for specific hostnames¶
-
In the previous examples, the limit definitions and therefore the counters were set indistinctly for all hostnames – i.e. no matter if the request is sent to games.toystore.acme.com or dolls.toystore.acme.com, the same counters are expected to be affected. In this example on the other hand, a 1000rpd rate limit is set for requests to /assets/* only when the hostname matches games.toystore.acme.com.
-
First, the user needs to edit the HTTPRoute to make the targeted hostname games.toystore.acme.com explicit:
Note: Additional meaning and context may be given to this use case in the future, when discussing defaults and overrides.
-
-
Targeting a Gateway is a shortcut to targeting all individual HTTPRoutes referencing the gateway as parent. This differs from Example 1 nonetheless because, by targeting the gateway rather than an individual HTTPRoute, the RLP applies automatically to all HTTPRoutes pointing to the gateway, including routes created before and after the creation of the RLP. Moreover, all those routes will share the same limit counters specified in the RLP.
Descriptor actions composed from selectors used in the limit definitions (spec.limits.<limit-name>.when.selector and spec.limits.<limit-name>.counters) plus a fixed identifier of the route rules (spec.limits.<limit-name>.routeSelectors)
-
-
-
Abstract the Envoy-specific concepts of "actions" and "descriptors"
-
No risk of mismatching descriptors keys between "actions" and actual usage in the limits
-
No user-defined generic descriptors (e.g. "limited = 1")
-
Source value of the selectors defined from an implicit "context" data structure
-
-
-
-
-
Key-value descriptors
-
Structured descriptors referring to a contextual well-known data structure
By completely dropping out the configurations field from the RLP, composing the RL descriptor actions is now done based essentially on the selectors listed in the when conditions and the counters, plus an artificial condition used to bind the HTTPRouteRules to the corresponding limits to trigger in Limitador.
-
The descriptor actions composed from the selectors in the "soft" when conditions and counter qualifiers originate from the direct references these selectors make to paths within a well-known data structure that stores information about the context (HTTP request and ext-authz filter). These selectors in "soft" when conditions and counter qualifiers are thereby called well-known selectors.
-
Other descriptor actions might be composed by the RLP controller to define additional RL conditions to bind HTTPRouteRules and corresponding limits.
Each selector used in a when condition or counter qualifier is a direct reference to a path within a well-known data structure that stores information about the context (L4 and L7 data of the original request handled by the proxy), as well as auth data (dynamic metadata occasionally exported by the external authorization filter and injected by the proxy into the rate-limit filter).
-
The well-known data structure for building RL descriptor actions resembles Authorino's "Authorization JSON", whose context component consists of Envoy's AttributeContext type of the external authorization API (marshalled as JSON). Compared to the more generic RateLimitRequest struct, the AttributeContext provides a more structured and arguibly more intuitive relation between the data sources for the RL descriptors actions and their corresponding key names through which the values are referred within the RLP, in a context of predominantly serving for HTTP applications.
-
To keep compatibility with the Envoy Rate Limit API, the well-known data structure can optionally be extended with the RateLimitRequest, thus resulting in the following final structure.
-
context:# Envoy's Ext-Authz `CheckRequest.AttributeContext` type
-source:
-address:…
-service:…
-…
-destination:
-address:…
-service:…
-…
-request:
-http:
-host:…
-path:…
-method:…
-headers:{…}
-
-auth:# Dynamic metadata exported by the external authorization service
-
-ratelimit:# Envoy's Rate Limit `RateLimitRequest` type
-domain:…# generated by the Kuadrant controller
-descriptors:{…}# descriptors configured by the user directly in the proxy (not generated by the Kuadrant controller, if allowed)
-hitsAddend:…# only in case we want to allow users to refer to this value in a policy
-
From the perspective of a user who writes a RLP, the selectors used in then when and counters fields are paths to the well-known data structure (see Well-known selectors). While desiging a policy, the user intuitively pictures the well-known data structure and states each limit definition having in mind the possible values assumed by each of those paths in the data plane. For example,
-
The user story:
-
-
Each distinct user (auth.identity.username) can send no more than 1rps to the same HTTP path (context.request.http.path).
Artificial Limitador condition for routeSelectors¶
-
For each limit definition that explicitly or implicitly defines a routeSelectors field, the RLP controller will generate an artificial Limitador condition that ensures that the limit applies only when the filterred rules are honoured when serving the request. This can be implemented with a 2-step procedure:
-1. generate an unique identifier of the limit - i.e. <policy-namespace>/<policy-name>/<limit-name>
-2. associate a generic_key type descriptor action with each HTTPRouteRule targeted by the limit – i.e. { descriptor_key: <unique identifier of the limit>, descriptor_value: "1" }.
This proposal tries to keep compatibility with the Envoy API for rate limit and does not introduce any new requirement that otherwise would require the use of wasm shim to be implemented.
-
In the case of implementation of this proposal in the wasm shim, all types of matchers supported by the HTTPRouteMatch type of Gateway API must be also supported in the rate_limit_policies.gateway_actions.rules field of the wasm plugin configuration. These include matchers based on path (prefix, exact), headers, query string parameters and method.
HTTPRoute editing occasionally required
-Need to duplicate rules that don't explicitly include a matcher wanted for the policy, so that matcher can be added as a special case for each of those rules.
-
Risk of over-targeting
-Some HTTPRouteRules might need to be split into more specific ones so a limit definition is not bound to beyond intended (e.g. target method: GET when the route matches method: POST|GET).
-
Prone to consistency issues
-Typos and updates to the HTTPRoute can easily cause a mismatch and invalidate a RLP.
-
Two types of conditions – routeSelectors and when conditions
-Although with different meanings (evaluates in the gateway vs. evaluated in Limitador) and meant for expressing different types of rules (HTTPRouteRule selectors vs. "soft" conditions based on attributes not related to the HTTP request), users might still perceive these as two ways of expressing conditions and find difficult to understand at first that "soft" conditions do not accept expressions related to attributes of the HTTP request.
Requiring users to specify full HTTPRouteRule matches in the RLP (as opposed to any subset of HTTPRoureMatches of targeted HTTPRouteRules – current proposal) contains some of the same drawbacks of this proposal, such as HTTPRoute editing occasionally required and prone to consistency issues. If, on one hand, it eliminates the risk of over-targeting, on the other hand, it does it at the cost of requiring excessively verbose policies written by the users, to the point of sometimes expecting user to have to specify trigger matching rules that are significantly more than what's originally and strictly intended.
-
E.g.:
-
On a HTTPRoute that contains the following HTTPRouteRules (simplified representation):
Then, user needs to include the following trigger in the RLP so only full HTTPRouteRules are specified:
-
{ header: x-canary=true, method: POST }
-{ method: POST }
-
-
The first matching rule of the trigger (i.e. { header: x-canary=true, method: POST }) is beoynd the original user intent of targeting simply { method: POST }.
-
This issue can be even more concerning in the case of targeting gateways with multiple child HTTPRoutes. All the HTTPRoutes would have to be fixed and the HTTPRouteRules that cover for all the cases in all HTTPRoutes listed in the policy targeting the gateway.
-
All limit definitions apply vs. Limit "shadowing"¶
-
The proposed binding between limit definition and HTTPRouteRules that trigger the limits was thought so multiple limit definitions can be bound to a same HTTPRouteRule that triggers those limits in Limitador. That means that no limit definition will "shadow" another at the level of the RLP controller, i.e. the RLP controller will honour the intended binding according to the selectors specified in the policy.
-
Due to how things work as of today in Limitador nonetheless, i.e., the rule of the most restrictive limit wins, and because all limit definitions triggered by a given shared HTTPRouteRule, it might be the case that, across multiple limits triggered, one limit ends up "shadowing" other limits. However, that is by implementation of Limitador and therefore beyond the scope of the API.
-
An alternative to the approach of allowing all limit definitions to be bound to a same selected HTTPRouteRules would be enforcing that, amongst multiple limit definitions targeting a same HTTPRouteRule, only the first of those limits definitions is bound to the HTTPRouteRule. This alternative approach effectively would cause the first limit to "shadow" any other on that particular HTTPRouteRule, as by implementation of the RLP controller (i.e., at API level).
-
While the first approach causes an artificial Limitador condition of the form <policy-ns>/<policy-name>/<limit-name> == "1", the alternative approach ("limit shadowing") could be implemented by generating a descriptor of the following form instead: ratelimit.binding == "<policy-ns>/<policy-name>/<limit-name>".
-
The downside of allowing multiple bindings to the same HTTPRouteRule is that all limits apply in Limitador, thus making status report frequently harder. The most restritive rate limit strategy implemented by Limitador might not be obvious to users who set multiple limit definitions and will require additional information reported back to the user about the actual status of the limit definitions stated in a RLP. On the other hand, it allows enables use cases of different limit definitions that vary on the counter qualifiers, additional "soft" conditions, or actual rate limit values to be triggered by a same HTTPRouteRule.
-
Writing "soft" when conditions based on attributes of the HTTP request¶
-
As a first step, users will not be able to write "soft" when conditions to selective apply rate limit definitions based on attributes of the HTTP request that otherwise could be specified using the routeSelectors field of the RLP instead.
-
On one hand, using when conditions for route filtering would make it easy to define limits when the HTTPRoute cannot be modified to include the special rule. On the other hand, users would miss information in the status. An HTTPRouteRule for GET|POST /toys*, for example, that is targeted with an additional "soft" when condition that specifies that the method must be equal to GET and the path exactly equal to /toys/special (see Example 3) would be reported as rate limited with extra details that this is in fact only for GET /toys/special. For small deployments, this might be considered acceptable; however it would easily explode to unmanageable number of cases for deployments with only a few limit definitions and HTTPRouteRules.
-
Moreover, by not specifying a more strict HTTPRouteRule for GET /toys/special, the RLP controller would bind the limit definition to other rules that would cause the rate limit filter to invoke the rate limit service (Limitador) for cases other than strictly GET /toys/special. Even if the rate limits would still be ensured to apply in Limitador only for GET /toys/special (due to the presence of a hypothetical "soft" when condition), an extra no-op hop to the rate limit service would happen. This is avoided with the current imposed limitation.
-
Example of "soft" when conditions for rate limit based on attributes of the HTTP request (NOT SUPPORTED):
-
apiVersion:kuadrant.io/v2beta1
-kind:RateLimitPolicy
-metadata:
-name:toystore-special-toys
-namespace:toystore
-spec:
-targetRef:
-group:gateway.networking.k8s.io
-kind:HTTPRoute
-name:toystore
-limits:
-specialToys:
-rates:
--limit:150
-unit:second
-routeSelectors:
--matches:# matches the original HTTPRouteRule GET|POST /toys*
--path:
-type:PathPrefix
-value:"/toys"
-method:GET
-when:
--selector:context.request.http.method# cannot omit this selector or POST /toys/special would also be rate limited
-operator:eq
-value:GET
--selector:context.request.http.path
-operator:eq
-value:/toys/special
-
-
- How is this RLP would be implemented under the hood if supported?
-
-
Possible variations for the selectors (conditions and counter qualifiers)¶
-
The main drivers behind the proposed design for the selectors (conditions and counter qualifiers), based on (i) structured condition expressions composed of fields selector, operator, and value, and (ii) when conditions and counters separated in two distinct fields (variation "C" below), are:
-1. consistency with the Authorino AuthConfig API, which also specifies when conditions expressed in selector, operator, and value fields;
-2. explicit user intent, without subtle distinction of meaning based on presence of optional fields.
-
Nonetheless here are a few alternative variations to consider:
Most implementations currently orbiting around Gateway API (e.g. Istio, Envoy Gateway, etc) for added RL functionality seem to have been leaning more to the direct route extension pattern instead of Policy Attachment. That might be an option particularly suitable for gateway implementations (gateway providers) and for those aiming to avoid dealing with defaults and overrides.
In case a limit definition lists route selectors such that some can be bound to HTTPRouteRules and some cannot (see Example 6), do we bind the valid route selectors and ignore the invalid ones or the limit definition is invalid altogether and bound to no HTTPRouteRule at all?
- A: By allowing multiple limit definitions to target a same HTTPRouteRule, the issue here stated will become less often. For the other cases where a limit definition still fails to select an HTTPRouteRule (e.g. due to mismatching trigger matches), the limit definition is not considered invalid. Possibly the limit definitions is considered "stale" (or "orphan"), i.e., not bound to any HTTPRouteRule.
-
What should we fill domain/namespace with, if no longer with the hostname? This can be useful for multi-tenancy.
- A: For now, the domain/namespace field of the RL configuration (Envoy and Limitador ends) will be filled with a fixed (configurable) string (e.g. "kuadrant"). This can change in future to better support multi-tenancy and/or other use cases where a total sharding of the limit definitions within a same instance of Kuadrant is desired.
-
How do we support lists of hostnames in Limitador conditions (single counter)? Should we open an issue for a new in operator?
- A: Not needed. The hostnames must exist in the targeted object explicitly, just like any other routing rules intended to be targeted by a limit definition. By setting the explicit hostname in the targeted network object (Gateway or HTTPRoute), the also becomes a route rules available for "hard" trigger configuration.
-
What "soft" condition operators do we need to support (e.g. eq, neq, exists, nexists, matches)?
-
Do we need special field to define shared counters across clusters/Limitador instances or that's to be solved at another layer (Limitador, Kuadrant CRDs, MCTC)?
Define a well-known structure for users to declare request data selectors in their RateLimitPolicies and AuthPolicies. This structure is referred to as the Kuadrant Well-known Attributes.
The well-known attributes let users write policy rules – conditions and, in general, dynamic values that refer to attributes in the data plane - in a concise and seamless way.
-
Decoupled from the policy CRDs, the well-known attributes:
-1. define a common language for referring to values of the data plane in the Kuadrant policies;
-2. allow dynamically evolving the policy APIs regarding how they admit references to data plane attributes;
-3. encompass all common and component-specific selectors for data plane attributes;
-4. have a single and unified specification, although this specification may occasionally link to additional, component-specific, external docs.
One who writes a Kuadrant policy and wants to build policy constructs such as conditions, qualifiers, variables, etc, based on dynamic values of the data plane, must refer the attributes that carry those values, using the declarative language of Kuadrant's Well-known Attributes.
-
A dynamic data plane value is typically a value of an attribute of the request or an Envoy Dynamic Metadata entry. It can be a value of the outer request being handled by the API gateway or proxy that is managed by Kuadrant ("context request") or an attribute of the direct request to the Kuadrant component that delivers the functionality in the data plane (rate-limiting or external auth).
-
A Well-known Selector is a construct of a policy API whose value contains a direct reference to a well-known attribute. The language of the well-known attributes and therefore what one would declare within a well-known selector resembles a JSON path for navigating a possibly complex JSON object.
-
Example 1. Well-known selector used in a condition
In the example, auth.identity.group is a well-known selector of an attribute group, known to be injected by the external authorization service (auth) to describe the group the user (identity) belongs to. In the data plane, whenever this value is equal to admin, the abstract PaintPolicy policy states that the traffic must be painted red.
In the example, request.headers.x-color-alpha is a selector of a well-known attribute request.headers that gives access to the headers of the context HTTP request. The selector retrieves the value of the x-color-alpha request header to dynamically fill the alpha property of the abstract PaintPolicy policy at each request.
From the Envoy attributes, only attributes that are available before establishing connection with the upstream server qualify as a Kuadrant well-known attribute. This excludes attributes such as the response attributes and the upstream attributes.
-
As for the attributes inherited from Authorino, these are either based on Envoy's AttributeContext type of the external auth request API or from internal types defined by Authorino to fulfill the Auth Pipeline.
-
These two subsets of attributes are unified into a single set of well-known attributes. For each attribute that exists in both subsets, the name of the attribute as specified in the Envoy attributes subset prevails. Example of such is request.id (to refer to the ID of the request) superseding context.request.http.id (as the same attribute is referred in an Authorino AuthConfig).
The following attributes are related to the context HTTP request that is handled by the API gateway or proxy managed by Kuadrant.
-
-
-
-
Attribute
-
Type
-
Description
-
Auth
-
RL
-
-
-
-
-
request.id
-
String
-
Request ID corresponding to x-request-id header value
-
✓
-
✓
-
-
-
request.time
-
Timestamp
-
Time of the first byte received
-
✓
-
✓
-
-
-
request.protocol
-
String
-
Request protocol (“HTTP/1.0”, “HTTP/1.1”, “HTTP/2”, or “HTTP/3”)
-
✓
-
✓
-
-
-
request.scheme
-
String
-
The scheme portion of the URL e.g. “http”
-
✓
-
✓
-
-
-
request.host
-
String
-
The host portion of the URL
-
✓
-
✓
-
-
-
request.method
-
String
-
Request method e.g. “GET”
-
✓
-
✓
-
-
-
request.path
-
String
-
The path portion of the URL
-
✓
-
✓
-
-
-
request.url_path
-
String
-
The path portion of the URL without the query string
-
-
✓
-
-
-
request.query
-
String
-
The query portion of the URL in the format of “name1=value1&name2=value2”
-
✓
-
✓
-
-
-
request.headers
-
Map<String, String>
-
All request headers indexed by the lower-cased header name
-
✓
-
✓
-
-
-
request.referer
-
String
-
Referer request header
-
-
✓
-
-
-
request.useragent
-
String
-
User agent request header
-
-
✓
-
-
-
request.size
-
Number
-
The HTTP request size in bytes. If unknown, it must be -1
-
✓
-
-
-
-
request.body
-
String
-
The HTTP request body. (Disabled by default. Requires additional proxy configuration to enabled it.)
-
✓
-
-
-
-
request.raw_body
-
Array<Number>
-
The HTTP request body in bytes. This is sometimes used instead of body depending on the proxy configuration.
-
✓
-
-
-
-
request.context_extensions
-
Map<String, String>
-
This is analogous to request.headers, however these contents are not sent to the upstream server. It provides an extension mechanism for sending additional information to the auth service without modifying the proto definition. It maps to the internal opaque context in the proxy filter chain. (Requires additional configuration in the proxy.)
The following attributes are available once the downstream connection with the API gateway or proxy managed by Kuadrant is established. They apply to HTTP requests (L7) as well, but also to proxied connections limited at L3/L4.
-
-
-
-
Attribute
-
Type
-
Description
-
Auth
-
RL
-
-
-
-
-
source.address
-
String
-
Downstream connection remote address
-
✓
-
✓
-
-
-
source.port
-
Number
-
Downstream connection remote port
-
✓
-
✓
-
-
-
source.service
-
String
-
The canonical service name of the peer
-
✓
-
-
-
-
source.labels
-
Map<String, String>
-
The labels associated with the peer. These could be pod labels for Kubernetes or tags for VMs. The source of the labels could be an X.509 certificate or other configuration.
-
✓
-
-
-
-
source.principal
-
String
-
The authenticated identity of this peer. If an X.509 certificate is used to assert the identity in the proxy, this field is sourced from “URI Subject Alternative Names“, “DNS Subject Alternate Names“ or “Subject“ in that order. The format is issuer specific – e.g. SPIFFE format is spiffe://trust-domain/path, Google account format is https://accounts.google.com/{userid}.
-
✓
-
-
-
-
source.certificate
-
String
-
The X.509 certificate used to authenticate the identify of this peer. When present, the certificate contents are encoded in URL and PEM format.
-
✓
-
-
-
-
destination.address
-
String
-
Downstream connection local address
-
✓
-
✓
-
-
-
destination.port
-
Number
-
Downstream connection local port
-
✓
-
✓
-
-
-
destination.service
-
String
-
The canonical service name of the peer
-
✓
-
-
-
-
destination.labels
-
Map<String, String>
-
The labels associated with the peer. These could be pod labels for Kubernetes or tags for VMs. The source of the labels could be an X.509 certificate or other configuration.
-
✓
-
-
-
-
destination.principal
-
String
-
The authenticated identity of this peer. If an X.509 certificate is used to assert the identity in the proxy, this field is sourced from “URI Subject Alternative Names“, “DNS Subject Alternate Names“ or “Subject“ in that order. The format is issuer specific – e.g. SPIFFE format is spiffe://trust-domain/path, Google account format is https://accounts.google.com/{userid}.
-
✓
-
-
-
-
destination.certificate
-
String
-
The X.509 certificate used to authenticate the identify of this peer. When present, the certificate contents are encoded in URL and PEM format.
-
✓
-
-
-
-
connection.id
-
Number
-
Downstream connection ID
-
-
✓
-
-
-
connection.mtls
-
Boolean
-
Indicates whether TLS is applied to the downstream connection and the peer ceritificate is presented
-
-
✓
-
-
-
connection.requested_server_name
-
String
-
Requested server name in the downstream TLS connection
-
-
✓
-
-
-
connection.tls_session.sni
-
String
-
SNI used for TLS session
-
✓
-
-
-
-
connection.tls_version
-
String
-
TLS version of the downstream TLS connection
-
-
✓
-
-
-
connection.subject_local_certificate
-
String
-
The subject field of the local certificate in the downstream TLS connection
-
-
✓
-
-
-
connection.subject_peer_certificate
-
String
-
The subject field of the peer certificate in the downstream TLS connection
-
-
✓
-
-
-
connection.dns_san_local_certificate
-
String
-
The first DNS entry in the SAN field of the local certificate in the downstream TLS connection
-
-
✓
-
-
-
connection.dns_san_peer_certificate
-
String
-
The first DNS entry in the SAN field of the peer certificate in the downstream TLS connection
-
-
✓
-
-
-
connection.uri_san_local_certificate
-
String
-
The first URI entry in the SAN field of the local certificate in the downstream TLS connection
-
-
✓
-
-
-
connection.uri_san_peer_certificate
-
String
-
The first URI entry in the SAN field of the peer certificate in the downstream TLS connection
-
-
✓
-
-
-
connection.sha256_peer_certificate_digest
-
String
-
SHA256 digest of the peer certificate in the downstream TLS connection if present
The following attributes are related to the Envoy proxy filter chain. They include metadata exported by the proxy throughout the filters and information about the states of the filters themselves.
The decoupling of the well-known attributes and the language of well-known attributes and selectors from the individual policy CRDs is what makes it somewhat flexible and common across the components (rate-limiting and auth). However, it's less structured and it introduces another syntax for users to get familiar with.
-
This additional language competes with the language of the route selectors (RFC 0001), based on Gateway API's HTTPRouteMatch type.
-
Being "soft-coded" in the policy specs (as opposed to a hard-coded sub-structure inside of each policy type) does not mean it's completely decoupled from implementation in the control plane and/or intermediary data plane components. Although many attributes can be supported almost as a pass-through, from being used in a selector in a policy, to a corresponding value requested by the wasm-shim to its host, that is not always the case. Some translation may be required for components not integrated via wasm-shim (e.g. Authorino), as well as for components integrated via wasm-shim (e.g. Limitador) in special cases of composite or abstraction well-known attributes (i.e. attributes not available as-is via ABI, e.g. auth.identity in a RLP). Either way, some validation of the values introduced by users in the selectors may be needed at some point in the control plane, thus requiring arguably a level of awaresness and coupling between the well-known selectors specification and the control plane (policy controllers) or intermediary data plane (wasm-shim) components.
As an alternative to JSON path-like selectors based on a well-known structure that induces the proposed language of well-known attributes, these same attributes could be defined as sub-types of each policy CRD. The Golang packages defining the common attributes across CRDs could be shared by the policy type definitions to reduce repetition. However, that approach would possibly involve a staggering number of new type definitions to cover all the cases for all the groups of attributes to be supported. These are constructs that not only need to be understood by the policy controllers, but also known by the user who writes a policy.
-
Additionally, all attributes, including new attributes occasionally introduced by Envoy and made available to the wasm-shim via ABI, would always require translation from the user-level abstraction how it's represented in a policy, to the actual form how it's used in the wasm-shim configuration and Authorino AuthConfigs.
-
Not implementing this proposal and keeping the current state of things mean little consistency between these common constructs for rules and conditions on how they are represented in each type of policy. This lack of consistency has a direct impact on the overhead faced by users to learn how to interact with Kuadrant and write different kinds of policies, as well as for the maintainers on tasks of coding for policy validation and reconciliation of data plane configurations.
Authorino's dynamic JSON paths, related to Authorino's Authorization JSON and used in when conditions and inside of multiple other constructs of the AuthConfig, are an example of feature of very similar approach to the one proposed here.
-
Arguably, Authorino's perceived flexibility would not have been possible with the Authorization JSON selectors. Users can write quite sophisticated policy rules (conditions, variable references, etc) by leveraging the those dynamic selectors. Because they are backed by JSON-based machinery in the code, Authorino's selectors have very little to, in some cases, none at all variation compared Open Policy Agent's Rego policy language, which is often used side by side in the same AuthConfigs.
-
Authorino's Authorization JSON selectors are, in one hand, more restrict to the structure of the CheckRequest payload (context.* attributes). At the same time, they are very open in the part associated with the internal attributes built along the Auth Pipeline (i.e. auth.* attributes). That makes Authorino's Authorization JSON selectors more limited, compared to the Envoy attributes made available to the wasm-shim via ABI, but also harder to validate. In some cases, such as of deep references to inside objects fetched from external sources of metadata, resolved OPA objects, JWT claims, etc, it is impossible to validate for correct references.
-
Another experience learned from Authorino's Authorization JSON selectors is that they depend substantially on the so-called "modifiers". Many use cases involving parsing and breaking down attributes that are originally available in a more complex form would not be possible without the modifiers. Examples of such cases are: extracting portions of the path and/or query string parameters (e.g. collection and resource identifiers), applying translations on HTTP verbs into corresponding operations, base64-decoding values from the context HTTP request, amongst several others.
How to deal with the differences regarding the availability and data types of the attributes across clients/hosts?
-
-
-
Can we make more attributes that are currently available to only one of the components common to both?
-
-
-
Will we need some kind of global support for modifiers (functions) in the well-known selectors or those can continue to be an Authorino-only feature?
-
-
-
Does Authorino, which is more strict regarding the data structure that induces the selectors, need to implement this specification or could/should it keep its current selectors and a translation be performed by the AuthPolicy controller?
Gateway admins, need a way to define the DNS policy for a multi-cluster gateway in order to control how much and which traffic reaches these gateways.
-Ideally we would allow them to express a strategy that they want to use without needing to get into the details of each provider and needing to create and maintain dns record structure and individual records for all the different gateways that may be within their infrastructure.
Allow definition of a DNSPolicy that configures load balancing to decide how traffic should be distributed across multiple gateway instances from the central control plane.
managed listener: This is a listener with a host backed by a DNS zone managed by the multi-cluster gateway controller
-
hub cluster: control plane cluster that managed 1 or more spokes
-
spoke cluster: a cluster managed by the hub control plane cluster. This is where gateway are instantiated
-
-
Provide a control plane DNSPolicy API that uses the idea of direct policy attachment from gateway API that allows a load balancing strategy to be applied to the DNS records structure for any managed listeners being served by the data plane instances of this gateway.
-The DNSPolicy also covers health checks that inform the DNS response but that is not covered in this document.
-
Below is a draft API for what we anticipate the DNSPolicy to look like
-
apiVersion:kuadrant.io/v1alpha1
-kind:DNSPolicy
-spec:
-targetRef:# defaults to gateway gvk and current namespace
-name:gateway-name
-health:
-...
-loadBalancing:
-weighted:
-defaultWeight:10
-custom:#optional
--value:AWS#optional with both GEO and weighted. With GEO the custom weight is applied to gateways within a Geographic region
-weight:10
--value:GCP
-weight:20
-GEO:#optional
-defaultGeo:IE# required with GEO. Chooses a default DNS response when no particular response is defined for a request from an unknown GEO.
-
GEO and Weighted load balancing are well understood strategies and this API effectively allow a complex requirement to be expressed relatively simply and executed by the gateway controller in the chosen DNS provider. Our default policy will execute a "Round Robin" weighted strategy which reflects the current default behaviour.
-
With the above API we can provide weighted and GEO and weighted within a GEO. A weighted strategy with a minimum of a default weight is always required and the simplest type of policy. The multi-cluster gateway controller will set up a default policy when a gateway is discovered (shown below). This policy can be replaced or modified by the user. A weighted strategy can be complimented with a GEO strategy IE they can be used together in order to provide a GEO and weighted (within a GEO) load balancing. By defining a GEO section, you are indicating that you want to use a GEO based strategy (how this works is covered below).
-
apiVersion:kuadrant.io/v1alpha1
-kind:DNSPolicy
-name:default-policy
-spec:
-targetRef:# defaults to gateway gvk and current namespace
-name:gateway-name
-loadBalancing:
-weighted:# required
-defaultWeight:10#required, all records created get this weight
-health:
-...
-
-
In order to provide GEO based DNS and allow customisation of the weighting, we need some additional information to be provided by the gateway / cluster admin about where this gateway has been placed. For example if they want to use GEO based DNS as a strategy, we need to know what GEO identifier(s) to use for each record we create and a default GEO to use as a catch-all. Also, if the desired load balancing approach is to provide custom weighting and no longer simply use Round Robin, we will need a way to identify which records to apply that custom weighting to based on the clusters the gateway is placed on.
-
To solve this we will allow two new attributes to be added to the ManagedCluster resource as labels:
These two labels allow setting values in the DNSPolicy that will be reflected into DNS records for gateways placed on that cluster depending on the strategies used. (see the first DNSPolicy definition above to see how these values are used) or take a look at the examples at the bottom.
The attributes provide the key and value we need in order to understand how to define records for a given LB address based on the DNSPolicy targeting the gateway.
-
The kuadrant.io/lb-attribute-geo-code attribute value is provider specific, using an invalid code will result in an error status condition in the DNSrecord resource.
Custom weighting will use the associated custom-weight attribute set on the ManagedCluster to decide which records should get a specific weight. The value of this attribute is up to the end user.
The above is then used in the DNSPolicy to set custom weights for the records associated with the target gateway.
-
-value:GCP
-weight:20
-
-
So any gateway targeted by a DNSPolicy with the above definition that is placed on a ManagedCluster with the kuadrant.io/lb-attribute-custom-weight set with a value of GCP will get an A record with a weight of 20
DNSPolicy should have a ready condition that reflect that the DNSRecords have been created and configured as expected. In the case that there is an invalid policy, the status message should reflect this and indicate to the user that the old DNS has been preserved.
-
We will also want to add a status condition to the gateway status indicating it is effected by this policy. Gateway API recommends the following status condition
apiVersion:kuadrant.io/v1alpha1
-kind:DNSPolicy
-name:SendMoreToAzure
-spec:
-targetRef:# defaults to gateway gvk and current namespace
-name:gateway-name
-loadBalancing:
-weighted:
-defaultWeight:10
-custom:
--attribute:cloud
-value:Azure#any record associated with a gateway on a cluster without this value gets the default
-weight:30
-
Add a DNSPolicy CRD that conforms to policy attachment spec
-
Add a new DNSPolicy controller to MCG
-
DNS logic and record management should all migrate out of the gateway controller into this new DNSPolicy controller as it is the responsibility and domain of the DNSPolicy controller to manage DNS
-
remove the Hosts interface as we want do not want other controllers using this to bring DNS Logic into other areas of the code.
An alternative is to configure all of this yourself manually in a dns provider. This is can be a highly complex dns configuration that it would be easy to get wrong.
At the time being, the RateLimitPolicy and AuthPolicy status doesn't clearly and truthfully communicate the actual state of
-reconciliation and healthiness with its operator managed services, i.e., the Rate Limit service ("Limitador") and
-the Auth service ("Authorino"), referred to as "Kuadrant services".
-
As a consequence, misleading information is shared causing unexpected errors and flawed assumptions.
-
The following are some issues reported in relation to the aforementioned problems:
-* https://github.com/Kuadrant/kuadrant-operator/issues/87
-* https://github.com/Kuadrant/kuadrant-operator/issues/96
-* https://github.com/Kuadrant/kuadrant-operator/issues/140
This design for setting the status of the Kuadrant policy CRs is divided in 2 stages, where each stage could be
-applied/developed in order and would reflect valuable and accurate information with different degrees of acuity.
-
The Policy CRD Status in the following diagrams are simplified as states, which in the
-Reference-level explanation will be translated to the actual Status Conditions.
State of the policy CR defined by: application, validation, and reconciliation of it
-
The main signalization at Stage 1 is about whether a policy CR has been Accepted or not.
-
States rationale:
-* Accepted: This state is reached after the Validation and Reconciliation event has being successfully passed.
-* Invalid: When the Validation process encounters an error, this state will be set.
-* TargetNotFound: This state will be set when the Reconciliation process encounters an error.
-* Conflicted: This state will be set when the Reconciliation process encounters an error.
-
-
Notes:
-* States from the Stage 2 could be implemented as well, but only relying on Validation and Reconciliation events.
Final state of the policy CR defined by: health check with the Kuadrant services (post-reconciliation)
-
The Enforced type is introduced to capture the difference between a policy been reconciled and it's been enforced
-at the service.
-
States rationale:
-* Enforced: After a successful response of the Service Probe, this states communicates the policy is finally enforced.
-* PartiallyEnforced: This state will be set when the Reconciliation event encounters an overlap with other policies.
-* Overridden: This state will be set when the Reconciliation event invalidates the policy because another one takes precedence.
In general, the new states and conditions align with GEP-713.
-
Besides the proposed AcceptedPolicyType, the EnforcedPolicyType would be added to reflect the final state of
-the policy, which means that the policy is showing the synced actual state of the Kuadrant services.
-
The missing FailedPolicyType would be implicitly represented by the TargetNotFound and InvalidPolicyTypeReason.
The Status stanza of the policy CRs must implement Gateway API's PolicyAncestorStatus
-struct. This will provide broader consistency and improved discoverability of effective policies.
Full implementation of Stage 2 states assumes reporting mechanisms in place, provided by the Kuadrant services, that
-allow tracing the state of the configurations applied on the services, back to the original policies, to infer the
-final state of the policy CRs (i.e. whether truly Enforced or not.)
-
Without such, Stage 2 could be only partially achieved, by relying only on Reconciliation events.
Another option was considered (previously referred to as "Option 1"). While valid, this alternative would not align
-with GEP-713, neither it would be as flexible as
-the final design proposed.
-
- Details of the discarded alternative
-
-
-
- This alternative design would come in 3 stages:
-
- **Stage 1 : State of the policy CR defined by: application and validation of it**
-
- This first stage is a simple version where the operator only relies on itself, not checking the healthiness with the
- Kuadrant services, but just validating the Spec.
-
- 
-
- States rationale:
- * `Created`: The initial state. It announces that the policy has successfully being created, the operator acknowledges it.
- * `Applied`: This state is reached after the `Validation` event has being successfully passed.
- * `Failed`: This one would be set when the `Validation` process encounters an error. This could be either condition's failed/error
- state or a top-level condition.
- * `Updated`: From `Failed` or `Applied`, it could be triggered a `Spec Change` event that would move it to this state.
-
- **Stage 2: Further reconciliation check provides a new state**
-
- This following one, besides checking what the former stage does, it also adds the states reflecting the reconciliation
- process of any needed Kubernets object, Kuadrant Services custom resources and any other 3rd party CR required.
- An example would be in the case of the RLP, it would create/update the `ConfigMap` holding the `Limitador` config file.
-
- 
-
- States rationale:
- * `Applied`: The __Applied__ state would not be final, and would be preceding a `Reconciliation` event.
- * `Reconciled`: It communicates that the policy has successfully being reconciled, and any K8s object or required CR has been updated.
- * `Failed`: This one would be reached when either of `Validation` and `Reconcilation` processes have encounter any errors.
-
- **Stage 3: Final state of the policy CR defined by: health check with the Kuadrant services (post-reconciliation)**
-
- The final stage would bring a greater degree of accuracy, thanks for a final process that would check the healthiness and
- configuration version the Kuadrant services currently enforces.
-
- 
-
- States rationale:
- * `Reconciled`: This state would precede the "Health check" process graphed as `Service Probe` event.
- * `Enforced`: After a successful response of the `Service Probe`, this states communicates the policy is finally enforced.
- This is the final top-level condition.
- * `Failed`: Now this state could also be set after encountering errors in the `Service Probe` check.
-
-
-
- The stages before mentioned would follow the [Kubernetes guidelines](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties)
- regarding the Status object definition.
-
- **Conditions**
-
- All conditions are top-level.
-
- | Type | Status | Reason | Message |
- |-------------|--------|-----------------------------|-----------------------------------------------------------------------------|
- | Progressing | True | "PolicyCreated" | "KuadrantPolicy created" |
- | | True | "PolicyUpdated" | "KuadrantPolicy has been updated" |
- | | True | "PolicyApplied" | "KuadrantPolicy has been successfully applied |
- | | True | "PolicyReconciled" | "KuadrantPolicy has been successfully reconciled" |
- | | False | "PolicyEnforced" | "KuadrantPolicy has been successfully enforced" |
- | | False | "PolicyError" | "KuadrantPolicy has encountered an error" |
- | Enforced | True | "PolicyEnforced" | "KuadrantPolicy has been successfully enforced" |
- | | False | "PolicyPartiallyEnforced" | "KuadrantPolicy has encountered some issues and has been partially applied" |
- | | False | "PolicyOverridden" | "KuadrantPolicy is overridden by [policy-ns/policy-name]" |
- | Failed | True | "PolicyValidationError" | "KuadrantPolicy has failed to validate" |
- | | True | "PolicyServiceError" | "KuadrantPolicy has encountered has failed to enforce" |
- | | False | "PolicyEnforced" | "KuadrantPolicy has been successfully enforced" |
-
-
-
Proposal for changes to the DNSPolicy API to allow it to provide a simple routing strategy as an option in a single cluster context. This will remove, but not negate, the complex DNS structure we use in a multi-cluster environment and in doing so allow use of popular dns integrators such as external-dns .
The DNSPolicy API (v1alpha1), was implemented as part of our multi cluster gateway offering using OCM and as such the design and implementation were influenced heavily by how we want multi cluster dns to work.
-
-
Decouple the API entirely from OCM and multi cluster specific concepts.
-
Simplify the DNS record structure created for a gateway listeners host for single cluster use.
-
Improve the likelihood of adoption by creating an integration path for other kubernetes dns controllers such as external-dns.
The DNSPolicy can be used to target a Gateway in a single cluster context and will create dns records for each listener host in an appropriately configured external dns provider.
-In this context the advanced loadbalancing configuration is unnecessary, and the resulting DNSRecord can be created mapping individual listener hosts to a single DNS A or CNAME record by using the simple routing strategy in the DNSPolicy.
-
Example 1. DNSPolicy using simple routing strategy
In the example the api listener has a hostname myapp.mn.hcpapps.net that matches a hosted zone being managed by the provider referenced my-route53-credentials in the DNSPolicy.
-As the simple routing strategy is set in the DNSPolicy a DNSRecord resource with the following contents will be created:
The providerRef is included in the DNSRecord to allow the dns record controller to load the appropriate provider configuration during reconciliation and create the DNS records in the dns provider service e.g. route 53.
-
Example 2. DNSPolicy using simple routing strategy on multi cluster gateway
Similar to example 1, except here the Gateway is a multi cluster gateway that has had its status updated by the Gateway controller to include kuadrant.io/MultiClusterIPAddress type addresses.
-As the simple routing strategy is set in the DNSPolicy a DNSRecord resource with the following contents will be created:
The providerRef field is mandatory and contains a reference to a secret containing provider credentials.
- - spec.providerRef.name - name of the provider resource.
-
A DNSPolicy referencing a providerRef secret will expect that secret to exist in the same namespace. The expected contents of the secrets data is comparable to the dnsProviderSecretRef used by ManageZones.
The CONFIG section of the secrets data will be added to allow provider specific configuration to be stored alongside the providers credentials and can be used during the instantiation of the provider client, and during any provider operations.
-The above for example would use the zoneIDFilter value to limit what hosted zones this provider is allowed to update.
The routingStrategy field is mandatory and dictates what kind of dns record structure the policy will create. Two routing strategy options are allowed simple or weightedGeo.
-
A reconciliation of DNSPolicy processes the target gateway and creates a DNSRecord per listener that is supported by the currently configured provider(hostname matches the hosted zones accessible with the credentials and config). The routing strategy used will determine the contents of the DNSRecord resources Endpoints array.
Simple creates a single endpoint for an A record with multiple targets. Although intended for use in a single cluster context a simple routing strategy can still be used in a multi-cluster environment (OCM hub). In this scenario each clusters address will be added to the targets array to create a multi answer section in the dns response.
WeightedGeo creates a more complex set of endpoints which use a combination of weighted and geo routing strategies. Although intended for use in a multi-cluster environment (OCM hub) it will still be possible to use it in a single cluster context. In this scenario the record structure described above would be created for the single cluster.
-
This is the current default for DNSPolicy in a multi-cluster environment (OCM hub) and more details about it can be found in the original DNSPolicy rfc.
The DNSRecord API is updated to remove the managedZone reference in favour of directly referencing the providerRef credentials instead. The DNSRecord reconciliation will be unchanged except for loading the provider client from providerRef credentials.
-
The DNSPolicy reconciliation will be updated to remove the requirement for a ManagedZone resource to be created before a DNSPolicy can create dns records for it, instead it will be replaced in favour of just listing available zones directly in the currently configured dns provider. If no matching zone is found, no DNSRecord will be created.
-
There is a potential for a DNSRecord to be created successfully, but then a provider updated to remove access. In this case it is the responsibility of the DNSPolicy controller to report appropriate status back to the policy and target resource about the failure to process the record. More details on how status will be reported can be found in rfc-0004
When a provider is configured using a kind not supported by the DNSPolicy controller e.g. ExternalDNS we will be relying on an external controller to correctly update the status of any DNSRecord resources created by our policy. This may have a negative impact on our ability to correctly report status back to the target resource.
-
When using a weightedGeo routing strategy in a single cluster context it is not expected that this will offer multi cluster capabilities without the use of OCM. Currently, it is expected that if you want to create a recordset that contains the addresses of multiple clusters you must use an OCM hub.
The ability to support other kubernetes dns controllers such as ExternalDNS would potentially allow us to contribute to some of these projects in the area of polices for dns management of Gateway resources in kubernetes.
The initial request comes from MGC to configure Redis for Limitador by the following issue #163.
-MGC's current work around is to update the Limitador CR after the deployment with the configuration setting for Redis Instance.
-This change would allow for the configuration of sub components before the Kuadrant is deployed.
-
This reduces the number of CRs that users of Kuadrant are required to modify to get the installation they require.
-The sub components CRs (Authorino, Limitador) never have to be modified by a Kuadrant user (and should never be modified by a Kuadrant User).
As the Kuadrant operator would be responsible for reconciling these configurations into the requested components, restrictions and limitations can be placed on the components which maybe allowed in a standalone installation.
-An example in this space is the disk storage for Limitador which is a new feature and the Kuadrant installation may not want to support it till there is a proven track record for the feature.
-
For existing Kuadrant Users this may be a possible breaking changes if those users manually configure the Kuadrant sub components via their CRs.
-A guide can be created to help migrate the users configurations to the Kuadrant CR.
-This guide can be part of the release notes and/or possibly released before the release of Kuadrant.
-
The deployment configuration for each component can be placed in the Kuadrant CR.
-These configurations are then reconciled into the CRs for each component.
-Only the options below are exposed in the Kuadrant CR.
-All fields in the spec are optional.
The Kuadrant operator will watch for changes in the Authorino and Limitador CRs, reconciling back any changes that a user may do to these configurations.
-How ever Kuadrant operator will not reconcile fields that are given above.
-An example of this is the image field on the Authorino CR.
-This field allows a user to set the image that Authorino is deployed with.
-The feature is meant for dev and testing purposes.
-If a user wishes to use a different image, they can.
-Kuadrant assumes they know what they are doing but requires the user to set the change on the component directly.
-
Only the sub component operator will be responsible for actioning the configurations pasted from the Kuadrant CR to the sub components CR.
-This ensure no extra changes will be required in the sub operators to meet the needs of Kuadrant.
-
Status errors related to the configuration of the sub components should be reported back to the Kuadrant CR.
-The errors messages in Kuadrant state what components are currently having issue and which resource to review for more details.
All the fields in the Authorino and Limitador CRs that are configurable in the Kuadrant CR are optional and have sound defaults.
-Kuadrant needs to remain installable with out having to set any spec in the Kuadrant CR.
-
The Kuadrant operator should only reconcile the spec that is given.
-This would mean if the user states the number of replicas to be used in one of the components only the replica field for that component should be reconciled.
-As the other fields would be blank at this stage, blank fields would not be reconciled to the component CR.
-By this behaviour a few things are being achieved.
-Component controllers define the defaults to be used in the components.
-Optional fields in the component CRs never get set with blank values.
-Blank values in the component CR could override the defaults of the components causing unexpected behaviour.
-Existing Kuadrant users may already have custom fields set in the component CRs.
-By only reconciling the fields set in the kuadrant CR this allows time for a user to migrate their custom configuration from the component CR to the Kuadrant CR.
There are a number of fields in both Authorino and Limitador that are not reconciled.
-Reasons for doing this are:
-
It is better to start with a sub set of features and expand to include more at a later date.
-Removing feature support is far harder than adding it.
-
There are four classifications the unreconciled fields fail into.
-- Deprecated, fields that are deprecated and/or have plans to be removed from the spec in the future.
-- Unsupported, the features would have hard coded or expected defaults in the Kuadrant operator.
-Work would be required to all the custom configurations options.
-- Dev/Testing focused, features that should only be used during development & testing and not recommended for production.
-The defaults would for the fields are the production recommendations.
-- Reconciled by others, this mostly affects Limitador as the deployment configuration and runtime configuration are in the same CR.
-In the case of Kuadrant the runtime configuration for Limitador is added via the RateLimitingPolicy CR.
As the Kuadrant CR spec will be a sub set of the features that can be configured in the sub components spec, extra maintenances will be required to ensure specs are in sync.
-
New features of a component will not be accessible in Kuadrant initially.
-This is both a pro and a con.
-
Documentation becomes harder, as the sub component should be documenting their own features but in Kuadrant the user does not configure the feature in sub component.
-This has the risk of confusing new users.
Why is this design the best in the space of possible designs?
-
What other designs have been considered and what is the rationale for not choosing them?
-
What is the impact of not doing this?
-
-
One alternative that was being looked at was allowing the user to bring their own Limitador instances by stating which Limitador CR Kuadrant should use.
-A major point of issue with this approach was knowing what limits the user had configured and what limits Kuadrant configured.
-Sharing global counters is a valid reason to want to share Limitador instances.
-How ever it this case Limitador would not be using one replica and therefore would have a back-end storage configured.
-It is the back-end storage that needs to be shared across instances.
-This can be done with adding the configuration in the Kuadrant CR.
Discuss prior art, both the good and the bad, in relation to this proposal.
-A few examples of what this can include are:
-
-
Does another project have a similar feature?
-
What can be learned from it? What's good? What's less optimal?
-
Papers: Are there any published papers or great posts that discuss this? If you have some relevant papers to refer to, this can serve as a more detailed theoretical background.
-
-
This section is intended to encourage you as an author to think about the lessons from other tentatives - successful or not, provide readers of your RFC with a fuller picture.
-
Note that while precedent set by other projects is some motivation, it does not on its own motivate an RFC.
What parts of the design do you expect to resolve through the RFC process before this gets merged?
-
What parts of the design do you expect to resolve through the implementation of this feature before stabilization?
-
What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC?
-
-
-
-
Is there a need to add validation on the configuration?
-
If a valid configuration is add to the Kuadrant CR and this configuration is pass to the sub components CR but there is a error trying to setting up the configuration.
-How is this error reported back to the user?
-An example of this is configuring Redis as the back-end in Limitador, this requires stating the name and namespace of a configmap.
-The Limitador CR will have an error if the configmap does not exist and as the user only configures the Kuadrant CR this error may go unnoticed.
-This is only one example but there is a need for good error reporting back to the user, where they would expect to see the error.
Think about what the natural extension and evolution of your proposal would be and how it would affect the platform and project as a whole. Try to use this section as a tool to further consider all possible interactions with the project and its components in your proposal. Also consider how this all fits into the roadmap for the project and of the relevant sub-team.
-
This is also a good place to "dump ideas", if they are out of scope for the RFC you are writing but otherwise related.
-
Note that having something written down in the future-possibilities section is not a reason to accept the current or a future RFC; such notes should be in the section on motivation or rationale in this or subsequent RFCs. The section merely provides additional information.
-
-
The implementation stated here allows the user to state spec fields in the component CRs or the Kuadrant CR (Kuadrant CR overrides the component CRs).
-A future possibility would be to warn the user if they add configuration to the components CRs that would get overridden if the same spec fields are configured in the Kuadrant CR.
The ability for the Multicluster Gateway Controller to sync policies defined in
-the hub cluster downstream to the spoke clusters, therefore allowing all policies
-to be defined in the same place. These policies will be reconciled by the downstream
-policy controller(s).
Policy: When refering to a Policy, this document is refering to a Gateway API
- policy as defined in the Policy Attachment Model. The Multicluster Gateway Controller
- relies on OCM as a Multicluster solution, which defines its own unrelated
- set of Policies and Policy Framework. Unless explicitely mentioned, this document
- refers to Policies as Gateway API Policies.
-
-
-
Policy overriding: The concept of policy overriding is mentioned in this document. It refers to the proposed ability of the downstream Gateway implementation to prioritise downstream Policies against synced Policies in case of conflicts.
Currently, Kuadrant's support for the Policy Attachment Model can be divided in
-two categories:
-* Policies targeting the Multicluster Gateway, defined in the hub cluster and
- reconciled by the Multicluster Gateway Controller
-* Policies targeting the downstream Gateway, defined in the spoke clusters and
- reconciled by the downstream Gateway controllers.
-
In a realistic multicluster scenario where multiple spoke clusters are present, the management of these policies can become tedious and error-prone, as policies have
-to be defined in the hub cluster, as well as replicated in the multiple spoke clusters.
-
As Kuadrant users:
-* Gateway-admin has a set of homogeneous clusters and needs to apply per cluster rate limits across the entire set.
-* Platform-admin with a set of clusters with rate limits applied needs to change rate limit for one particular cluster.
The policy sync feature will allow a gateway-admin to configure, via GatewayClass
-parameters, a set of Policy GVRs to be synced by the Multicluster Gateway Controller.
-
The policiesToSync field in the parameters defines those GVRs. For example, in
-order to configure the controller to sync AuthPolicies:
The support for resources that the controller can sync is limited by the following:
-* The controller ServiceAccount must have permission to watch, list, and get the
- resource to be synced
-* The resource must implement the Policy schema:
- * Have a .spec.targetRef field
-
When a Policy is configured to be synced in a GatewayClass, the Multicluster
-Gateway Controller starts watching events on the resources, and propagates changes
-by placing the policy in the spoke clusters, with the following mutations:
-* The TargetRef of the policy is changed to reference the downstream Gateway
-* The kuadrant.io/policy-synced annotation is set
-
The upstream policy is annotated with a reference to the name and namespace
-of the downstream policies:
-
The Multicluster Gateway Controller reconciles parameters referenced by the
-GatewayClass of a Gateway. A new field is added to the parameters that allows
-the configuration of a set of GVRs of Policies to be synced.
-
The GatewayClass reconciler validates that:
-* The GVRs reference existing resource definitions
-* The GVRs reference resources that implement the Policy schema.
-
Validation failures are reported as part of the status of the GatewayClass
-
The Gateway reconciler sets up dynamic watches to react to events on the configured
-Policies, calling the PolicySyncer component with the updated Policy as well
-as the associated Gateway.
The PolicySyncer component is in charge of reconciling Policy watch events to
-apply the necessary changes and place the Policies in the spoke clusters.
-
This component is injected in the event source and called when a change is made
-to a hub Policy that has been configured to be synced.
-
The PolicySyncer implementation uses OCM ManifestWorks to place the policies in
-the spoke clusters. Through the ManifestWorks, OCM allows to:
-* Place the Policy in each spoke cluster
-* Report the desired status back to the hub using JSON feedback rules
In order to avoid conflict with Policies created directly in the spoke clusters,
-a hierarchy must be defined to prioritise those Policies.
-
The controller will set the kuadrant.io/policy-synced annotation on the policy
-when placing it in the spoke cluster.
-
The Kuadrant operator will be aware of the presence of this annotation, and, in case
-of conflicts, override Policies that contain this annotation. When a policy is
-overriden due to conflicts, the Enforced status will be set to False, with
-the reason being Overriden and a human readable message explaining the reason
-why the policy was overriden. See Policy Status RFC
In order for a Policy to be supported for syncing, the MGC must have permissions
-to watch/list/get the resource, and the implementation of the downstream Gateway
-controller must be aware of the policy-synced annotation.
Different technology stacks are available to sync resources across clusters. However, adoption of these technologies for the purpose of the goal this RFC intends to achieve, implies adding another dependency to the current stack, with the cost of added complexity and maintainance effort.
-
The MGC currently uses OCM to place Gateways across clusters. Relying on OCM for the purpose of placing Policies is the most straightforward alternative from a design and implementation point of view.
Gateway-admins will have no centralized system for handling spoke-level policies targeting a gateway created there from the hub.
-
OCMs Policy Framework will not be used to complete this objective:¶
-
OCMs Policy Framework is a system designed to make assertions about the state of a spoke, and potentially take actions based on that state, as such it is not a suitable replacement for manifestworks in the case of syncing resources to a spoke.
-
Potential migration from ManifestWorks to ManifestWorkReplicaSets¶
-
ManifestWorkPeplicaSets may be a future improvement that the MGC could support
-to simplify the placement of related resources, but beyond the scope of this RFC.
While the controller can assume common status fields among the Policies that it
-syncs, there might be a scenario where certain policies use custom status fields
-that are not handled by the controller. In order to support this, two alternatives
-are identified:
-
-
-
Configurable rules.
-
An extra field is added in the GatewayClass params that configures the policies
-to sync, to specify custom fields that the controller must propagate back from
-the spokes to the hub.
-
-
-
Hard-coded support.
-
The PolicySync component can identify the Policy type and select which extra
-status fields are propagated
If OCMs Policy Framework is updated to enable syncing of resources status back to the hub, it could be an opportunity to refactor the MGC to use this framework in place of the current approach of creating ManifestWorks directly.
-
This system could mutate over time to dynamically sync more CRDs than policies to spoke clusters.
Kuadrant is a set of components whose artifacts are built and delivered independently. This RFC aims to define every
-aspect of the event of releasing a new version of the whole, in terms of versioning, cadence, communication, channels,
-handover to other teams, etc.
At the time being, there's no clear process nor guidelines to follow when releasing a new version of Kuadrant, which
-leads to confusion and lack of transparency. We are currently relying on internal communication and certain people
-in charge of the release process, which is not ideal.
First, we need to define what releasing Kuadrant means, in a clear and transparent way that communicates to the community
-what's happening and what to expect. The Kuadrant suite is composed of several components, each of them with its own
-set of artifacts and versioning scheme. Defining the release process of the whole suite is a complex task, and it's
-not only about the technical details of releasing the components, but also about the communication and transparency
-with the community, the definition of the frequency of the releases, and when it's ready to be handover to other teams like
-QA. This section aims to provide guidelines for the different aspects of the release process.
DNS Operator: A Kubernetes Operator to manage DNS in single and multi-cluster
-environments.
-
Kuadrant Operator: The Operator to install and manage the lifecycle
- of the Kuadrant components deployments. Example alerts and dashboards are also included as optional.
-
kuadrantctl: A CLI tool for managing Kuadrant configurations and resources.
-
-
Each of them needs to be versioned independently, and the versioning scheme should follow Semantic Versioning.
-At the time of cutting a release for any of them, it's important to keep in mind what section of the version to bump,
-given a version number MAJOR.MINOR.PATCH, increment the:
-
-
MAJOR version when you make incompatible API changes
-
MINOR version when you add functionality in a backward compatible manner
-
PATCH version when you make backward compatible bug fixes
-
-
Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.
-
A more detailed explanation of the versioning scheme can be found in the Semantic Versioning website.
-
By releasing a new version of Kuadrant, we mean releasing the set of components with their corresponding semantic versioning,
-some of them maybe freshly released, or others still using versioning from the previous one, and being the version of the
-Kuadrant Operator the one that defines the version of the whole suite.
Probably the most important and currently missing step in the release process is the green flagging from the Quality Assurance
-(QA) team. The QA team is responsible for testing the different components of the Kuadrant suite, and they need to
-be aware of the new version of the suite that is going to be released, what are the changes that are included, bug fixes
-and new features in order they can plan their testing processes accordingly. This check is not meant to be a fully fledged
-assessment from the QA team when it's handover to them, it's aimed to not take more than 1-2 days, and ideally expected
-to be fully automated. This step will happen once the release candidate has no PRs pending to be merged, and it has been
-tested by the Engineering team. The QA team should work closely to the engineering throughout the process, both teams aiming
-for zero handover time and continuous delivery mindset, so immediate testing can be triggered on release candidates once
-handed over. This process should happen without the need of formal communication between the teams or any overhead in
-general, but by keeping constant synergy between quality and product engineering instead.
-
There is an ideal time to hand over to the QA team for testing, especially since we are using GitHub for orchestration,
-we could briefly define it in the following steps:
-
-
Complete Development Work: The engineering team completes their work included in the milestone.
-
Create Release Candidate: The engineering team creates Release Candidate builds and manifests for all components required
-for the release
-
Flagging/Testing: The QA team do the actual assertion/testing of the release candidate, checking for any obvious bugs or issues.
-Then QA reports all the bugs as GitHub issues and communicates testing status back publicly on Slack and/or email.
-
Iterate: Based on the feedback from the QA team, the Engineering team makes any necessary adjustments and repeats the
-process until the release candidate is deemed ready for production.
-
Publish Release: Once QA communicates that the testing has been successfully finished, the engineering team will publish
-the release both on Github and in the corresponding registries, updates documentation for the new release, and communicates
-it to all channels specified in Communication section.
Once the project is stable enough, and its adoption increases, the community will be expecting a certain degree of
-commitment from the maintainers, and that includes a regular release cadence. The frequency of the releases of the
-different components could vary depending on the particular component needs. However, the Kuadrant Operator
-it's been discussed in the past that it should be released every 3-4 weeks initially, including the latest released version
-of every component in the suite. There's another RFC that focuses on the actual frequency of each component, one could
-refer to the Kuadrant Release Cadence RFC.
-
There are a few reasons for this:
-
-
Delivering Unparalleled Value to Users: Regular releases can provide users with regular updates and improvements.
- These updates can include new features and essential bug fixes, thus enhancing the overall value delivered to the users.
-
Resource Optimization: By releasing software at regular intervals, teams can align their activities with
- available resources and environments, ensuring optimal utilization. This leads to increased efficiency in the deployment
- process and reduces the risk of resource wastage.
-
Risk Management: Regular releases can help identify and fix issues early, reducing the risk of major failures that could
- affect users.
-
Feedback Cycle: Regular releases allow for quicker feedback cycles. This means that any issues or improvements
- identified by users can be addressed promptly, leading to a more refined product over time.
-
Synchronization: Regular releases can help synchronize work across different teams or departments, creating a more
- reliable, dependable solution development and delivery process.
-
Reduced Complexity: Managing a smaller number of releases can reduce complexity. For example, having many different
- releases out in the field can lead to confusion and management overhead.
-
-
By committing to a release cadence, software projects can benefit from improved efficiency, risk management, faster feedback
-cycles, synchronization, and reduced complexity.
Every component in Kuadrant has its own repository, and the source code is hosted in GitHub, mentioned in the previous
-section. However, the images built and manifests generated are hosted in different registries, depending on the
-component. The following table shows the different registries used by each component:
It's important to note that keeping the documentation up to date is a responsibility of the component maintainers, and
-it needs to be done before releasing a new version of the component. The importance of keeping a clear and up-to-date
-documentation is crucial for the success of the project.
-
The documentation for the Kuadrant suite is compiled and available on the Kuadrant website. One
-can find the source of the documentation within each component repository, in the docs directory. However, making this
-information available on the website is a manual process, and should be done by the maintainers of the project. The
-process of updating the documentation is simple and consists of the following steps:
-
-
Update the documentation in the corresponding component repository.
-
Follow the instruction in https://github.com/Kuadrant/docs.kuadrant.io/
-to update the Docs pointers to the tag or branch of the component repository that contains the updated documentation.
-
Once the changes are merged to main, the workflow that updates the website will be triggered, and the documentation
-will be updated.
-
If for some reason it's needed to trigger the workflow manually, one can do it from the GitHub Actions tab in the
-docs.kuadrant.io (Actions > ci > Run Workflow).
Another important aspect of releasing a new version of the Kuadrant suite is the communication with the community and
-other teams within the organization. A few examples of the communication channels that need to be updated are:
There's been an organically grown process for releasing new versions of the Kuadrant suite, which is not documented and
-it's been changing over time. However, there are some documentation for some of the components, worth mentioning:
This is a proposal for extending the Kuadrant Policy APIs to fully support use cases of Defaults & Overrides (D/O) for Inherited Policies, including the base use cases of full default and full override, and more specific nuances that involve merging individual policy rules (as defaults or overrides), declaring constraints and unsetting defaults.
As of Kuadrant Operator v0.6.0, Kuadrant policy resources that have hierarchical effect across the tree of network objects (Gateway, HTTPRoute), or what is known as Inherited Policies, provide only limited support for setting defaults and no support for overrides at all.
-
The above is notably the case of the AuthPolicy and the RateLimitPolicy v1beta2 APIs, shipped with the aforementioned version of Kuadrant. These kinds of policies can be attached to Gateways or to HTTPRoutes, with cascading effects through the hierarchy that result in one effective policy per gateway-route combination. This effective policy is either the policy attached to the Gateway or, if present, the one attached to the HTTRoute, thus conforming with a strict case of implicit defaults set at the level of the gateway.
-
Enhancing the Kuadrant Inherited Policy CRDs, so the corresponding policy instances can declare defaults and overrides stanzas, is imperative:
-1. to provide full support for D/O along the lines proposed by GEP-713 (to be superseded by GEP-26491) of the Kubernetes Gateway API special group (base use cases);
-2. to extend D/O support to other derivative cases, learnt to be just as important for platform engineers and app developers who require more granular policy interaction on top of the base cases;
-3. to support more sophisticated hierarchies with other kinds of network objects and/or multiples policies targetting at the same level of the hierarchy (possibly, in the future.)
The base use cases for Defaults & Overrides (D/O) are:
-- Defaults (D): policies declared lower in the hierarchy supersede ones set (as "defaults") at a higher level, or "more specific beats less specific"
-- Overrides (O): policies declared higher in the hierarchy (as "overrides") supersede ones set at the lower levels, or "less specific beats more specific"
-
The base cases are expanded with the following additional derivative cases and concepts:
-- Merged defaults (DR): "higher" default policy rules that are merged into more specific "lower" policies (as opposed to an atomic less specific set of rules that is activated only when another more specific one is absent)
-- Merged overrides (OR): "higher" override policy rules that are merged into more specific "lower" policies (as opposed to an atomic less specific set of rules that is activated fully replacing another more specific one that is present)
-- Constraints (C): specialization of an override that, rather than declaring concrete values, specify higher level constraints (e.g., min value, max value, enums) for lower level values, with the semantics of "clipping" lower level values so they are enforced, in an override fashion, to be the boundaries dictated by the constraints; typically employed for constraining numeric values and regular patterns (e.g. limited sets)
-- Unsetting (U): specialization that completes a merge default use case by allowing lower level policies to disable ("unset") individual defaults set a higher level (as opposed to superseding those defaults with actual, more specific, policy rules with proper meaning)
-
Together, these concepts relate to solve the following user stories:
-
-
-
-
User story
-
Group
-
Unique ID
-
-
-
-
-
As a Platform Engineer, when configuring a Gateway, I want to set a default policy for all routes linked to my Gateway, that can be fully replaced with more specific ones(*).
-
D
-
gateway-default-policy
-
-
-
As a Platform Engineer, when configuring a Gateway, I want to set default policy rules (parts of a policy) for all routes linked to my Gateway, that can be individually replaced and/or expanded by more specific rules(*).
-
DR
-
gateway-default-policy-rule
-
-
-
As a Platform Engineer, when defining a policy that configures a Gateway, I want to set constraints (e.g. minimum/maximum value, enumerated options, etc) for more specific policy rules that are declared(*) with the purpose of replacing the defaults I set for the routes linked to my Gateway.
-
C
-
policy-constraints
-
-
-
As a Platform Engineer, when configuring a Gateway, I want to set a policy for all routes linked to my Gateway, that cannot be replaced nor expanded by more specific ones(*).
-
O
-
gateway-override-policy
-
-
-
As a Platform Engineer, when configuring a Gateway, I want to set policy rules (parts of a policy) for all routes linked to my Gateway, that cannot be individually replaced by more specific ones(*), but only expanded with additional more specific rules(*).
-
OR
-
gateway-override-policy-rule
-
-
-
As an Application Developer, when managing an application, I want to set a policy for my application, that fully replaces any default policy that may exist for the application at the level of the Gateway, without having to know about the existence of the default policy.
-
D
-
route-replace-policy
-
-
-
As an Application Developer, when managing an application, I want to expand a default set of policy rules set for my application at the level of the gateway, without having to refer to those existing rules by name.
-
D/O
-
route-add-policy-rule
-
-
-
As an Application Developer, when managing an application, I want to unset an individual default rule set for my application at the level of the gateway.
-
U
-
route-unset-policy-rule
-
-
-
-
(*) declared in the past or in the future, by myself or any other authorized user.
-
The interactive nature of setting policies at levels in the hierarchy and by different personas, make that the following additional user stories arise. These are stories here grouped under the Observability (Ob) aspect of D/O, but referred to as well in relation to the "Discoverability Problem" described by Gateway API.
-
-
-
-
User story
-
Group
-
Unique ID
-
-
-
-
-
As one who has read access to Kuadrant policies, I want to view the effective policy enforced at the traffic routed to an application, considering all active defaults and overrides at different policies(*).
-
Ob
-
view-effective-policy
-
-
-
As a Platform Engineer, I want to view all default policy rules that may have been replaced by more specific ones(*).
-
Ob
-
view-policy-rule-status
-
-
-
As a Policy Manager, I want to view all gateways and/or routes whose traffic is subject to enforcement of a particular policy rule referred by name.
-
Ob
-
view-policy-rule-reach
-
-
-
-
(*) declared in the past or in the future, by myself or any other authorized user.
Writing a Kuadrant policy enabled for Defaults & Overrides (D/O), to be attached to a network object, involves declaring the following fields at the first level of the spec:
defaults: a block of default policy rules with further specification of a strategy (atomic set of rules or individual rules to be merged into lower policies), and optional conditions for applying the defaults down through the hierarchy
-
overrides: a block of override policy rules with further specification of a strategy (atomic set of rules or individual rules to be merged into lower policies), and optional conditions for applying the overrides down through the hierarchy
-
the bare policy rules block without further qualification as a default or override set of rules – e.g. the rules field in a Kuadrant AuthPolicy, the limits field in a RateLimitPolicy.
-
-
Between the following mutually exclusive options, either one or the other shall be used in a policy:
-1. defaults and/or overrides blocks; or
-2. the bare set of policy rules (without further qualification as neither defaults nor overrides.)
-
In case the bare set of policy rules is used, it is treated implicitly as a block of defaults.
-
Supporting specifying the bare set of policy rules at the first level of the spec, alternatively to the defaults and overrides blocks, is a strategy that aims to provide:
-1. more natural usability, especially for those who write policies attached to the lowest level of the hierarchy supported; as well as
-2. backward compatibility for policies that did not support explicit D/O and later on have moved to doing so.
A policy that does not specify D/O fields (defaults, overrides) is a policy that declares an intent.
-
One who writes a policy without specifying defaults or overrides, but only the bare set of policy rules, may feel like declaring a Direct Policy.
-Depending on the state of other policies indirectly affecting the same object or not, the final outcome can be the same as writing a direct policy.
-This is especially true when the policy that declares the intent targets an object whose kind is the lowest kind accepted by Kuadrant in the hierarchy of network resources, and there are no other policies with lower precedence.
-
Nevertheless, because other policies can affect the final behavior of the target (e.g. by injecting defaults, by overriding rules, by adding more definitions beneath), policies that simply declare an intent, conceptually, are still Inherited Policies.
-
Compared to the inherited policy that misses D/O blocks, these other policies affecting the behavior may be declared:
-- at higher levels in the hierarchy,
-- at lower levels in hierarchy, or even
-- at the same level in the hierarchy but happening to have lower precedence (if such case is allowed by the kind of policy.)
-
At any time, any one of these policies can be created and therefore the final behavior of a target should never be assumed to be equivalent to the intent declared by any individual policy in particular, but always collectively determined by the combination of all intents, defaults and overrides from all inherited policies affecting the target.
If a Policy can be used as an Inherited Policy, it MUST be treated as an Inherited Policy, regardless of whether a specific instance of the Policy is only affecting a single object.
-
-
An inherited policy that simply declares an intent (i.e. without specifying D/O) will be treated as a policy that implicitly declares an atomic set of defaults, whether the policy targets higher levels in the hierarchy or lower ones.
-In the absence of any other conflicting policy affecting the same target, the behavior equals the defaults which equal the intent.
A policy that specifies D/O fields (defaults, overrides) is a policy explicitly declared to modify an intent.
-
Without any other policy with lower precedence, there is no special meaning in choosing whether defaults or overrides in a inherited policy that targets an object whose kind is the lowest kind accepted by Kuadrant in the hierarchy of network resources.
-The sets of rules specified in these policies affect indistinctively the targeted objects, regardless of how they are qualified.
-
However, because other policies may ocasionally be declared with lower precedence (i.e. targeting lower levels in the hierarchy or due to ordering, see Conflict Resolution), one who declares a policy to modify an intent must carefuly choose between defaults and/or overrides blocks to organize the policy rules, regardless if the targeted object is of a kind that is the lowest kind in the hierarchy of network resources accepted by Kuadrant.
-
Even in the cases where no more than one policy of a kind is allowed to target a same object (1:1 relationship) and thus there should never exist two policies affecting a target from a same level of the hierarchy simultaneaously (or equivalently a policy with lower precedence than another, both at the lowest level of the hierarchy), users must assume that this constraint may change (i.e. N:1 relationship between policies of a kind and target become allowed.)
-
In all cases, defaults and overrides must be used with the semantics of declaring rules that modify an intent.
-- When an intent does not specify a rule for which there is a higher default declared, the default modifies the intent by setting the value specified by the default.
-- For an intent that whether specifies or omits a rule for which there is a higher override declared, the override modifies the intent by setting the value specified by the override.
All Custom Resource Definitions (CRDs) that define a Kuadrant inherited policy must be labeled gateway.networking.k8s.io/policy: inherited.
-
Users can rely on the presence of that label to identify policy kinds whose instances are treated as inhertied policies.
-
In some exceptional cases, there may be kinds of Kuadrant policies that do not specify defaults and overrides blocks, but that are still labeled as inherited policy kinds.
-Instances of these kinds of policies implicitly declare an atomic sets of defaults, similarly to described in Inherited Policies that declare an intent.
The above is a proper Inherited Policy that sets a default atomic set of auth rules that will be set at lower objects in case those lower object do not have policies attached of their own at all.
-
The following is a sligthly different example that defines auth rules that will be individually merged into lower objects, evaluated one by one if already defined at the "lower" (more specific) level and therefore should take precedence, or if otherwise is missing at the lower level and therefore the default should be activated.
There are 2 supported strategies for applying proper Inherited Policies down to the lower levels of the herarchy:
-- Atomic policy rules: the bare set of policy rules in a defaults or overrides block is applied as an atomic piece; i.e., a lower object than the target of the policy, that is evaluated to be potentially affected by the policy, also has an atomic set of rules if another policy is attached to this object, therefore either the entire set of rules declared by the higher (less specific) policy is taken or the entire set of rules declared by the lower (more specific) policy is taken (depending if it's defaults or overrides), but the two sets are never merged into one.
-- Merged policy rules: each individual policy rule within a defaults or overrides block is compared one to one against lower level policy rules and, when they conflict (i.e. have the same key with different values), either one or the other (more specific or less specific) is taken (depending if it's defaults or overrides), in a way that the final effective policy is a merge between the two policies.
-
Each block of defaults and overrides must specify a strategy field whose value is set to either atomic or merge. If omitted, atomic is assumed.
Atomic versus merge strategies, as a specification of the defaults and overrides blocks, imply that there are only two levels of granularity for comparing policies vis-a-vis.
-
-
-
atomic means that the level of granularity is the entire set of policy rules within the defaults or overrides block. I.e., the policy is atomic, or, equivalently, the final effective policy will be either one indivisible ("atomic") set of rules ("policy") or the other.
-
-
-
For the merge strategy, on the other hand, the granularity is of each named policy rule, where the name of the policy rule is the key and the value is an atomic object that specifies that policy rule. The final effective policy will be a merge of two policies.
When two policies are compared to compute a so-called Effective Policy out of their sets of policy rules and given default or override semantics, plus specified atomic or merge strategies, the following matrix applies:
-
-
-
-
-
Atomic (entire sets of rules)
-
Merge (individual policy rules at a given granularity)
-
-
-
-
-
Defaults
-
More specific entire set of rules beats less specific entire set of rules → takes all the rules from the lower policy
-
More specific individual policy rules beat less specific individual set of rules → compare one by one each pair of policy rules and take the lower one if they conflict
-
-
-
Overrides
-
Less specific entire set of rules beats more specific entire set of rules → takes all the rules from the higher policy
-
Less specific individual policy rules beat more specific individual set of rules → compare one by one each pair of policy rules and take the higher one if they conflict
-
-
-
-
The order of the policies, from less specific (or "higher") to more specific (or "lower), is determined according to the Gateway API hierarchy of network resources, based on the kind of the object targeted by the policy. The policy that sets higher in the hierarchy dictates the strategy to be applied.
-
For a more detailed reference, including how to resolve conflicts in case of policies targeting objects at the same level, see GEP-713's section Hierarchy and Conflict Resolution.
In some cases, it may be desirable to be able to unset, at a lower policy, a merged default that is inherited from a higher one. In fact, some inherited defaults could be harmful to an application, at the same time as they are unfeasible to remove from scope for all applications altogether, and therefore require an exception.
-
Unsetting defaults via specification at lower level policies provides users who own policy rules at different levels of the hirarchy the option of not having to coordinate those exceptions "offline", nor having to accept the addition of special cases (conditions) at the higher level to exempt only specific lower policies from being affected by a particular default, which otherwise would configure a violation of the inheritance pattern, as well as an imposition of additional cognitive complexity for one who reads a higher policy with too many conditions.
-
Instead, users should continue to be able to declare their intents through policies, and redeem an entitlement to unset unapplicable defaults, without any leakage of lower level details upwards at the higher policies.
-
The option of unsetting inherited defaults is presented as part of the volition implied by the inheritance of policy rules, which are tipically specified for the more general case (e.g. at the level of a gateway, for all routes), though not necessarily applicable for all special cases beneath. If enabled, this feature helps disambiguate the concept of "default", which should not be understood strictly as the option to set values that protect the system in case of lack of specialisation. Rather, by its property of volition and changeability. I.e., by definition, every default policy rule is opt-out and specifies a value that is modifiable.
-
In constrast, a policy rule that is neither opt-out nor modifiable better fits the definition of an override. While a policy rule that is not opt-out, nor it sets a concrete default value to be enforced in the lack of specialisation, defines a requirement.
-
Finally, for the use case where users want to set defaults that cannot be unset (though still modifable), the very feature of unsetting defaults itself should be configurable, at least at the level of the system. This can be achieved with feature switches and policy validation, including backed by the cluster's RBAC if needed.
-
The capability of unsetting inherited defaults from an effective policy can be identified by the presence of the spec.unset field in a policy. The value is a list of default named policy rules to be unset.
Users should be able to specify conditions for applying their blocks of defaults and overrides. These conditions aim to support exceptional cases where the blocks cannot be simply applied downwards, but rather depend on specifics found in the lower policies, while still defined in generic terms – as opposed to conditions that leak details of individual lower policies upwards.
-
Between a higher and a lower set of policy rules, the higher level dictates the conditions for its rules to be applied (either as defaults or as overrides) over the lower level, and never the other way around.
-
D/O conditions are identfied by the presence of the spec.defaults.when or spec.overrides.when fields in a policy. Those should be defined using Common Expression Language (CEL), evaluated in the control plane against the lower level specification that the higher level is being applied to. I.e. self in the CEL expression is the lower policy.
-
A concrete useful application for conditionally enforcing a block of D/O is for specifying constraints for lower values. E.g. if a lower policy tries to set a value on a numeric field that is greater (or lower) than a given threshold, apply an override that sets that field value to equal to the threshold; otherwise, use the value declared by the lower policy.
-
In contrast, an example of trivially redundant application of D/O conditions would be specifying a default block of rules that is only applied when the lower level does not declare a more specific replacement. Since this is natural semantics of a default, one does not have to use conditions for that.
The following sets of examples generalize D/O applications for the presented user stories, regardless of details about specific personas and kinds of targeted resources. They illustrate the expected behavior for different cases involving defaults, overrides, constraints and unsetting.
In all the examples, a Gateway and a HTTPRoute objects are targeted by two policies, and an effective policy is presented highlighting the expected outcome. This poses no harm to generalizations involving same or different kinds of targeted resources, multiples policies targeting a same object, etc.
-
The leftmost YAML is always the "higher" (less specific) policy; the one in the middle, separated from the leftmost one by a "+" sign, is the "lower" (more specific) policy; and the rightmost YAML is the expected Effective Policy.
-
For a complete reference of the order of hierarchy, from least specific to most specific kinds of resources, as well as how to resolve conflicts of hierarchy in case of policies targeting objects at the same level, see Gateway API's Hierarchy definition for Policy Attachment and Conflict Resolution.
-
Examples A - Default policy entirely replaced by another at lower level¶
-
Example A1. A default policy that is replaced entirely if another one is set at a lower level
-
-
Examples B - Default policy rules merged into policies at lower level¶
-
Example B1. A default policy whose rules are merged into other policies at a lower level, where individual default policy rules can be overridden or unset - without conflict
-
-
Example B2. A default policy whose rules are merged into other policies at a lower level, where individual default policy rules can be overridden or unset - with conflict
-
-
Examples C - Override policy entirely replacing other at lower level¶
-
Example C1. An override policy that replaces any other that is set at a lower level entirely
-
-
Examples D - Override policy rules merged into other at lower level¶
-
Example D1. An override policy whose rules are merged into other policies at a lower level, overriding individual policy rules with same identification - without conflict
-
-
Example D2. An override policy whose rules are merged into other policies at a lower level, overriding individual policy rules with same identification - with conflict
-
-
Examples E - Override policy rules setting constraints to other at lower level¶
-
The examples in this section introduce the proposal for a new when field for the defaults and overrides blocks. This field dictates the conditions to be found in a lower policy that would make a higher policy or policy rule to apply, according to the corresponding defaults or overrides semantics and atomic or merge strategy.
-
Combined with a simple case of override policy (see Examples C), the when condition field allows modeling for use cases of setting constraints for lower-level policies.
Example E1. An override policy whose rules set constraints to field values of other policies at a lower level, overriding individual policy values of rules with same identification if those values violate the constraints - lower policy is compliant with the constraint
-
-
Example E2. An override policy whose rules set constraints to field values of other policies at a lower level, overriding individual policy values of rules with same identification if those values violate the constraints - lower level violates the constraint
-
-
Example E3. An override policy whose rules set constraints to field values of other policies at a lower level, overriding individual policy values of rules with same identification if those values violate the constraints - merge granularity problem
-
The following example illustrates the possibly unintended consequences of enforcing D/O at strict levels of granularity, and the flip side of the strategy field offering a closed set of options (atomic, merge).
-
On one hand, the API is simple and straightforward, and there are no deeper side effects to be concerned about, other than at the two levels provided (atomic sets or merged individual policy rules.) On the other hand, this design may require more offline interaction between the actors who manage conflicting policies.
-
-
Examples F - Policy rule that unsets a default from higher level¶
-
The examples in this section introduce a new field unset: []string at the same level as the bare set of policy rules. The value of this field, provided as a list, dictates the default policy rules declared at a higher level to be removed ("unset") from the effective policy, specified by name of the policy rules.
-
Example F1. A policy that unsets a default policy rule set at a higher level
-
-
Example F2. A policy that tries to unset an override policy rule set a higher level
A special condition must be added to every object that is targeted by a Kuadrant inherited policy if the policy's Enforced status condition is True.
-
This special condition to be added to the target object is kuadrant.io/xPolicyAffected, where "xPolicy" is the kind of the inherited policy (e.g. AuthPolicy, RateLimitPolicy.)
-
The possible statuses of an object regarding its sensitivity to one or more inherited policies are:
-
-
-
-
Type
-
Status
-
Reason
-
Message
-
-
-
-
-
xPolicyAffected
-
False
-
"Unaffected"
-
"The object is not affected by any xPolicy"
-
-
-
-
True
-
"Affected"
-
"The object is affected by xPolicy <policy-ns/policy-name>"
-
-
-
-
True
-
"PartiallyAffected"
-
"The following sections of the object are affected by xPolicy <policy-ns/policy-name>: rules.0, rules.2"
-
-
-
-
The presence of the PolicyAffected status condition helps identify that an object is sensitive to one of more policies of a kind, and gives some specifics about the scope of that effect (entire object or selected sections.)
-In many cases, this should be enough for inferring the actual policy rules being enforced for that object.
-
For other cases where any of the following situations hold, a more detailed view of the final Effective Policy must be provided to the user:
-- If the rules of the policy cannot be inferred by the name of the policy and/or the user lacks permission to read the policy object;
-- If the object is affected by more than one policy.
-
To help visualize the effective policy for a given target object in that situation, at least one of the following options must be provided to the user:
-1. A read-only EffectivePolicy custom resource, defined for each kind of inherited policy, and with an instance created for each affected object, that is reconciled and updated by the policy controller.
-2. A HTTP endpoint of the policy controller that users can consume to read the effective policy.
-3. A CLI tool that offers a command that queries the cluster and returns the effective policy – either by leveraging any of the methods above or computing the effective policy "on-the-fly" in the same fashion as the policy controller does.
The following diagrams are a high level model to guide the process of applying a set of policies of a kind for a given Gateway object, where the Gateway object is considered the root of a hierarchy, and for all objects beneath, being the xRoute objects the leaves of the hierarchical tree.
-
As presented, policies can target either Gateways of route objects (HTTPRoutes, GRPCRoutes), with no restriction regarding the number of policies of a kind that target a same particular object. I.e. N:1 relationship allowed. Without any loss of generality, 1:1 relationship between policies of a kind and targeted objects can be imposed if preferred as a measure to initially reduce the blast of information for the user and corresponding cognitive load.
-
Apply policies to a Gateway (root object) and all objects beneath¶
-
%%{ init: { "theme": "neutral" } }%%
-flowchart LR
- start([For a Gateway <i>g</i><br>and policy kind <i>pk</i>]) -->
- list-routes[List all routes<br>accepted by <i>g</i> as <i>R</i>] -->
- apply-policies-for-r
- subgraph for-each-route[For each <i>r in R</i>]
- apply-policies-for-r[[Apply policies<br>of kind <i>pk</i><br>scoped for <i>r</i>]] -->
- apply-policies-for-r
- end
- for-each-route -->
- build-virtual-route[Build a virtual route <i>vr</i><br>with all route rules not<br>target by any policy] -->
- apply-policies-for-vr[[Apply policies<br>of kind <i>pk</i><br>scoped for <i>vr</i>]] -->
- finish(((END)))
-
This section proposes a possible path for the implementation of this RFC for Kuadrant's existing kinds of policies that are affected by D/O – notably AuthPolicy and RateLimitPolicy.
-
The path is divided in 3 tiers that could be delivered in steps, additionaly to a series of enhancements & refactoring.
Each block of defaults and overrides specify a field strategy: atomic | merge, with atomic assumed if the field is omitted.
-
All the examples in the RFC are based on this design for the API spec.
-
Some of the implications of the design are explained in the section Atomic vs. individually merged policy rules, with highlights to the support for specifying the level of atomicity of the rules in the policy based on only 2 granularities – entire set of policy rules (atomic) or to the level of each named policy rule (merge.)
-
-
-
-
✅ Pros
-
❌ Cons
-
-
-
-
-
-
-
Same schema as a normal policy without D/O
-
Declarative
-
Safe against "unmergeable objects" (e.g. two rules declaring different one-of options)
-
Strong types
-
Extensible (by adding more fields, e.g.: to support unsetting defaults)
-
Easy to learn
-
-
-
-
-
2 levels of granularity only – either all (‘atomic’) or policy rule (‘merge’)
-
1 granularity declaration per D/O block → declaring both ‘atomic’ and ‘merge’ simultaneously requires 2 separate policies targeting the same object
-
-
-
-
-
-
-
The design option based on the strategy field is the RECOMMENDED design for the implementation of Kuadrant Policies enabled for D/O. This is due to the pros above, plus the fact that this design can evolve to other, more versatile forms, such as granularity field, when conditions or CEL functions, in the future, while the opposite would be harder to achieve.
Each block of defaults and overrides would specify a granularity field, set to a numeric integer value that describes which level of the policy spec, from the root of the set of policy rules until that number of levels down, to treat as the key, and the rest as the atomic value.
-
Example:
-
kind:AuthPolicy
-metadata:
-name:gw-policy
-spec:
-targetRef:
-kind:Gateway
-defaults:
-rules:
-authentication:
-"a":{…}
-authorization:
-"b":{…}
-granularity:0# the entire spec ("rules") is an atomic value
-overrides:
-rules:
-metadata:
-"c":{…}
-response:
-"d":{…}
-granularity:2# each policy rule ("c", "d") is an atomic value
-
Unlimited levels of granularity (values can be pointed as atomic at any level)
-
-
-
-
-
1 granularity declaration per D/O block → N levels simultaneously require N policies
-
Granularity specified as a number - user needs to count the levels
-
Setting a deep level of granularity can cause merging "unmergeable objects"
-
-
-
-
-
-
-
Design option: when conditions (at any level of the spec)¶
-
Inspired by the extension of the API for D/O with an additional when field (see Examples E), this design alternative would use the presence of this field to signal the granularity of the atomic operation of default or override.
As many granularity declarations per D/O block as complex objects in the policy
-
Granularity specified “in-place”
-
-
-
-
-
Setting a deep level of granularity can cause merging "unmergeable objects"
-
Implementation nightmare - hard to define the API from existing types
-
-
-
-
-
-
-
Design option: CEL functions (at any level of the spec)¶
-
This design option leans on the power of Common Expression Language (CEL), extrapolating the design alternative with when conditions beyond declaring a CEL expression just to determine if a statically declared value should apply. Rather, it proposes the use of CEL functions that outputs the value to default to or to ovrride with, taking the conflicting "lower" value as input, with or without a condition as part of the CEL expression. The value of a key set to a CEL function indicates the level of granularity of the D/O operation.
A more radical alternative considered consisted of defining defaults and overrides blocks whose schemas would not match the ones of a normal policy without D/O. Instead, these blocks would consist of simple key-value pairs, where the keys specify the paths in an affected policy where to apply the value atomically.
Other than the primitive support only for implicit atomic defaults provided by Kuadrant for the AuthPolicy and RateLimitPolicy, other real-life implementations of D/O along the lines proposed by Gateway API are currently unknown.
-
Some orientative examples provided in:
-- GEP-2649 - search for "CDNCachingPolicy" as well as "Merging into existing spec fields";
-- gwctl effective policy calculation for inherited policies - see policy manager's merge test cases.
A use case often described in association with D/O is the one for declaring policy requirements. These are high level policies that declare requirements to be fulfilled by more specific (lower level) policies without specifying concrete default or override values nor constraints. E.g.: "an authentication policy must be enforced, but none is provided by default."
-
A typical generic policy requirement user story is:
-
-
As a Platform Engineer, when configuring a Gateway, I want to set policy requirements to be fulfilled by one who manages an application/route linked to my Gateway, so all interested parties, including myself, can be aware of applications deployed to the cluster that lack a particular policy protection being enforced.
-
-
Policy requirements as here described are out of scope of this RFC.
-
We believe policy requirement use cases can be stated and solved as an observability problem, by defining metrics and alerts that cover for missing policies or policy rules, without necessarily having to write a policy of the same kind to express such requirement to be fulfilled.
Merging policies with references to external objects¶
-
How to handle merges of policies from different namespaces that contain references to other objects (e.g. Secrets)?
-
Often policies rules include references to other Kubernetes objects, such as Secrets, typically defined in the same namespace as the policy object. When merging policies from different namespaces, these references need to be taken into account.
-
If not carried along with the derivative resources (e.g. Authorino AuthConfig objects) that are created from a merge of policies (or from the computed effective policy), composed out of definitions from different namespaces, and that depend on those references, these references to external objects can be broken.
-
This is not much of a problem for atomic D/O only, as the derivative objects that depend on the references could be forced to be created in the same namespace as the policy that wins against all the others – and therefore in the same namespace of the winning referents as well. However, when merging policies, we can run into a situation where final effective policies (thus also other derivative resources) contain references to objects inherited from definitions from other namespaces.
-
Possible solutions to this problem include:
-1. Copying the referenced objects into the namespace where the derivative resources will be created.
- - Involves maintaining (watching and reconciling) those referenced objects
- - May raise security concerns
-2. Allowing derivative resources (e.g. Authorino AuthConfigs) to reference objects across namespaces, as well as giving permissions to the components that process those references (e.g. Authorino) to read across namespaces
- - May raise security concerns
- - Should probably be restricted to derivative resources created by Kuadrant and not allowed to users who create the derivative resources themselves
Should Kuadrant's inherited policy specs resemble more the specs of the objects they target?
-
The UX for one who writes a Kuadrant policy of the inherited class of policies is arguably not very different from writing any custom resource that happens to specify a targetRef field.
-Other than name and kind of target object, there is nothing much in a Kuadrant policy custom resource that provides the user with an experience almost close to be "adding fields" in the target object.
-
With the exception of a few types reused for the route selectors, the spec of a Kuadrant policy is very different from the spec of the object that ultimately the policy augments, i.e. the spec of the route object. This remains basically unchanged after this RFC. However, another way to think on the design of those APIs is one where, in contrast, the specs of the policies partially mirror the spec of the route, so users can write policies in a more intuitive fashion, as if the definitions of the policy would look like extensions of the routes they target (directly or by targeting gateways the routes are attached to.)
An inherited policy that targets the HTTPRoute above could otherwise look like the following:
-
kind:Policy
-metadata:
-name:my-policy
-spec:
-targetRef:
-kind:HTTPRoute
-name:my-route
-defaults:# mirrors the spec of the httproute object
-policySpecificDef:{…}# augments the entire httproute object
-overrides:# mirrors the spec of the httproute object
-rules:
--name:rule-2
-policySpecificDef:{…}# augments only httprouterule rule-2 of the httproute object
-
-
The above already is somewhat closer to being true for the AuthPolicy API, than it is for the RateLimitPolicy one. However, that is strictly coincidental, because the AuthPolicy's spec happens to specify a rules field, where the equivalent at the same level in RateLimitPolicy is called limits.
-
This alternative design could make writing policies more like defining filters in an HTTPRoute, with the difference that policies are external to the target they extend (while filters are internal.) At the same time, it could be a replacement for Kuadrant route selectors, where the context of applicability of a policy rule is given by the very structure within the spec how the policy rule is declared (resembling the one of the target), thus also would shaping context for D/O.
-
One caveat of this design though is that each policy specific definition (i.e. the rule specification that extends the object at a given point defined by the very structure of the spec) is exclusive of that given point in the structure of the object. I.e., one cannot specify a single policy rule that augments N > 1 specific rules of a target HTTPRoute.
-
Due to its relevance to the design of the API that enables D/O, this was left as an unresolved question. To be nonetheless noticed that, as a pattern, this alternative API design extends beyond inherited policies, impacting as well the direct policy kinds DNSPolicy and TLSPolicy.
Although this proposal was thought to keep options open for multiple policies of a kind targeting a same network resource, this is currently not the state of things for Kuadrant. Instead, Kuadrant enforces 1:1 relationship between policies of a kind and target resources.
-
Supporting N:1 relationships could enable use cases such as of App Developers defining D/O for each other at the same level of a shared xRoute, as well as Platform Engineers setting different policy rules on the same Gateway.
-
This could provide an alternative to achieving separation of concerns for complex policy kinds such as the AuthPolicy, where different users could be responsible for authentication and authorization, without necessarily depending on defining new kinds of policies.
If Gateway API's GEP-995 is accepted (i.e. kubernetes-sigs/gateway-api#2593 gets merged) and the name field for route rules implemented in the APIs (HTTPRoute and GRPCRoute), this could impact how Kuadrant delivers D/O. Although the semantics could remain the same, the users specify the scope for a given set of policy rules could simplify significantly.
-
As of today, Kuadrant's AuthPolicy and RateLimitPolicy APIs allow users to target sections of a HTTPRoute based on route selectors, and thus all the conflict resolution involved in handling D/O must take that logics into account.
-
With named route rules supported by Gateway API, either route selectors could be redefined in a simpler form where each selector consists of a list of names of rules and/or entire policies could be scoped for a section of a resource, by defining the targetRef field based on the PolicyTargetReferenceWithSectionName type.
-
To be noted GEP-2649's recommendation of not defining inherited policies that allow for sectionName in the targetRef. Nonetheless, this is a general rule from the spec said to be acceptable to be broken in the spirit of offering better functionality to users, provided it can deal with the associated discoverability and complexity problems of this feature.
Despite having recently modified the AuthPolicy and RateLimitPolicy APIs to use maps for declaring policy rules instead of lists (RFC 0001), reverting this design in future versions of these APIs, plus treating those lists as listMapType, could let us leverage the API server's strategic merge type to handle merges between policy objects.
-
In the Policy CRDs, the policy rule types must specify a name field (required). The list of rules type (i.e. []Rule) must then speficy the following Kubebuilder CRD processing annotations:
-
// +listType=map
-// +listMapKey=name
-
-
-
-
-
-
As the time of writing, GEP-713 (Kubernetes Gateway API, SIG-NETWORK) is under revision, expected to be split into two separate GEPs, one for Direct Policies (GEP-2648) and one for Inherited Policies (GEP-2649.) Once these new GEPs supersede GEP-713, all references to the previous GEP in this document must be updated to GEP-2649. ↩
-
- Tip: Deploy a custom image of the Operator
-
- To deploy an image of the Operator other than the default quay.io/kuadrant/authorino-operator:latest, specify by setting the OPERATOR_IMAGE parameter. E.g.:
-
-
To install the Operator using the Operator Lifecycle Manager, you need to make the
-Operator CSVs available in the cluster by creating a CatalogSource resource.
-
The bundle and catalog images of the Operator are available in Quay.io:
API to install, manage and configure Authorino authorization services .
-
Each Authorino
-Custom Resource (CR) represents an instance of Authorino deployed to the cluster. The Authorino Operator will reconcile
-the state of the Kubernetes Deployment and associated resources, based on the state of the CR.
Enable/disable allowing AuthConfigs to supersede strict subsets of hosts already taken.
-
Default: false
-
-
-
replicas
-
Integer
-
Number of replicas desired for the Authorino instance. Values greater than 1 enable leader election in the Authorino service, where the leader updates the statuses of the AuthConfig CRs).
-
Default: 1
-
-
-
evaluatorCacheSize
-
Integer
-
Cache size (in megabytes) of each Authorino evaluator (when enabled in an AuthConfig).
-
Default: 1
-
-
-
image
-
String
-
Authorino image to be deployed (for dev/testing purpose only).
-
Default: quay.io/kuadrant/authorino:latest
-
-
-
imagePullPolicy
-
String
-
Sets the imagePullPolicy of the Authorino Deployment (for dev/testing purpose only).
-
Default: k8s default
-
-
-
logLevel
-
String
-
Defines the level of log you want to enable in Authorino (debug, info and error).
-
Default: info
-
-
-
logMode
-
String
-
Defines the log mode in Authorino (development or production).
Additional volumes to project in the Authorino pods. Useful for validation of TLS self-signed certificates of external
-services known to have to be contacted by Authorino at runtime.
There are a few concepts to understand Authorino's architecture. The main components are: Authorino, Envoy and the Upstream service to be protected. Envoy proxies requests to the configured virtual host upstream service, first contacting with Authorino to decide on authN/authZ.
-
The topology can vary from centralized proxy and centralized authorization service, to dedicated sidecars, with the nuances in between. Read more about the topologies in the Topologies section below.
-
Authorino is deployed using the Authorino Operator, from an Authorino Kubernetes custom resource. Then, from another kind of custom resource, the AuthConfig CRs, each Authorino instance reads and adds to the index the exact rules of authN/authZ to enforce for each protected host ("index reconciliation").
-
Everything that the AuthConfig reconciler can fetch in reconciliation-time is stored in the index. This is the case of static parameters such as signing keys, authentication secrets and authorization policies from external policy registries.
-
AuthConfigs can refer to identity providers (IdP) and trusted auth servers whose access tokens will be accepted to authenticate to the protected host. Consumers obtain an authentication token (short-lived access token or long-lived API key) and send those in the requests to the protected service.
-
When Authorino is triggered by Envoy via the gRPC interface, it starts evaluating the Auth Pipeline, i.e. it applies to the request the parameters to verify the identity and to enforce authorization, as found in the index for the requested host (See host lookup for details).
-
Apart from static rules, these parameters can include instructions to contact online with external identity verifiers, external sources of metadata and policy decision points (PDPs).
-
On every request, Authorino's "working memory" is called Authorization JSON, a data structure that holds information about the context (the HTTP request) and objects from each phase of the auth pipeline: i.e., authentication verification (phase i), ad-hoc metadata fetching (phase ii), authorization policy enforcement (phase iii), dynamic response (phase iv), and callbacks (phase v). The evaluators in each of these phases can both read and write from the Authorization JSON for dynamic steps and decisions of authN/authZ.
Typically, upstream APIs are deployed to the same Kubernetes cluster and namespace where the Envoy proxy and Authorino is running (although not necessarily). Whatever is the case, Envoy must be proxying to the upstream API (see Envoy's HTTP route components and virtual hosts) and pointing to Authorino in the external authorization filter.
-
This can be achieved with different topologies:
-
-
Envoy can be a centralized gateway with one dedicated instance of Authorino, proxying to one or more upstream services
Recommended in the protected services to validate the origin of the traffic. It must have been proxied by Envoy. See Authorino JSON injection for an extra validation option using a shared secret passed in HTTP header.
Authorino instances can run in either cluster-wide or namespaced mode.
-
Namespace-scoped instances only watch resources (AuthConfigs and Secrets) created in a given namespace. This deployment mode does not require admin privileges over the Kubernetes cluster to deploy the instance of the service (given Authorino's CRDs have been installed beforehand, such as when Authorino is installed using the Authorino Operator).
-
Cluster-wide deployment mode, in contraposition, deploys instances of Authorino that watch resources across the entire cluster, consolidating all resources into a multi-namespace index of auth configs. Admin privileges over the Kubernetes cluster is required to deploy Authorino in cluster-wide mode.
-
Be careful to avoid superposition when combining multiple Authorino instances and instance modes in the same Kubernetes cluster. Apart from caching unnecessary auth config data in the instances depending on your routing settings, the leaders of each instance (set of replicas) may compete for updating the status of the custom resources that are reconciled. See Resource reconciliation and status update for more information.
-
If necessary, use label selectors to narrow down the space of resources watched and reconciled by each Authorino instance. Check out the Sharding section below for details.
-
The Authorino AuthConfig Custom Resource Definition (CRD)¶
-
The desired protection for a service is declaratively stated by applying an AuthConfigCustom Resource to the Kubernetes cluster running Authorino.
-
An AuthConfig resource typically looks like the following:
-
apiVersion:authorino.kuadrant.io/v1beta2
-kind:AuthConfig
-metadata:
-name:my-api-protection
-spec:
-# The list of public host names of the services protected by this AuthConfig resource.
-# Authorino uses the host name provided in the payload of external authorization request to lookup for the corresponding AuthConfig to enforce.
-# Hostname collisions are prevented by rejecting to index a hostname already taken by another AuthConfig.
-# Format: hostname[:port]
-hosts:
--my-api.io:443# north-south traffic
--my-api.ns.svc.cluster.local# east-west traffic
-
-# Set of stored named patterns to be reused in conditions and pattern-matching authorization rules
-patterns:{"name"→{selector,operator,value},…}
-
-# Top-level conditions for the AuthConfig to be enforced.
-# If omitted, the AuthConfig will be enforced at all requests.
-# If present, all conditions must match for the AuthConfig to be enforced; otherwise, Authorino skips the AuthConfig and returns to the auth request with status OK.
-when:[{selector,operator,value | named pattern ref},…]
-
-# List of one or more trusted sources of identity:
-# - Configurations to verify JSON Web Tokens (JWTs) issued by an OpenID Connect (OIDC) server
-# - Endpoints for OAuth 2.0 token introspection
-# - Attributes for the Kubernetes `TokenReview` API
-# - Label selectors for API keys (stored in Kubernetes `Secret`s)
-# - Label selectors trusted x509 issuer certificates (stored in Kubernetes `Secret`s)
-# - Selectors for plain identity objects supplied in the payload of the authorization request
-# - Anonymous access configs
-authentication:{"name"→{…},…}
-
-# List of sources of external metadata for the authorization (optional):
-# - Endpoints for HTTP GET or GET-by-POST requests
-# - OIDC UserInfo endpoints (associated with an OIDC token issuer specified in the authentication configs)
-# - User-Managed Access (UMA) resource registries
-metadata:{"name"→{…},…}
-
-# List of authorization policies to be enforced (optional):
-# - Pattern-matching rules (e.g. `context.request.http.path eq '/pets'`)
-# - Open Policy Agent (OPA) inline or external Rego policies
-# - Attributes for the Kubernetes `SubjectAccessReview` API
-# – Attributes for authorization with an external SpiceDB server
-authorization:{"name"→{…},…}
-
-# Customization to the response to the external authorization request (optional)
-response:
-# List of dynamic response elements into the request on success authoization (optional):
-# - Plain text
-# - JSON objects
-# - Festival Wristbands (signed JWTs issued by Authorino)
-success:
-# List of HTTP headers to inject into the request post-authorization (optional):
-headers:{"name"→{…},…}
-
-# List of Envoy Dynamic Metadata to inject into the request post-authorization (optional):
-dynamicMetadata:{"name"→{…},…}
-
-# Custom HTTP status code, message and headers to replace the default `401 Unauthorized` response (optional)
-unauthenticated:
-code:302
-message:Redirecting to login
-headers:
-"Location":
-value:https://my-app.io/login
-
-# Custom HTTP status code, message and headers to replace the default `and `403 Forbidden` response (optional)
-unauthorized:{code,message,headers,body}
-
-# List of callback targets:
-# - Endpoints for HTTP requests
-callbacks:{"name"→{…},…}
-
-
Check out the OAS of the AuthConfig CRD for a formal specification of the options for authentication verification, external metadata fetching, authorization policies, and dynamic response, as well as any other host protection capability implemented by Authorino.
-
You can also read the specification from the CLI using the kubectl explain command. The Authorino CRD is required to have been installed in Kubernetes cluster. E.g. kubectl explain authconfigs.spec.authentication.overrides.
-
A complete description of supported features and corresponding configuration options within an AuthConfig CR can be found in the Features page.
-
More concrete examples of AuthConfigs for specific use-cases can be found in the User guides.
The instances of the Authorino authorization service workload, following the Operator pattern, watch events related to the AuthConfig custom resources, to build and reconcile an in-memory index of configs. Whenever a replica receives traffic for authorization request, it looks up in the index of AuthConfigs and then triggers the "Auth Pipeline", i.e. enforces the associated auth spec onto the request.
-
An instance can be a single authorization service workload or a set of replicas. All replicas watch and reconcile the same set of resources that match the --auth-config-label-selector and --secret-label-selector configuration options. (See both Cluster-wide vs. Namespaced instances and Sharding, for details about defining the reconciliation space of Authorino instances.)
-
The above means that all replicas of an Authorino instance should be able to receive traffic for authorization requests.
-
Among the multiple replicas of an instance, Authorino elects one replica to be leader. The leader is responsible for updating the status of reconciled AuthConfigs. If the leader eventually becomes unavailable, the instance will automatically elect another replica take its place as the new leader.
-
The status of an AuthConfig tells whether the resource is "ready" (i.e. indexed). It also includes summary information regarding the numbers of authentication configs, metadata configs, authorization configs and response configs within the spec, as well as whether Festival Wristband tokens are being issued by the Authorino instance as by spec.
-
Apart from watching events related to AuthConfig custom resources, Authorino also watches events related to Kubernetes Secrets, as part of Authorino's API key authentication feature. Secret resources that store API keys are linked to their corresponding AuthConfigs in the index. Whenever the Authorino instance detects a change in the set of API key Secrets linked to an AuthConfigs, the instance reconciles the index.
-
Authorino only watches events related to Secrets whose metadata.labels match the label selector --secret-label-selector of the Authorino instance. The default values of the label selector for Kubernetes Secrets representing Authorino API keys is authorino.kuadrant.io/managed-by=authorino.
-
The "Auth Pipeline" (aka: enforcing protection in request-time)¶
-
-
In each request to the protected API, Authorino triggers the so-called "Auth Pipeline", a set of configured evaluators that are organized in a 5-phase pipeline:
-
-
(i) Authentication phase: at least one source of identity (i.e., one authentication config) must resolve the supplied credential in the request into a valid identity or Authorino will otherwise reject the request as unauthenticated (401 HTTP response status).
-
(ii) Metadata phase: optional fetching of additional data from external sources, to add up to context and identity information, and used in authorization policies, dynamic responses and callback requests (phases iii to v).
-
(iii) Authorization phase: all unskipped policies must evaluate to a positive result ("authorized"), or Authorino will otherwise reject the request as unauthorized (403 HTTP response code).
-
(iv) Response phase – Authorino builds all user-defined response items (dynamic JSON objects and/or Festival Wristband OIDC tokens), which are supplied back to the external authorization client within added HTTP headers or as Envoy Dynamic Metadata
Each phase is sequential to the other, from (i) to (v), while the evaluators within each phase are triggered concurrently or as prioritized. The Authentication phase (i) is the only one required to list at least one evaluator (i.e. 1+ authentication configs); Metadata, Authorization and Response phases can have any number of evaluators (including zero, and even be omitted in this case).
Authorino reads the request host from Attributes.Http.Host of Envoy's CheckRequest type, and uses it as key to lookup in the index of AuthConfigs, matched against spec.hosts.
-
Alternatively to Attributes.Http.Host, a host entry can be supplied in the Attributes.ContextExtensions map of the external authorino request. This will take precedence before the host attribute of the HTTP request.
-
The host context extension is useful to support use cases such as of path prefix-based lookup and wildcard subdomains lookup with lookup strongly dictated by the external authorization client (e.g. Envoy), which often knows about routing and the expected AuthConfig to enforce beyond what Authorino can infer strictly based on the host name.
-
Wildcards can also be used in the host names specified in the AuthConfig, resolved by Authorino. E.g. if *.pets.com is in spec.hosts, Authorino will match the concrete host names dogs.pets.com, cats.pets.com, etc. In case, of multiple possible matches, Authorino will try the longest match first (in terms of host name labels) and fall back to the closest wildcard upwards in the domain tree (if any).
-
When more than one host name is specified in the AuthConfig, all of them can be used as key, i.e. all of them can be requested in the authorization request and will be mapped to the same config.
-
Example. Host lookup with wildcards.
-
-
The domain tree above induces the following relation:
The host can include the port number (i.e. hostname:port) or it can be just the name of the host name. Authorino will first try finding in the index a config associated to hostname:port, as supplied in the authorization request; if the index misses an entry for hostname:port, Authorino will then remove the :port suffix and repeat the lookup using just hostname as key. This provides implicit support for multiple port numbers for a same host without having to list all combinations in the AuthConfig.
Authorino tries to prevent host name collision between AuthConfigs by rejecting to link in the index any AuthConfig and host name if the host name is already linked to a different AuthConfig in the index. This was intentionally designed to prevent users from superseding each other's AuthConfigs, partially or fully, by just picking the same host names or overlapping host names as others.
-
When wildcards are involved, a host name that matches a host wildcard already linked in the index to another AuthConfig will be considered taken, and therefore the newest AuthConfig will be rejected to be linked to that host.
-
This behavior can be disabled to allow AuthConfigs to partially supersede each others' host names (limited to strict host subsets), by supplying the --allow-superseding-host-subsets command-line flag when running the Authorino instance.
On every Auth Pipeline, Authorino builds the Authorization JSON, a "working-memory" data structure composed of context (information about the request, as supplied by the Envoy proxy to Authorino) and auth (objects resolved in phases (i) to (v) of the pipeline). The evaluators of each phase can read from the Authorization JSON and implement dynamic properties and decisions based on its values.
-
At phase (iii), the authorization evaluators count on an Authorization JSON payload that looks like the following:
-
// The authorization JSON combined along Authorino's auth pipeline for each request
-{
- "context": { // the input from the proxy
- "origin": {…},
- "request": {
- "http": {
- "method": "…",
- "headers": {…},
- "path": "/…",
- "host": "…",
- …
- }
- }
- },
- "auth": {
- "identity": {
- // the identity resolved, from the supplied credentials, by one of the evaluators of phase (i)
- },
- "metadata": {
- // each metadata object/collection resolved by the evaluators of phase (ii), by name of the evaluator
- }
- }
-}
-
-
The policies evaluated can use any data from the authorization JSON to define authorization rules.
-
After phase (iii), Authorino appends to the authorization JSON the results of this phase as well, and the payload available for phase (iv) becomes:
-
// The authorization JSON combined along Authorino's auth pipeline for each request
-{
- "context": { // the input from the proxy
- "origin": {…},
- "request": {
- "http": {
- "method": "…",
- "headers": {…},
- "path": "/…",
- "host": "…",
- …
- }
- }
- },
- "auth": {
- "identity": {
- // the identity resolved, from the supplied credentials, by one of the evaluators of phase (i)
- },
- "metadata": {
- // each metadata object/collection resolved by the evaluators of phase (ii), by name of the evaluator
- },
- "authorization": {
- // each authorization policy result resolved by the evaluators of phase (iii), by name of the evaluator
- }
- }
-}
-
Besides providing the gRPC authorization interface – that implements the Envoy gRPC authorization server –, Authorino also provides another interface for raw HTTP authorization. This second interface responds to GET and POST HTTP requests sent to :5001/check, and is suitable for other forms of integration, such as:
-
-
using Authorino as Kubernetes ValidatingWebhook service (example);
-
other HTTP proxies and API gateways;
-
old versions of Envoy incompatible with the latest version of gRPC external authorization protocol (Authorino is based on v3.19.1 of Envoy external authorization API)
-
-
In the raw HTTP interface, the host used to lookup for an AuthConfig must be supplied in the Host HTTP header of the request. Other attributes of the HTTP request are also passed in the context to evaluate the AuthConfig, including the body of the request.
OpenID Connect and User-Managed Access configurations, discovered usually at reconciliation-time from well-known discovery endpoints.
-
Cached individual OpenID Connect configurations discovered by Authorino can be configured to be auto-refreshed, by setting the corresponding spec.authentication.jwt.ttl field in the AuthConfig (given in seconds, default: 0 – i.e. no cache update).
-
JSON Web Keys (JWKs) and JSON Web Key Sets (JWKS)¶
-
JSON signature verification certificates linked by discovered OpenID Connect configurations, fetched usually at reconciliation-time.
Performed automatically by Authorino at reconciliation-time for the authorization policies based on the built-in OPA module.
-
Precompiled and cached individual Rego policies originally pulled by Authorino from external registries can be configured to be auto-refreshed, by setting the corresponding spec.authorization.opa.externalRegistry.ttl field in the AuthConfig (given in seconds, default: 0 – i.e. no cache update).
For consecutive requests performed, within a given period of time, by a same user that request for a same resource, such that the result of the auth pipeline can be proven that would not change.
By default, Authorino instances will watch AuthConfig CRs in the entire space (namespace or entire cluster; see Cluster-wide vs. Namespaced instances for details). To support combining multiple Authorino instances and instance modes in the same Kubernetes cluster, and yet avoiding superposition between the instances (i.e. multiple instances reconciling the same AuthConfigs), Authorino offers support for data sharding, i.e. to horizontally narrow down the space of reconciliation of an Authorino instance to a subset of that space.
-
The benefits of limiting the space of reconciliation of an Authorino instance include avoiding unnecessary caching and workload in instances that do not receive corresponding traffic (according to your routing settings) and preventing leaders of multiple instances (sets of replicas) to compete on resource status updates (see Resource reconciliation and status update for details).
-
Use-cases for sharding of AuthConfigs:
-
-
Horizontal load balancing of traffic of authorization requests
-
Supporting for managed centralized instances of Authorino to API owners who create and maintain their own AuthConfigs within their own user namespaces.
-
-
Authorino's custom controllers filter the AuthConfig-related events to be reconciled using Kubernetes label selectors, defined for the Authorino instance via --auth-config-label-selector command-line flag. By default, --auth-config-label-selector is empty, meaning all AuthConfigs in the space are watched; this variable can be set to any value parseable as a valid label selector, causing Authorino to then watch only events of AuthConfigs whose metadata.labels match the selector.
-
The following are all valid examples of AuthConfig label selector filters:
-
--auth-config-label-selector="authorino.kuadrant.io/managed-by=authorino"
---auth-config-label-selector="authorino.kuadrant.io/managed-by=authorino,other-label=other-value"
---auth-config-label-selector="authorino.kuadrant.io/managed-by in (authorino,kuadrant)"
---auth-config-label-selector="authorino.kuadrant.io/managed-by!=authorino-v0.4"
---auth-config-label-selector="!disabled"
-
In case you want to contribute with an idea for enhancement, a bug fix, or question, please make sure to describe the issue so we can start a conversation together and help you find the best way to get your contribution merged.
Build an image of Authorino based on the current branch
-
Push the freshly built image to the cluster's registry
-
Generate TLS certificates for the Authorino service
-
Deploy an instance of Authorino
-
Deploy the example application Talker API, a simple HTTP API that echoes back whatever it gets in the request
-
Setup Envoy for proxying to the Talker API and using Authorino for external authorization
-
-
makelocal-setup
-
-
You will be prompted to edit the Authorino custom resource.
-
The main workload composed of Authorino instance and user apps (Envoy, Talker API) will be deployed to the default Kubernetes namespace.
-
Once the deployment is ready, you can forward the requests on port 8000 to the Envoy service
-
kubectlport-forwarddeployment/envoy8000:8000&
-
-
-Pro tips
-
-
Change the default workload namespace by supplying the NAMESPACE argument to your make local-setup and other deployment, apps and local cluster related targets. If the namespace does not exist, it will be created.
-
Switch to TLS disabled by default when deploying locally by supplying TLS_ENABLED=0 to your make local-setup and make deploy commands. E.g. make local-setup TLS_ENABLED=0.
-
Skip being prompted to edit the Authorino CR and default to an Authorino deployment with TLS enabled, debug/development log level/mode, and standard name 'authorino', by supplying FF=1 to your make local-setup and make deploy commands. E.g. make local-setup FF=1
-
Supply DEPLOY_IDPS=1 to make local-setup and make user-apps to deploy Keycloak and Dex to the cluster. DEPLOY_KEYCLOAK and DEPLOY_DEX are also available. Read more about additional tools for specific use cases in the section below.
-
Saving the ID of the process (PID) of the port-forward command spawned in the background can be useful to later kill and restart the process. E.g. kubectl port-forward deployment/envoy 8000:8000 &;PID=$!; then kill $PID.
To deploy Limitador – pre-configured in Envoy for rate-limiting the Talker API to 5 hits per minute per user_id when available in the cluster workload –, run:
Authorino examples include a bundle of Dex preloaded with the following setup:
-
-
Preloaded clients:
-
demo: to which API consumers delegate access and therefore the one which access tokens are issued to (Client secret: aaf88e0e-d41d-4325-a068-57c4b0d61d8e)
service-account-1: to obtain access tokens via client_credentials OAuth2 grant type, to consume the Talker API (Client secret: DbgXROi3uhWYCxNUq_U1ZXjGfLHOIM8X3C2bJLpeEdE); includes metadata privilege: { "talker-api": ["read"] } that can be used to write authorization policies
-
talker-api: to authenticate to the token introspect endpoint (Client secret: V6g-2Eq2ALB1_WHAswzoeZofJ_e86RI4tdjClDDDb4g)
Do not add sensitive data to your info log messages; instead, redact all sensitive data in your log messages or use debug log level by mutating the logger with V(1) before outputting the message.
We call features of Authorino the different things one can do to enforce identity verification & authentication and authorization on requests to protected services. These can be a specific identity verification method based on a supported authentication protocol, or a method to fetch additional auth metadata in request-time, etc.
At a deeper level, a feature can also be an additional functionality within a bigger feature, usually applicable to the whole class the bigger feature belongs to. For instance, the configuration of how auth credentials expected to be carried in the request, which is broadly available for any identity verification method. Other examples are: Identity extension and Priorities.
-
A full specification of all features of Authorino that can be configured in an AuthConfig can be found in the official spec of the custom resource definition.
-
You can also learn about Authorino features by using the kubectl explain command in a Kubernetes cluster where the Authorino CRD has been installed. E.g. kubectl explain authconfigs.spec.authentication.credentials.
The first feature of Authorino to learn about is a common functionality used in the specification of many other features. JSON paths are selectors of data from the Authorization JSON used in parts of an AuthConfig for referring to dynamic values of each authorization request.
-
Usage examples of JSON paths are: dynamic URLs and request parameters when fetching metadata from external sources, dynamic authorization policy rules, and dynamic authorization response attributes (e.g. injected HTTP headers, Festival Wristband token claims, etc).
@strip
-Strips out any non-printable characters such as carriage return. E.g. auth.identity.email.@strip → "jane@petcorp.com".
-
@case:upper|lower
-Changes the case of a string. E.g. auth.identity.username.@case:upper → "JANE".
-
@replace:{"old":string,"new":string}
-Replaces a substring within a string. E.g. auth.identity.username.@replace:{"old":"Smith","new":"Doe"} → "Jane Doe".
-
@extract:{"sep":string,"pos":int}
-Splits a string at occurrences of a separator (default: " ") and selects the substring at the pos-th position (default: 0). E.g. context.request.path.@extract:{"sep":"/","pos":2} → 123.
-
@base64:encode|decode
-base64-encodes or decodes a string value. E.g. auth.identity.username.decoded.@base64:encode → "amFuZQo=".
-
In combination with @extract, @base64 can be used to extract the username in an HTTP Basic Authentication request. E.g. context.request.headers.authorization.@extract:{"pos":1}|@base64:decode|@extract:{"sep":":","pos":1} → "jane".
Authorino relies on Kubernetes Secret resources to represent API keys.
-
To define an API key, create a Secret in the cluster containing an api_key entry that holds the value of the API key.
-
API key secrets must be created in the same namespace of the AuthConfig (default) or spec.authentication.apiKey.allNamespaces must be set to true (only works with cluster-wide Authorino instances).
-
API key secrets must be labeled with the labels that match the selectors specified in spec.authentication.apiKey.selector in the AuthConfig.
-
Whenever an AuthConfig is indexed, Authorino will also index all matching API key secrets. In order for Authorino to also watch events related to API key secrets individually (e.g. new Secret created, updates, deletion/revocation), Secrets must also include a label that matches Authorino's bootstrap configuration --secret-label-selector (default: authorino.kuadrant.io/managed-by=authorino). This label may or may not be present to spec.authentication.apiKey.selector in the AuthConfig without implications for the caching of the API keys when triggered by the reconciliation of the AuthConfig; however, if not present, individual changes related to the API key secret (i.e. without touching the AuthConfig) will be ignored by the reconciler.
-
Example. For the following AuthConfig:
-
apiVersion:authorino.kuadrant.io/v1beta2
-kind:AuthConfig
-metadata:
-name:my-api-protection
-namespace:authorino-system
-spec:
-hosts:
--my-api.io
-authentication:
-"api-key-users":
-apiKey:
-selector:
-matchLabels:# the key-value set used to select the matching `Secret`s; resources including these labels will be accepted as valid API keys to authenticate to this service
-group:friends# some custom label
-allNamespaces:true# only works with cluster-wide Authorino instances; otherwise, create the API key secrets in the same namespace of the AuthConfig
-
-
The following Kubernetes Secret represents a valid API key:
-
apiVersion:v1
-kind:Secret
-metadata:
-name:user-1-api-key-1
-namespace:default
-labels:
-authorino.kuadrant.io/managed-by:authorino# so the Authorino controller reconciles events related to this secret
-group:friends
-stringData:
-api_key:<some-randomly-generated-api-key-value>
-type:Opaque
-
-
The resolved identity object, added to the authorization JSON following an API key identity source evaluation, is the Kubernetes Secret resource (as JSON).
Authorino can verify Kubernetes-valid access tokens (using Kubernetes TokenReview API).
-
These tokens can be either ServiceAccount tokens such as the ones issued by kubelet as part of Kubernetes Service Account Token Volume Projection, or any valid user access tokens issued to users of the Kubernetes server API.
-
The list of audiences of the token must include the requested host and port of the protected API (default), or all audiences specified in the Authorino AuthConfig custom resource. For example:
-
For the following AuthConfig CR, the Kubernetes token must include the audience my-api.io:
The resolved identity object added to the authorization JSON following a successful Kubernetes authentication identity evaluation is the status field of TokenReview response (see TokenReviewStatus for reference).
In reconciliation-time, using OpenID Connect Discovery well-known endpoint, Authorino automatically discovers and caches OpenID Connect configurations and associated JSON Web Key Sets (JWKS) for all OpenID Connect issuers declared in an AuthConfig. Then, in request-time, Authorino verifies the JSON Web Signature (JWS) and check the time validity of signed JSON Web Tokens (JWT) supplied on each request.
-
Important! Authorino does not implement OAuth2 grants nor OIDC authentication flows. As a common recommendation of good practice, obtaining and refreshing access tokens is for clients to negotiate directly with the auth servers and token issuers. Authorino will only validate those tokens using the parameters provided by the trusted issuer authorities.
-
-
The kid claim stated in the JWT header must match one of the keys cached by Authorino during OpenID Connect Discovery, therefore supporting JWK rotation.
-
The decoded payload of the validated JWT is appended to the authorization JSON as the resolved identity.
-
OpenID Connect configurations and linked JSON Web Key Sets can be configured to be automatically refreshed (pull again from the OpenID Connect Discovery well-known endpoints), by setting the authentication.jwt.ttl field (given in seconds, default: 0 – i.e. auto-refresh disabled).
For bare OAuth 2.0 implementations, Authorino can perform token introspection on the access tokens supplied in the requests to protected APIs.
-
Authorino does not implement any of OAuth 2.0 grants for the applications to obtain the token. However, it can verify supplied tokens with the OAuth server, including opaque tokens, as long as the server exposes the token_introspect endpoint (RFC 7662).
-
Developers must set the token introspection endpoint in the AuthConfig, as well as a reference to the Kubernetes secret storing the credentials of the OAuth client to be used by Authorino when requesting the introspect.
-
-
The response returned by the OAuth2 server to the token introspection request is the resolved identity appended to the authorization JSON.
Authorino can verify X.509 certificates presented by clients for authentication on the request to the protected APIs, at application level.
-
Trusted root Certificate Authorities (CA) are stored in Kubernetes Secrets labeled according to selectors specified in the AuthConfig, watched and indexed by Authorino. Make sure to create proper kubernetes.io/tls-typed Kubernetes Secrets, containing the public certificates of the CA stored in either a tls.crt or ca.crt entry inside the secret.
-
Trusted root CA secrets must be created in the same namespace of the AuthConfig (default) or spec.authentication.x509.allNamespaces must be set to true (only works with cluster-wide Authorino instances).
-
Client certificates must include x509 v3 extension specifying 'Client Authentication' extended key usage.
-
The identity object resolved out of a client x509 certificate is equal to the subject field of the certificate, and it serializes as JSON within the Authorization JSON usually as follows:
Authorino can read plain identity objects, based on authentication tokens provided and verified beforehand using other means (e.g. Envoy JWT Authentication filter, Kubernetes API server authentication), and injected into the payload to the external authorization service.
-
The plain identity object is retrieved from the Authorization JSON based on a JSON path specified in the AuthConfig.
-
This feature is particularly useful in cases where authentication/identity verification is handled before invoking the authorization service and its resolved value injected in the payload can be trusted. Examples of applications for this feature include:
-- Authentication handled in Envoy leveraging the Envoy JWT Authentication filter (decoded JWT injected as 'metadata_context')
-- Use of Authorino as Kubernetes ValidatingWebhook service (Kubernetes 'userInfo' injected in the body of the AdmissionReview request)
-
Example of AuthConfig to retrieve plain identity object from the Authorization JSON.
If the specified JSON path does not exist in the Authorization JSON or the value is null, the identity verification will fail and, unless other identity config succeeds, Authorino will halt the Auth Pipeline with the usual 401 Unauthorized.
Literally a no-op evaluator for the identity verification phase that returns a static identity object {"anonymous":true}.
-
It allows to implement AuthConfigs that bypasses the identity verification phase of Authorino, to such as:
-- enable anonymous access to protected services (always or combined with Priorities)
-- postpone authentication in the Auth Pipeline to be resolved as part of an OPA policy
-
Example of AuthConfig spec that falls back to anonymous access when OIDC authentication fails, enforcing read-only access to the protected service in such cases:
The value of the issuer must be the same issuer specified in the custom resource for the protected API originally issuing wristband. Eventually, this can be the same custom resource where the wristband is configured as a valid source of identity, but not necessarily.
All the identity verification methods supported by Authorino can be configured regarding the location where access tokens and credentials (i.e. authentication secrets) fly within the request.
-
By default, authentication secrets are expected to be supplied in the Authorization HTTP header, with the default Bearer prefix and the plain authentication secret separated by space.
-
The full list of supported options is exemplified below:
Resolved identity objects can be extended with user-defined JSON properties. Values can be static or fetched from the Authorization JSON.
-
A typical use-case for this feature is token normalization. Say you have more than one identity source listed in your AuthConfig but each source issues an access token with a different JSON structure – e.g. two OIDC issuers that use different names for custom JWT claims of similar meaning; when two different identity verification/authentication methods are combined, such as API keys (whose identity objects are the corresponding Kubernetes Secrets) and Kubernetes tokens (whose identity objects are Kubernetes UserInfo data).
-
In such cases, identity extension can be used to normalize the token to always include the same set of JSON properties of interest, regardless of the source of identity that issued the original token verified by Authorino. This simplifies the writing of authorization policies and configuration of dynamic responses.
-
In case of extending an existing property of the identity object (replacing), the API allows to control whether to overwrite the value or not. This is particularly useful for normalizing tokens of a same identity source that nonetheless may occasionally differ in structure, such as in the case of JWT claims that sometimes may not be present but can be safely replaced with another (e.g. username or sub).
Generic HTTP adapter that sends a request to an external service. It can be used to fetch external metadata for the authorization policies (phase ii of the Authorino Auth Pipeline), or as a web hook.
-
The adapter allows issuing requests either by GET or POST methods; in both cases with URL and parameters defined by the user in the spec. Dynamic values fetched from the Authorization JSON can be used.
-
POST request parameters as well as the encoding of the content can be controlled using the bodyParameters and contentType fields of the config, respectively. The Content-Type of POST requests can be either application/x-www-form-urlencoded (default) or application/json.
-
Authentication of Authorino with the external metadata server can be set either via long-lived shared secret stored in a Kubernetes Secret or via OAuth2 client credentials grant. For long-lived shared secret, set the sharedSecretRef field. For OAuth2 client credentials grant, use the oauth2 option.
-
In both cases, the location where the secret (long-lived or OAuth2 access token) travels in the request performed to the external HTTP service can be specified in the credentials field. By default, the authentication secret is supplied in the Authorization header with the Bearer prefix.
-
Custom headers can be set with the headers field. Nevertheless, headers such as Content-Type and Authorization (or eventual custom header used for carrying the authentication secret, set instead via the credentials option) will be superseded by the respective values defined for the fields contentType and sharedSecretRef.
Online fetching of OpenID Connect (OIDC) UserInfo data (phase ii of the Authorino Auth Pipeline), associated with an OIDC identity source configured and resolved in phase (i).
-
Apart from possibly complementing information of the JWT, fetching OpenID Connect UserInfo in request-time can be particularly useful for remote checking the state of the session, as opposed to only verifying the JWT/JWS offline.
-
Implementation requires a JWT verification authentication config (spec.authentication.jwt) in the same AuthConfig, so the well-known configuration of the OpenId Connect (OIDC) issuer can be reused.
-
The response returned by the OIDC server to the UserInfo request is appended (as JSON) to auth.metadata in the authorization JSON.
User-Managed Access (UMA) is an OAuth-based protocol for resource owners to allow other users to access their resources. Since the UMA-compliant server is expected to know about the resources, Authorino includes a client that fetches resource data from the server and adds that as metadata of the authorization payload.
-
This enables the implementation of resource-level Attribute-Based Access Control (ABAC) policies. Attributes of the resource fetched in a UMA flow can be, e.g., the owner of the resource, or any business-level attributes stored in the UMA-compliant server.
-
A UMA-compliant server is an external authorization server (e.g., Keycloak) where the protected resources are registered. It can be as well the upstream API itself, as long as it implements the UMA protocol, with initial authentication by client_credentials grant to exchange for a Protected API Token (PAT).
-
-
It's important to notice that Authorino does NOT manage resources in the UMA-compliant server. As shown in the flow above, Authorino's UMA client is only to fetch data about the requested resources. Authorino exchanges client credentials for a Protected API Token (PAT), then queries for resources whose URI match the path of the HTTP request (as passed to Authorino by the Envoy proxy) and fetches data of each matching resource.
-
The resources data is added as metadata of the authorization payload and passed as input for the configured authorization policies. All resources returned by the UMA-compliant server in the query by URI are passed along. They are available in the PDPs (authorization payload) as input.auth.metadata.custom-name => Array. (See The "Auth Pipeline" for details.)
Grant/deny access based on simple pattern-matching expressions ("patterns") compared against values selected from the Authorization JSON.
-
Each expression is a tuple composed of:
-- a selector, to fetch from the Authorization JSON – see Common feature: JSON paths for details about syntax;
-- an operator – eq (equals), neq (not equal); incl (includes) and excl (excludes), for arrays; and matches, for regular expressions;
-- a fixed comparable value
-
Rules can mix and combine literal expressions and references to expression sets ("named patterns") defined at the upper level of the AuthConfig spec. (See Common feature: Conditions)
-
spec:
-authorization:
-"my-simple-json-pattern-matching-policy":
-patternMatching:
-patterns:# All patterns must match for access to be granted
--selector:auth.identity.email_verified
-operator:eq
-value:"true"
--patternRef:admin
-
-patterns:
-admin:# a named pattern that can be reused in other sets of rules or conditions
--selector:auth.identity.roles
-operator:incl
-value:admin
-
You can model authorization policies in Rego language and add them as part of the protection of your APIs.
-
Policies can be either declared in-line in Rego language (rego) or as an HTTP endpoint where Authorino will fetch the source code of the policy in reconciliation-time (externalPolicy).
-
Policies pulled from external registries can be configured to be automatically refreshed (pulled again from the external registry), by setting the authorization.opa.externalPolicy.ttl field (given in seconds, default: 0 – i.e. auto-refresh disabled).
-
Authorino's built-in OPA module precompiles the policies during reconciliation of the AuthConfig and caches the precompiled policies for fast evaluation in runtime, where they receive the Authorization JSON as input.
-
-
An optional field allValues: boolean makes the values of all rules declared in the Rego document to be returned in the OPA output after policy evaluation. When disabled (default), only the boolean value allow is returned. Values of internal rules of the Rego document can be referenced in subsequent policies/phases of the Auth Pipeline.
Access control enforcement based on rules defined in the Kubernetes authorization system, i.e. Role, ClusterRole, RoleBinding and ClusterRoleBinding resources of Kubernetes RBAC.
-
Authorino issues a SubjectAccessReview (SAR) inquiry that checks with the underlying Kubernetes server whether the user can access a particular resource, resource kind or generic URL.
-
It supports resource attributes authorization check (parameters defined in the AuthConfig) and non-resource attributes authorization check (HTTP endpoint inferred from the original request).
-- Resource attributes: adequate for permissions set at namespace level, defined in terms of common attributes of operations on Kubernetes resources (namespace, API group, kind, name, subresource, verb)
-- Non-resource attributes: adequate for permissions set at cluster scope, defined for protected endpoints of a generic HTTP API (URL path + verb)
-
Example of Kubernetes role for resource attributes authorization:
Kubernetes' authorization policy configs look like the following in an Authorino AuthConfig:
-
authorization:
-"kubernetes-rbac":
-kubernetesSubjectAccessReview:
-user:# values of the parameter can be fixed (`value`) or fetched from the Authorization JSON (`selector`)
-selector:auth.identity.metadata.annotations.userid
-
-groups:[]# user groups to test for.
-
-# for resource attributes permission checks; omit it to perform a non-resource attributes SubjectAccessReview with path and method/verb assumed from the original request
-# if included, use the resource attributes, where the values for each parameter can be fixed (`value`) or fetched from the Authorization JSON (`selector`)
-resourceAttributes:
-namespace:
-value:default
-group:
-value:pets.io# the api group of the protected resource to be checked for permissions for the user
-resource:
-value:pets# the resource kind
-name:
-selector:context.request.http.path.@extract:{"sep":"/","pos":2}# resource name – e.g., the {id} in `/pets/{id}`
-verb:
-selector:context.request.http.method.@case:lower# api operation – e.g., copying from the context to use the same http method of the request
-
-
user and properties of resourceAttributes can be defined from fixed values or patterns of the Authorization JSON.
-
An array of groups (optional) can as well be set. When defined, it will be used in the SubjectAccessReview request.
Custom response forms: successful authorization vs custom denial status¶
-
The response to the external authorization request can be customized in the following fashion:
-- Successful authorization (response.success)
- - Added HTTP headers (response.success.headers)
- - Envoy Dynamic Metadata (response.success.dynamicMetadata)
-- Custom denial status
- - Unauthenticated (response.unauthenticated)
- - Unauthorized (response.unauthorized)
-
Successful authorization custom responses can be set based on any of the supported custom authorization methods:
-- Plain text value
-- JSON injection
-- Festival Wristband Tokens
Set custom responses as HTTP headers injected in the request post-successful authorization by specifying one of the supported methods under response.success.headers.
-
The name of the response config (default) or the value of the key option (if provided) will used as the name of the header.
Authorino custom response methods can also be used to propagate Envoy Dynamic Metadata. To do so, set one of the supported methods under response.success.dynamicMetadata.
-
The name of the response config (default) or the value of the key option (if provided) will used as the name of the root property of the dynamic metadata content.
-
A custom response exported as Envoy Dynamic Metadata can be set in the Envoy route or virtual host configuration as input to a consecutive filter in the filter chain.
-
E.g., to read metadata emitted by the authorization service with scheme { "auth-data": { "api-key-ns": string, "api-key-name": string } }, as input in a rate limit configuration placed in the filter chain after the external authorization, the Envoy config may look like the following:
-
# Envoy config snippet to inject `user_namespace` and `username` rate limit descriptors from metadata emitted by Authorino
-rate_limits:
--actions:
--metadata:
-metadata_key:
-key:"envoy.filters.http.ext_authz"
-path:
--key:auth-data# root of the dynamic metadata object, as declared in a custom response config of the AuthConfig (name or key)
--key:api-key-ns
-descriptor_key:user_namespace
--metadata:
-metadata_key:
-key:"envoy.filters.http.ext_authz"
-path:
--key:auth-data# root of the dynamic metadata object, as declared in a custom response config of the AuthConfig (name or key)
--key:api-key-name
-descriptor_key:username
-
By default, Authorino will inform Envoy to respond with 401 Unauthorized or 403 Forbidden respectively when the identity verification (phase i of the Auth Pipeline) or authorization (phase ii) fail. These can be customized respectively by specifying spec.response.unauthanticated and spec.response.unauthorized in the AuthConfig.
Simpler, yet more generalized form, for extending the authorization response for header mutation and Envoy Dynamic Metadata, based on plain text values.
User-defined dynamic JSON objects generated by Authorino in the response phase, from static or dynamic data of the auth pipeline, and passed back to the external authorization client within added HTTP headers or Dynamic Metadata.
-
The following Authorino AuthConfig custom resource is an example that defines 3 dynamic JSON response items, where two items are returned to the client, stringified, in added HTTP headers, and the third as Envoy Dynamic Metadata. Envoy proxy can be configured to propagate the dynamic metadata emitted by Authorino into another filter – e.g. the rate limit filter.
Festival Wristbands are signed OpenID Connect JSON Web Tokens (JWTs) issued by Authorino at the end of the auth pipeline and passed back to the client, typically in added HTTP response header. It is an opt-in feature that can be used to implement Edge Authentication Architecture (EAA) and enable token normalization. Authorino wristbands include minimal standard JWT claims such as iss, iat, and exp, and optional user-defined custom claims, whose values can be static or dynamically fetched from the authorization JSON.
-
The Authorino AuthConfig custom resource below sets an API protection that issues a wristband after a successful authentication via API key. Apart from standard JWT claims, the wristband contains 2 custom claims: a static value aud=internal and a dynamic value born that fetches from the authorization JSON the date/time of creation of the secret that represents the API key used to authenticate.
The signing key names listed in signingKeyRefs must match the names of Kubernetes Secret resources created in the same namespace, where each secret contains a key.pem entry that holds the value of the private key that will be used to sign the wristbands issued, formatted as PEM. The first key in this list will be used to sign the wristbands, while the others are kept to support key rotation.
-
For each protected API configured for the Festival Wristband issuing, Authorino exposes the following OpenID Connect Discovery well-known endpoints (available for requests within the cluster):
-- OpenID Connect configuration:
- https://authorino-oidc.default.svc:8083/{namespace}/{api-protection-name}/{response-config-name}/.well-known/openid-configuration
-- JSON Web Key Set (JWKS) well-known endpoint:
- https://authorino-oidc.default.svc:8083/{namespace}/{api-protection-name}/{response-config-name}/.well-known/openid-connect/certs
Priorities allow to set sequence of execution for blocks of concurrent evaluators within phases of the Auth Pipeline.
-
Evaluators of same priority execute concurrently to each other "in a block". After syncing that block (i.e. after all evaluators of the block have returned), the next block of evaluator configs of consecutive priority is triggered.
-
Use cases for priorities are:
-1. Saving expensive tasks to be triggered when there's a high chance of returning immediately after finishing executing a less expensive one – e.g.
- - an identity config that calls an external IdP to verify a token that is rarely used, compared to verifying JWTs preferred by most users of the service;
- - an authorization policy that performs some quick checks first, such as verifying allowed paths, and only if it passes, moves to the evaluation of a more expensive policy.
-2. Establishing dependencies between evaluators - e.g.
- - an external metadata request that needs to wait until a previous metadata responds first (in order to use data from the response)
-
Priorities can be set using the priority property available in all evaluator configs of all phases of the Auth Pipeline (identity, metadata, authorization and response). The lower the number, the highest the priority. By default, all evaluators have priority 0 (i.e. highest priority).
-
Consider the following example to understand how priorities work:
Identity configs tier-2 and tier-3 (priority 1) will only trigger (concurrently) in case tier-1 (priority 0) fails to validate the authentication token first. (This behavior happens without prejudice of context canceling between concurrent evaluators – i.e. evaluators that are triggered concurrently to another, such as tier-2 and tier-3, continue to cancel the context of each other if any of them succeeds validating the token first.)
-
-
-
Metadata source second (priority 1) uses the response of the request issued by metadata source first (priority 0), so it will wait for first to finish by triggering only in the second block.
-
-
-
Authorization policy allowed-endpoints (priority 0) is considered to be a lot less expensive than more-expensive-policy (priority 1) and has a high chance of denying access to the protected service (if the path is not one of the allowed endpoints). By setting different priorities to these policies we ensure the more expensive policy if triggered in sequence of the less expensive one, instead of concurrently.
Conditions, named when in the AuthConfig API, are logical expressions, composed of patterns and logical operator AND and OR, that can be used to condition the evaluation of a particular auth rule within an AuthConfig, as well as of the AuthConfig altogether ("top-level conditions").
-
The patterns are evaluated against the Authorization JSON, where each pattern is a tuple composed of:
-- selector: a JSON path to fetch a value from the Authorization JSON
-- operator: one of: eq (equals); neq (not equal); incl (includes) and excl (excludes), for when the value fetched from the Authorization JSON is expected to be an array; matches, for regular expressions
-- value: a static string value to compare the value selected from the Authorization JSON with.
-
An expression contains one or more patterns and they must either all evaluate to true ("AND" operator, declared by grouping the patterns within an all block) or at least one of the patterns must be true ("OR" operator, when grouped within an any block.) Patterns not explicitly grouped are AND'ed by default.
-
To avoid repetitions when listing patterns, any set of literal { pattern, operator, value } tuples can be stored at the top-level of the AuthConfig spec, indexed by name, and later referred within an expression by including a patternRef in the block of conditions.
-
Examples of when conditions
-
i) to skip an entire AuthConfig based on the context (AND operator assumed by default):
-
spec:
-when:# auth enforced only on requests to POST /resources/*
--selector:context.request.http.method
-operator:eq
-value:POST
--selector:context.request.http.path
-operator:matches
-value:^/resources/.*
-
-
ii) equivalent to the above with explicit AND operator (i.e., all block):
-
spec:
-when:# auth enforced only on requests to POST /resources/*
--all:
--selector:context.request.http.method
-operator:eq
-value:POST
--selector:context.request.http.path
-operator:matches
-value:^/resources/.*
-
-
iii) OR condition (i.e., any block):
-
spec:
-when:# auth enforced only on requests with HTTP method equals to POST or PUT
--any:
--selector:context.request.http.method
-operator:eq
-value:POST
--selector:context.request.http.method
-operator:eq
-value:PUT
-
-
iv) complex expression with nested operations:
-
spec:
-when:# auth enforced only on requests to POST /resources/* or PUT /resources/*
--any:
--all:
--selector:context.request.http.method
-operator:eq
-value:POST
--selector:context.request.http.path
-operator:matches
-value:^/resources/.*
--all:
--selector:context.request.http.method
-operator:eq
-value:PUT
--selector:context.request.http.path
-operator:matches
-value:^/resources/.*
-
-
v) more concise equivalent of the above (with implicit AND operator at the top level):
-
spec:
-when:# auth enforced only on requests to /resources/* path with method equals to POST or PUT
--selector:context.request.http.path
-operator:matches
-value:^/resources/.*
--any:
--selector:context.request.http.method
-operator:eq
-value:POST
--selector:context.request.http.method
-operator:eq
-value:PUT
-
-
vi) to skip part of an AuthConfig (i.e., a specific auth rule):
-
spec:
-metadata:
-"metadata-source":
-http:
-url:https://my-metadata-source.io
-when:# only fetch the external metadata if the context is HTTP method other than OPTIONS
--selector:context.request.http.method
-operator:neq
-value:OPTIONS
-
-
vii) skipping part of an AuthConfig will not affect other auth rules:
-
spec:
-authentication:
-"authn-meth-1":
-apiKey:{…}# this auth rule only triggers for POST requests to /foo[/*]
-when:
--selector:context.request.http.method
-operator:eq
-value:POST
--selector:context.request.http.path
-operator:matches
-value:^/foo(/.*)?$
-
-"authn-meth-2":# this auth rule triggerred regardless
-jwt:{…}
-
-
viii) concrete use-case: evaluating only the necessary identity checks based on the user's indication of the preferred authentication method (prefix of the value supplied in the HTTP Authorization request header):
Objects resolved at runtime in an Auth Pipeline can be cached "in-memory", and avoided being evaluated again at a subsequent request, until it expires. A lookup cache key and a TTL can be set individually for any evaluator config in an AuthConfig.
-
Each cache config induces a completely independent cache table (or "cache namespace"). Consequently, different evaluator configs can use the same cache key and there will be no collision between entries from different evaluators.
The example above sets caching for the 'external-metadata' metadata config and for the 'complex-policy' authorization policy. In the case of 'external-metadata', the cache key is the path of the original HTTP request being authorized by Authorino (fetched dynamically from the Authorization JSON); i.e., after obtaining a metadata object from the external source for a given contextual HTTP path one first time, whenever that same HTTP path repeats in a subsequent request, Authorino will use the cached object instead of sending a request again to the external source of metadata. After 5 minutes (300 seconds), the cache entry will expire and Authorino will fetch again from the source if requested.
-
As for the 'complex-policy' authorization policy, the cache key is a string composed the 'group' the identity belongs to, the method of the HTTP request and the path of the HTTP request. Whenever these repeat, Authorino will use the result of the policy that was evaluated and cached priorly. Cache entries in this namespace expire after 60 seconds.
-
Notes on evaluator caching
-
Capacity - By default, each cache namespace is limited to 1 mb. Entries will be evicted following First-In-First-Out (FIFO) policy to release space. The individual capacity of cache namespaces is set at the level of the Authorino instance (via --evaluator-cache-size command-line flag or spec.evaluatorCacheSize field of the Authorino CR).
-
Usage - Avoid caching objects whose evaluation is considered to be relatively cheap. Examples of operations associated to Authorino auth features that are usually NOT worth caching: validation of JSON Web Tokens (JWT), Kubernetes TokenReviews and SubjectAccessReviews, API key validation, simple JSON pattern-matching authorization rules, simple OPA policies. Examples of operations where caching may be desired: OAuth2 token introspection, fetching of metadata from external sources (via HTTP request), complex OPA policies.
By default, Authorino will only export metrics down to the level of the AuthConfig. Deeper metrics at the level of each evaluator within an AuthConfig can be activated by setting the common field metrics: true of the evaluator config.
The above will enable the metrics auth_server_evaluator_duration_seconds (histogram) and auth_server_evaluator_total (counter) with labels namespace="my-ns", authconfig="my-authconfig", evaluator_type="METADATA_GENERIC_HTTP" and evaluator_name="my-external-metadata".
-
The same pattern works for other types of evaluators. Find below the list of all types and corresponding label constant used in the metric:
-
-
-
-
Evaluator type
-
Metric's evaluator_type label
-
-
-
-
-
authentication.apiKey
-
IDENTITY_APIKEY
-
-
-
authentication.kubernetesTokenReview
-
IDENTITY_KUBERNETES
-
-
-
authentication.jwt
-
IDENTITY_OIDC
-
-
-
authentication.oauth2Introspection
-
IDENTITY_OAUTH2
-
-
-
authentication.x509
-
IDENTITY_MTLS
-
-
-
authentication.plain
-
IDENTITY_PLAIN
-
-
-
authentication.anonymous
-
IDENTITY_NOOP
-
-
-
metadata.http
-
METADATA_GENERIC_HTTP
-
-
-
metadata.userInfo
-
METADATA_USERINFO
-
-
-
metadata.uma
-
METADATA_UMA
-
-
-
authorization.patternMatching
-
AUTHORIZATION_JSON
-
-
-
authorization.opa
-
AUTHORIZATION_OPA
-
-
-
authorization.kubernetesSubjectAccessReview
-
AUTHORIZATION_KUBERNETES
-
-
-
authorization.spicedb
-
AUTHORIZATION_AUTHZED
-
-
-
response.success..plain
-
RESPONSE_PLAIN
-
-
-
response.success..json
-
RESPONSE_JSON
-
-
-
response.success..wristband
-
RESPONSE_WRISTBAND
-
-
-
-
Metrics at the level of the evaluators can also be enforced to an entire Authorino instance, by setting the --deep-metrics-enabled command-line flag. In this case, regardless of the value of the field spec.(authentication|metadata|authorization|response).metrics in the AuthConfigs, individual metrics for all evaluators of all AuthConfigs will be exported.
-
For more information about metrics exported by Authorino, see Observability.
This page covers requirements and instructions to deploy Authorino on a Kubernetes cluster, as well as the steps to declare, apply and try out a protection layer of authentication and authorization over your service, clean-up and complete uninstallation.
-
If you prefer learning with an example, check out our Hello World.
These are the platform requirements to use Authorino:
-
-
-
Kubernetes server (recommended v1.20 or later), with permission to create Kubernetes Custom Resource Definitions (CRDs) (for bootstrapping Authorino and Authorino Operator)
Envoy proxy (recommended v1.19 or later), to wire up Upstream services (i.e. the services to be protected with Authorino) and external authorization filter (Authorino) for integrations based on the reverse-proxy architecture - example
-
-Alternative: Non-reverse-proxy integration
-
Technically, any client that implements Envoy's external authorization gRPC protocol should be compatible with Authorino. For integrations based on the reverse-proxy architecture nevertheless, we strongly recommended that you leverage Envoy alongside Authorino.
-
For OpenID Connect, make sure you have access to an identity provider (IdP) and an authority that can issue ID tokens (JWTs). Check out Keycloak which can solve both and connect to external identity sources and user federation like LDAP.
-
-
-
For Kubernetes authentication tokens, platform support for the TokenReview and SubjectAccessReview APIs of Kubernetes shall be required. In case you want to be able to requests access tokens for clients running outside the custer, you may also want to check out the requisites for using Kubernetes TokenRequest API (GA in v1.20).
-
-
-
For User-Managed Access (UMA) resource data, you will need a UMA-compliant server running as well. This can be an implementation of the UMA protocol by each upstream API itself or (more typically) an external server that knows about the resources. Again, Keycloak can be a good fit here as well. Just keep in mind that, whatever resource server you choose, changing-state actions commanded in the upstream APIs or other parties will have to be reflected in the resource server. Authorino will not do that for you.
The above will install the latest build of the Authorino Operator and latest version of the manifests (CRDs and RBAC), which by default points as well to the latest build of Authorino, both based on the main branches of each component. To install a stable released version of the Operator and therefore also defaults to its latest compatible stable release of Authorino, replace main with another tag of a proper release of the Operator, e.g. 'v0.2.0'.
-
This step will also install cert-manager in the cluster (required).
-
Alternatively, you can deploy the Authorino Operator using the Operator Lifecycle Manager bundles. For instructions, check out Installing via OLM.
Choose either cluster-wide or namespaced deployment mode and whether you want TLS termination enabled for the Authorino endpoints (gRPC authorization, raw HTTP authorization, and OIDC Festival Wristband Discovery listeners), and follow the corresponding instructions below.
-
The instructions here are for centralized gateway or centralized authorization service architecture. Check out the Topologies section of the docs for alternatively running Authorino in a sidecar container.
-
-Cluster-wide (with TLS)
-
Create the namespace:
-
kubectlcreatenamespaceauthorino
-
-
Create the TLS certificates (requires cert-manager; skip if you already have certificates and certificate keys created and stored in Kubernetes Secrets in the namespace):
-
Create the TLS certificates (requires cert-manager; skip if you already have certificates and certificate keys created and stored in Kubernetes Secrets in the namespace):
-
The most typical integration to protect services with Authorino is by putting the service (upstream) behind a reverse-proxy or API gateway, enabled with an authorization filter that ensures all requests to the service are first checked with the authorization server (Authorino).
-
To do that, make sure you have your upstream service deployed and running, usually in the same Kubernetes server where you installed Authorino. Then, setup an Envoy proxy and create an Authorino AuthConfig for your service.
-
Authorino exposes 2 interfaces to serve the authorization requests:
a raw HTTP authorization interface, suitable for using Authorino with Kubernetes ValidatingWebhook, for Envoy external authorization via HTTP, and other integrations (e.g. other proxies).
-
-
To use Authorino as a simple satellite (sidecar) Policy Decision Point (PDP), applications can integrate directly via any of these interfaces. By integrating via a proxy or API gateway, the combination makes Authorino to perform as an external Policy Enforcement Point (PEP) completely decoupled from the application.
To configure Envoy for proxying requests targeting the upstream service and authorizing with Authorino, setup an Envoy configuration that enables Envoy's external authorization HTTP filter. Store the configuration in a ConfigMap.
-
These are the important bits in the Envoy configuration to activate Authorino:
-
static_resources:
-listeners:
--address:{…}# TCP socket address and port of the proxy
-filter_chains:
--filters:
--name:envoy.http_connection_manager
-typed_config:
-"@type":type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
-route_config:{…}# routing configs - virtual host domain and endpoint matching patterns and corresponding upstream services to redirect the traffic
-http_filters:
--name:envoy.filters.http.ext_authz# the external authorization filter
-typed_config:
-"@type":type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
-transport_api_version:V3
-failure_mode_allow:false# ensures only authenticated and authorized traffic goes through
-grpc_service:
-envoy_grpc:
-cluster_name:authorino
-timeout:1s
-clusters:
--name:authorino
-connect_timeout:0.25s
-type:strict_dns
-lb_policy:round_robin
-http2_protocol_options:{}
-load_assignment:
-cluster_name:authorino
-endpoints:
--lb_endpoints:
--endpoint:
-address:
-socket_address:
-address:authorino-authorino-authorization# name of the Authorino service deployed – it can be the fully qualified name with `.<namespace>.svc.cluster.local` suffix (e.g. `authorino-authorino-authorization.myapp.svc.cluster.local`)
-port_value:50051
-transport_socket:# in case TLS termination is enabled in Authorino; omit it otherwise
-name:envoy.transport_sockets.tls
-typed_config:
-"@type":type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
-common_tls_context:
-validation_context:
-trusted_ca:
-filename:/etc/ssl/certs/authorino-ca-cert.crt
-
-
For a complete Envoy ConfigMap containing an upstream API protected with Authorino, with TLS enabled and option for rate limiting with Limitador, plus a webapp served with under the same domain of the protected API, check out this example.
-
After creating the ConfigMap with the Envoy configuration, create an Envoy Deployment and Service. E.g.:
-
kubectl-nmyappapply-f-<<EOF
-apiVersion: apps/v1
-kind: Deployment
-metadata:
- name: envoy
- labels:
- app: envoy
-spec:
- selector:
- matchLabels:
- app: envoy
- template:
- metadata:
- labels:
- app: envoy
- spec:
- containers:
- - name: envoy
- image: envoyproxy/envoy:v1.19-latest
- command: ["/usr/local/bin/envoy"]
- args:
- - --config-path /usr/local/etc/envoy/envoy.yaml
- - --service-cluster front-proxy
- - --log-level info
- - --component-log-level filter:trace,http:debug,router:debug
- ports:
- - name: web
- containerPort: 8000 # matches the address of the listener in the envoy config
- volumeMounts:
- - name: config
- mountPath: /usr/local/etc/envoy
- readOnly: true
- - name: authorino-ca-cert # in case TLS termination is enabled in Authorino; omit it otherwise
- subPath: ca.crt
- mountPath: /etc/ssl/certs/authorino-ca-cert.crt
- readOnly: true
- volumes:
- - name: config
- configMap:
- name: envoy
- items:
- - key: envoy.yaml
- path: envoy.yaml
- - name: authorino-ca-cert # in case TLS termination is enabled in Authorino; omit it otherwise
- secret:
- defaultMode: 420
- secretName: authorino-ca-cert
- replicas: 1
-EOF
-
Check out the docs for a full description of Authorino's AuthConfig Custom Resource Definition (CRD) and its features.
-
For examples based on specific use-cases, check out the User guides.
-
For authentication based on OpenID Connect (OIDC) JSON Web Tokens (JWT), plus one simple JWT claim authorization check, a typical AuthConfig custom resource looks like the following:
-
kubectl-nmyappapply-f-<<EOF
-apiVersion: authorino.kuadrant.io/v1beta2
-kind: AuthConfig
-metadata:
- name: my-api-protection
-spec:
- hosts: # any hosts that resolve to the envoy service and envoy routing config where the external authorization filter is enabled
- - my-api.io # north-south traffic through a Kubernetes `Ingress` or OpenShift `Route`
- - my-api.myapp.svc.cluster.local # east-west traffic (between applications within the cluster)
- authentication:
- "idp-users":
- jwt:
- issuerUrl: https://my-idp.com/auth/realm
- authorization:
- "check-claim":
- patternMatching:
- patterns:
- - selector: auth.identity.group
- operator: eq
- value: allowed-users
-EOF
-
-
After applying the AuthConfig, consumers of the protected service should be able to start sending requests.
Here we define some terms that are used in the project, with the goal of avoiding confusion and facilitating more
-accurate conversations related to Authorino.
-
If you see terms used that are not here (or are used in place of terms here) please consider contributing a definition
-to this doc with a PR, or modifying the use elsewhere to align with these terms.
Access token
-Type of temporary password (security token), tied to an authenticated identity, issued by an auth server as of request from either the identity subject itself or a registered auth client known by the auth server, and that delegates to a party powers to operate on behalf of that identity before a resource server; it can be formatted as an opaque data string or as an encoded JSON Web Token (JWT).
-
Application Programming Interface (API)
-Interface that defines interactions between multiple software applications; (in HTTP communication) set of endpoints and specification to expose resources hosted by a resource server, to be consumed by client applications; the access facade of a resource server.
-
Attribute-based Access Control (ABAC)
-Authorization model that grants/denies access to resources based on evaluation of authorization policies which combine attributes together (from claims, from the request, from the resource, etc).
-
Auth
-Usually employed as a short for authentication and authorization together (AuthN/AuthZ).
-
Auth client
-Application client (software) that uses an auth server, either in the process of authenticating and/or authorizing identity subjects (including self) who want to consume resources from a resources server or auth server.
-
Auth server
-Server where auth clients, users, roles, scopes, resources, policies and permissions can be stored and managed.
-
Authentication (AuthN)
-Process of verifying that a given credential belongs to a claimed-to-be identity; usually resulting in the issuing of an access token.
-
Authorization (AuthZ)
-Process of granting (or denying) access over a resource to a party based on the set of authorization rules, policies and/or permissions enforced.
-
Authorization header
-HTTP request header frequently used to carry credentials to authenticate a user in an HTTP communication, like in requests sent to an API; alternatives usually include credentials carried in another (custom) HTTP header, query string parameter or HTTP cookie.
-
Capability
-Usually employed to refer to a management feature of a Kubernetes-native system, based on the definition and use of Kubernetes Custom Resources (CRDs and CRs), that enables that system to one of the following “capability levels”: Basic Install, Seamless Upgrades, Full Lifecycle, Deep Insights, Auto Pilot.
-
Claim
-Attribute packed in a security token which represents a claim that one who bears the token is making about an entity, usually an identity subject.
-
Client ID
-Unique identifier of an auth client within an auth server domain (or auth server realm).
-
Client secret
-Password presented by auth clients together with their Client IDs while authenticating with an auth server, either when requesting access tokens to be issued or when consuming services from the auth servers in general.
-
Delegation
-Process of granting a party (usually an auth client) with powers to act, often with limited scope, on behalf of an identity, to access resources from a resource server. See also OAuth2.
-
Hash-based Message Authentication Code (HMAC)
-Specific type of message authentication code (MAC) that involves a cryptographic hash function and a shared secret cryptographic key; it can be used to verify the authenticity of a message and therefore as an authentication method.
-
Identity
-Set of properties that qualifies a subject as a strong identifiable entity (usually a user), who can be authenticated by an auth server. See also Claims.
-
Identity and Access Management (IAM) system
-Auth system that implements and/or connects with sources of identity (IdP) and offers interfaces for managing access (authorization policies and permissions). See also Auth server.
-
Identity Provider (IdP)
-Source of identity; it can be a feature of an auth server or external source connected to an auth server.
-
ID token
-Special type of access token; an encoded JSON Web Token (JWT) that packs claims about an identity.
-
JSON Web Token (JWT)
-JSON Web Tokens are an open, industry standard RFC 7519 method for representing claims securely between two parties.
-
JSON Web Signature (JWS)
-Standard for signing arbitrary data, especially JSON Web Tokens (JWT).
-
JSON Web Key Set (JWKS)
-Set of keys containing the public keys used to verify any JSON Web Token (JWT).
-
Keycloak
-Open source auth server to allow single sign-on with identity and access management.
-
Lightweight Directory Access Protocol (LDAP)
-Open standard for distributed directory information services for sharing of information about users, systems, networks, services and applications.
-
Mutual Transport Layer Security (mTLS)
-Protocol for the mutual authentication of client-server communication, i.e., the client authenticates the server and the server authenticates the client, based on the acceptance of the X.509 certificates of each party.
OpenID Connect (OIDC)
-Simple identity verification (authentication) layer built on top of the OAuth2 protocol.
-
Open Policy Agent (OPA)
-Authorization policy agent that enables the usage of declarative authorization policies written in Rego language.
-
Opaque token
-Security token devoid of explicit meaning (e.g. random string); it requires the usage of lookup mechanism to be translated into a meaningful set claims representing an identity.
-
Permission
-Association between a protected resource the authorization policies that must be evaluated whether access should be granted; e.g. <user|group|role> CAN DO <action> ON RESOURCE <X>.
-
Policy
-Rule or condition (authorization policy) that must be satisfied to grant access to a resource; strongly related to the different access control mechanisms (ACMs) and strategies one can use to protect resources, e.g. attribute-based access control (ABAC), role-based access control (RBAC), context-based access control, user-based access control (UBAC).
-
Policy Administration Point (PAP)
-Set of UIs and APIs to manage resources servers, resources, scopes, policies and permissions; it is where the auth system is configured.
-
Policy Decision Point (PDP)
-Where the authorization requests are sent, with permissions being requested, and authorization policies are evaluated accordingly.
-
Policy Enforcement Point (PEP)
-Where the authorization is effectively enforced, usually at the resource server or at a proxy, based on a response provided by the Policy Decision Point (PDP).
-
Policy storage
-Where policies are stored and from where they can be fetched, perhaps to be cached.
-
Red Hat SSO
-Auth server; downstream product created from the Keycloak Open Source project.
-
Refresh token
-Special type of security token, often provided together with an access token in an OAuth2 flow, used to renew the duration of an access token before it expires; it requires client authentication.
-
Request Party Token (RPT)
-JSON Web Token (JWT) digitally signed using JSON Web Signature (JWS), issued by the Keycloak auth server.
-
Resource
-One or more endpoints of a system, API or server, that can be protected.
-
Resource-level Access Control (RLAC)
-Authorization model that takes into consideration attributes of each specific request resource to grant/deny access to those resources (e.g. the resource's owner).
-
Resource server
-Server that hosts protected resources.
-
Role
-Aspect of a user’s identity assigned to the user to indicate the level of access they should have to the system; essentially, roles represent collections of permissions
-
Role-based Access Control (RBAC)
-Authorization model that grants/denies access to resources based on the roles of authenticated users (rather than on complex attributes/policy rules).
-
Scope
-Mechanism that defines the specific operations that applications can be allowed to do or information that they can request on an identity’s behalf; often presented as a parameter when access is requested as a way to communicate what access is needed, and used by auth server to respond what actual access is granted.
-
Single Page Application (SPA)
-Web application or website that interacts with the user by dynamically rewriting the current web page with new data from the web server.
-
Single Sign-on (SSO)
-Authentication scheme that allows a user to log in with a single ID and password to any of several related, yet independent, software systems.
-
Upstream
-(In the context of authentication/authorization) API whose endpoints must be protected by the auth system; the unprotected service in front of which a protection layer is added (by connecting with a Policy Decision Point).
-
User-based Access Control (UBAC)
-Authorization model that grants/denies access to resources based on claims of the identity (attributes of the user).
-
User-Managed Access (UMA)
-OAuth2-based access management protocol, used for users of an auth server to control the authorization process, i.e. directly granting/denying access to user-owned resources to other requesting parties.
Kubernetes server with permissions to install cluster-scoped resources (operator, CRDs and RBAC)
-
-
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
-
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❺.
-
At step ❺, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
The example above enables anonymous access (i.e. removes authentication), without adding any extra layer of protection to the API. This is virtually equivalent to setting a top-level condition to the AuthConfig that always skips the configuration, or to switching authentication/authorization off completely in the route to the API.
-
For more sophisticated use cases of anonymous access with Authorino, consider combining this feature with other identity sources in the AuthConfig while playing with the priorities of each source, as well as combination with whenconditions, and/or adding authorization policies that either cover authentication or address anonymous access with proper rules (e.g. enforcing read-only access).
-
Check out the docs for the Anonymous access feature for an example of an AuthConfig that falls back to anonymous access when a priority OIDC/JWT-based authentication fails, and enforces a read-only policy in such cases.
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
In Authorino, API keys are stored as Kubernetes Secrets. Each resource must contain an api_key entry with the value of the API key, and labeled to match the selectors specified in spec.identity.apiKey.selector of the AuthConfig.
-
API key Secrets must also include labels that match the secretLabelSelector field of the Authorino instance. See Resource reconciliation and status update for details.
-
For further details about Authorino features in general, check the docs.
Kubernetes server with permissions to install cluster-scoped resources (operator, CRDs and RBAC)
-
-
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
-
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❺.
-
At step ❺, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
Dynamic JSON objects built out of static values and values fetched from the Authorization JSON can be wrapped to be returned to the reverse-proxy as Envoy Well Known Dynamic Metadata content. Envoy can use those to inject data returned by the external authorization service into the other filters, such as the rate limiting filter.
Kubernetes server with permissions to install cluster-scoped resources (operator, CRDs and RBAC)
-
-
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
-
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❻.
-
At step ❻, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
Limitador is a lightweight rate limiting service that can be used with Envoy.
-
On this bundle, we will deploy Limitador pre-configured to limit requests to the talker-api domain up to 5 requests per interval of 60 seconds per user_id. Envoy will be configured to recognize the presence of Limitador and activate it on requests to the Talker API.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
Create an Authorino AuthConfig custom resource declaring the auth rules to be enforced.
-
An annotation auth-data/username will be read from the Kubernetes API Key secret and passed as dynamic metadata { "ext_auth_data": { "username": «annotations.auth-data/username» } }.
-
-
-
-
- Kuadrant users –
- Remember to create an AuthPolicy instead of an AuthConfig.
- For more, see Kuadrant auth.
-
-
-
-
-
-
kubectlapply-f-<<EOF
-apiVersion: authorino.kuadrant.io/v1beta2
-kind: AuthConfig
-metadata:
- name: talker-api-protection
-spec:
- hosts:
- - talker-api.127.0.0.1.nip.io
- authentication:
- "friends":
- apiKey:
- selector:
- matchLabels:
- group: friends
- credentials:
- authorizationHeader:
- prefix: APIKEY
- response:
- success:
- dynamicMetadata:
- "rate-limit":
- json:
- properties:
- "username":
- selector: auth.identity.metadata.annotations.auth-data\/username
- key: ext_auth_data # how this bit of dynamic metadata from the ext authz service is named in the Envoy config
-EOF
-
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
Kubernetes server with permissions to install cluster-scoped resources (operator, CRDs and RBAC)
-
-
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
-
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❻.
-
At step ❻, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
Cache auth objects resolved at runtime for any configuration bit of an AuthConfig (i.e. any evaluator), of any phase (identity, metadata, authorization and dynamic response), for easy access in subsequent requests, whenever an arbitrary (user-defined) cache key repeats, until the cache entry expires.
-
This is particularly useful for configuration bits whose evaluation is significantly more expensive than accessing the cache. E.g.:
-
-
Caching of metadata fetched from external sources in general
-
Caching of previously validated identity access tokens (e.g. for OAuth2 opaque tokens that involve consuming the token introspection endpoint of an external auth server)
-
Caching of complex Rego policies that involve sending requests to external services
-
-
Cases where one will NOT want to enable caching, due to relatively cheap compared to accessing and managing the cache:
-
-
Validation of OIDC/JWT access tokens
-
OPA/Rego policies that do not involve external requests
-
JSON pattern-matching authorization
-
Dynamic JSON responses
-
Anonymous access
-
-
-
- Authorino capabilities featured in this guide:
-
Kubernetes server with permissions to install cluster-scoped resources (operator, CRDs and RBAC)
-
-
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
-
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❺.
-
At step ❺, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
Create an Authorino AuthConfig custom resource declaring the auth rules to be enforced.
-
The example below enables caching for the external source of metadata, which in this case, for convenience, is the same upstream API protected by Authorino (i.e. the Talker API), though consumed directly by Authorino, without passing through the proxy. This API generates a uuid random hash that it injects in the JSON response. This value is different in every request processed by the API.
-
The example also enables caching of returned OPA virtual documents. cached-authz is a trivial Rego policy that always grants access, but generates a timestamp, which Authorino will cache.
-
In both cases, the path of the HTTP request is used as cache key. I.e., whenever the path repeats, Authorino reuse the values stored previously in each cache table (cached-metadata and cached-authz), respectively saving a request to the external source of metadata and the evaluation of the OPA policy. Cache entries will expire in both cases after 60 seconds they were stored in the cache.
-
The cached values will be visible in the response returned by the Talker API in x-authz-data header injected by Authorino. This way, we can tell when an existing value in the cache was used and when a new one was generated and stored.
-
-
-
-
- Kuadrant users –
- Remember to create an AuthPolicy instead of an AuthConfig.
- For more, see Kuadrant auth.
-
To /hello again before the cache entry expires (60 seconds from the first request sent to this path)
-
-
curlhttp://talker-api.127.0.0.1.nip.io:8000/hello
-# […]
-# "X-Authz-Data": "{\"cached-authz\":\"1649343067462380300\",\"cached-metadata\":\"92c111cd-a10f-4e86-8bf0-e0cd646c6f79\"}", <=== same cache-id as before
-# […]
-
-
-
To /hello again after the cache entry expires (60 seconds from the first request sent to this path)
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
Customize response status code and headers on failed requests to redirect users of a web application protected with Authorino to a login page instead of a 401 Unauthorized.
-
-
- Authorino capabilities featured in this guide:
-
Authorino's default response status codes, messages and headers for unauthenticated (401) and unauthorized (403) requests can be customized with static values and values fetched from the Authorization JSON.
Kubernetes server with permissions to install cluster-scoped resources (operator, CRDs and RBAC)
-
-
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
-
The next steps walk you through installing Authorino, deploying and configuring a sample web application called Matrix Quotes to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❺.
-
At step ❺, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Matrix Quotes webapp behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name matrix-quotes.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
Resolve local Keycloak domain so it can be accessed from the local host and inside the cluster with the name: (This will be needed to redirect to Keycloak's login page and at the same time validate issued tokens.)
-
echo'127.0.0.1 keycloak'>>/etc/hosts
-
-
Forward local requests to the instance of Keycloak running in the cluster:
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
User guide: Edge Authentication Architecture (EAA)¶
-
Edge Authentication Architecture (EAA) is a pattern where more than extracting authentication logics and specifics from the application codebase to a proper authN/authZ layer, this is pushed to the edge of your cloud network, without violating the Zero Trust principle nevertheless.
-
The very definition of "edge" is subject to discussion, but the underlying idea is that clients (e.g. API clients, IoT devices, etc.) authenticate with a layer that, before moving traffic to inside the network:
-- understands the complexity of all the different methods of authentication supported;
-- sometimes some token normalization is involved;
-- eventually enforces some preliminary authorization policies; and
-- possibly filters data bits that are sensitive to privacy concerns (e.g. to comply with local legislation such as GRPD, CCPA, etc)
-
As a minimum, EAA allows to simplify authentication between applications and microservices inside the network, as well as to reduce authorization to domain-specific rules and policies, rather than having to deal all the complexity to support all types of clients in every node.
-
-
- Authorino capabilities featured in this guide:
-
Festival Wristbands are OpenID Connect ID tokens (signed JWTs) issued by Authorino by the end of the Auth Pipeline, for authorized requests. It can be configured to include claims based on static values and values fetched from the Authorization JSON.
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
Deploy the identity provider and authentication server by executing the command below. For the examples in this guide, we are going to use a Keycloak server preloaded with all required realm settings.
The next steps walk you through installing Authorino and configuring 2 environments of an architecture, edge and internal.
-
The first environment is a facade for handling the first layer of authentication and exchanging any valid presented authentication token for a Festival Wristband token. In the second, we will deploy a sample service called Talker API that the authorization service will ensure to receive only authenticated traffic presented with a valid Festival Wristband.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❹.
-
At steps ❹ and ❺, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
For simplicity, this examples will set up edge and internal nodes in different namespaces of the same Kubernetes cluster. Those will share a same single cluster-wide Authorino instance. In real-life scenarios, it does not have to be like that.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources cluster-wide2, with TLS disabled3.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up external authorization with the Authorino instance.4
The command above creates an Ingress with host name edge.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 9000 to the Envoy service running inside the cluster:
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
This other bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
Obtain an access token with the Keycloak server for Jane:
-
The AuthConfig deployed in the previous step is suitable for validating access tokens requested inside the cluster. This is because Keycloak's iss claim added to the JWTs matches always the host used to request the token and Authorino will later try to match this host to the host that provides the OpenID Connect configuration.
-
Obtain an access token from within the cluster for the user Jane, whose e-mail has been verified:
If your Keycloak server is reachable from outside the cluster, feel free to obtain the token directly. Make sure the host name set in the OIDC issuer endpoint in the AuthConfig matches the one used to obtain the token and is as well reachable from within the cluster.
-
(Optional) Inspect the access token issue by Keycloak and verify and how it contains more details about the identity than required to authenticate and authorize with internal apps.
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
User guide: Mixing Envoy built-in filter for auth and Authorino¶
-
Have JWT validation handled by Envoy beforehand and the JWT payload injected into the request to Authorino, to be used in custom authorization policies defined in a AuthConfig.
-
In this user guide, we will set up Envoy and Authorino to protect a service called the Talker API service, with JWT authentication handled in Envoy and a more complex authorization policy enforced in Authorino.
-
The policy defines a geo-fence by which only requests originated in Great Britain (country code: GB) will be accepted, unless the user is bound to a role called 'admin' in the auth server, in which case no geofence is enforced.
-
All requests to the Talker API will be authenticated in Envoy. However, requests to /global will not trigger the external authorization.
-
-
- Authorino capabilities featured in this guide:
-
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
Deploy the identity provider and authentication server by executing the command below. For the examples in this guide, we are going to use a Keycloak server preloaded with all required realm settings.
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❻.
-
At step ❻, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
The following command deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
Obtain an access token and consume the API as John (member)¶
-
Obtain an access token with the Keycloak server for John:
-
The AuthConfig deployed in the previous step is suitable for validating access tokens requested inside the cluster. This is because Keycloak's iss claim added to the JWTs matches always the host used to request the token and Authorino will later try to match this host to the host that provides the OpenID Connect configuration.
-
Obtain an access token from within the cluster for the user John, a non-admin (member) user:
If your Keycloak server is reachable from outside the cluster, feel free to obtain the token directly. Make sure the host name set in the OIDC issuer endpoint in the AuthConfig matches the one used to obtain the token and is as well reachable from within the cluster.
-
As John, consume the API inside the area where the policy applies:
As John, consume the API outside the area where the policy applies:
-
curl-H"Authorization: Bearer $ACCESS_TOKEN"\
--H'X-Forwarded-For: 109.69.200.56'\
-http://talker-api.127.0.0.1.nip.io:8000-i
-# HTTP/1.1 403 Forbidden
-# x-ext-auth-reason: The requested resource is not available in Italy
-
-
As John, consume a path of the API that will cause Envoy to skip external authorization:
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
You can configure Authorino to fetch additional metadata from external sources in request-time, by sending either GET or POST request to an HTTP service. The service is expected to return a JSON content which is appended to the Authorization JSON, thus becoming available for usage in other configs of the Auth Pipeline, such as in authorization policies or custom responses.
-
URL, parameters and headers of the request to the external source of metadata can be configured, including with dynamic values. Authentication between Authorino and the service can be set as part of these configuration options, or based on shared authentication token stored in a Kubernetes Secret.
Kubernetes server with permissions to install cluster-scoped resources (operator, CRDs and RBAC)
-
-
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
-
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❺.
-
At step ❺, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
Create an Authorino AuthConfig custom resource declaring the auth rules to be enforced.
-
In this example, we will implement a geofence policy for the API, using OPA and metadata fetching from an external service that returns geolocalization JSON data for a given IP address. The policy establishes that only GET requests are allowed and the path of the request should be in the form /{country-code}/*, where {country-code} is the 2-character code of the country where the client is identified as being physically present.
-
The implementation relies on the X-Forwarded-For HTTP header to read the client's IP address.
-
-
-
-
- Kuadrant users –
- Remember to create an AuthPolicy instead of an AuthConfig.
- For more, see Kuadrant auth.
-
Check out the docs for information about the common feature JSON paths for reading from the Authorization JSON, including the description of the @extract string modifier.
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
Kubernetes server with permissions to install cluster-scoped resources (operator, CRDs and RBAC)
-
-
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant, you can skip step ❸. You may already have Authorino installed and running as well. In this case, skip also step ❺. If you even have your workload cluster configured, with sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, go straight to step ❼.
-
At step ❼, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
-
kubectl-nhello-worldapply-fhttps://raw.githubusercontent.com/kuadrant/authorino-examples/main/talker-api/talker-api-deploy.yaml
-# deployment.apps/talker-api created
-# service/talker-api created
-
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.1
-
kubectl-nhello-worldapply-fhttps://raw.githubusercontent.com/kuadrant/authorino-examples/main/hello-world/envoy-deploy.yaml
-# configmap/envoy created
-# deployment.apps/envoy created
-# service/envoy created
-
-
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
The following command will request an instance of Authorino as a separate service2 that watches for AuthConfig resources in the hello-world namespace3, with TLS disabled4.
-
kubectl-nhello-worldapply-fhttps://raw.githubusercontent.com/kuadrant/authorino-examples/main/hello-world/authorino.yaml
-# authorino.operator.authorino.kuadrant.io/authorino created
-
curlhttp://talker-api.127.0.0.1.nip.io:8000/hello-i
-# HTTP/1.1 404 Not Found
-# x-ext-auth-reason: Service not found
-
-
Authorino does not know about the talker-api.127.0.0.1.nip.io host, hence the 404 Not Found. Let's teach Authorino about this host by applying an AuthConfig.
Create an Authorino AuthConfig custom resource declaring the auth rules to be enforced:
-
-
-
-
- Kuadrant users –
- Remember to create an AuthPolicy instead of an AuthConfig.
- For more, see Kuadrant auth.
-
-
-
-
-
-
kubectl-nhello-worldapply-fhttps://raw.githubusercontent.com/kuadrant/authorino-examples/main/hello-world/authconfig.yaml
-# authconfig.authorino.kuadrant.io/talker-api-protection created
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
-
-
-
In contrast to a dedicated sidecar of the protected service and other architectures. Check out Architecture > Topologies for all options. ↩
By default, Authorino uses the host information of the HTTP request (Attributes.Http.Host) to lookup for an indexed AuthConfig to be enforced1. The host info be overridden by supplying a host entry as a (per-route) context extension (Attributes.ContextExtensions), which takes precedence whenever present.
-
Overriding the host attribute of the HTTP request can be useful to support use cases such as of path prefix-based lookup and wildcard subdomains lookup.
-
-
-
-
- ⚠️ Important:
- This feature may not be available to users of Authorino via Kuadrant.
-
Example of host override for path prefix-based lookup¶
-
In this use case, 2 different APIs (i.e. Dogs API and Cats API) are served under the same base domain, and differentiated by the path prefix:
-
-
pets.com/dogs → Dogs API
-
pets.com/cats → Cats API
-
-
Edit the Envoy config to extend the external authorization settings at the level of the routes, with the host value that will be favored by Authorino before the actual host attribute of the HTTP request:
Notice that the host subdomains dogs.pets.com and cats.pets.com are not really requested by the API consumers. Rather, users send requests to pets.com/dogs and pets.com/cats. When routing those requests, Envoy makes sure to inject the corresponding context extensions that will induce the right lookup in Authorino.
-
Example of host override for wildcard subdomain lookup¶
-
In this use case, a single Pets API serves requests for any subdomain that matches *.pets.com, e.g.:
-
-
dogs.pets.com → Pets API
-
cats.pets.com → Pets API
-
-
Edit the Envoy config to extend the external authorization settings at the level of the virtual host, with the host value that will be favored by Authorino before the actual host attribute of the HTTP request:
Notice that requests to dogs.pets.com and to cats.pets.com are all routed by Envoy to the same API, with same external authorization configuration. in all the cases, Authorino will lookup for the indexed AuthConfig associated with pets.com. The same is valid for a request sent, e.g., to birds.pets.com.
-
-
-
-
-
For further details about Authorino lookup of AuthConfig, check out Host lookup. ↩
HTTP "Basic" Authentication (RFC 7235) is not recommended if you can afford other more secure methods such as OpenID Connect. To support legacy nonetheless it is sometimes necessary to implement it.
-
In Authorino, HTTP "Basic" Authentication can be modeled leveraging the API key authentication feature (stored as Kubernetes Secrets with an api_key entry and labeled to match selectors specified in spec.identity.apiKey.selector of the AuthConfig).
Kubernetes server with permissions to install cluster-scoped resources (operator, CRDs and RBAC)
-
-
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
-
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❺.
-
At step ❺, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
Create an Authorino AuthConfig custom resource declaring the auth rules to be enforced.
-
The config uses API Key secrets to store base64-encoded username:password HTTP "Basic" authentication credentials. The config also specifies an Access Control List (ACL) by which only user john is authorized to consume the /bye endpoint of the API.
-
-
-
-
- Kuadrant users –
- Remember to create an AuthPolicy instead of an AuthConfig.
- For more, see Kuadrant auth.
-
Check out the docs for information about the common feature JSON paths for reading from the Authorization JSON, including the description of the string modifiers @extract and @case used above. Check out as well the common feature Conditions about skipping parts of an AuthConfig in the auth pipeline based on context.
To create credentials for HTTP "Basic" Authentication, store each username:password, base64-encoded, in the api_key value of the Kubernetes Secret resources. E.g.:
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
OpenID Connect Discovery and authentication with JWTs
-Validate JSON Web Tokens (JWT) issued and signed by an OpenID Connect server; leverage OpenID Connect Discovery to automatically fetch JSON Web Key Sets (JWKS).
Anonymous access
-Bypass identity verification or fall back to anonymous access when credentials fail to validate
-
-
-
Token normalization
-Normalize identity claims from trusted sources and reduce complexity in your policies.
-
-
-
Edge Authentication Architecture (EAA)
-Exchange satellite (outer-layer) authentication tokens for "Festival Wristbands" accepted ubiquitously at the inside of your network. Normalize from multiple and varied sources of identity and authentication methods in the edge of your architecture; filter privacy data, limit the scope of permissions, and simplify authorization rules to your internal microservices.
OpenID Connect UserInfo
-Fetch user info for OpenID Connect ID tokens in request-time for extra metadata for your policies and online verification of token validity.
Simple pattern-matching authorization policies
-Write simple authorization rules based on JSON patterns matched against Authorino's Authorization JSON; check contextual information of the request, validate JWT claims, cross metadata fetched from external sources, etc.
Open Policy Agent (OPA) Rego policies
-Leverage the power of Open Policy Agent (OPA) policies, evaluated against Authorino's Authorization JSON in a built-in runtime compiled together with Authorino; pre-cache policies defined in Rego language inline or fetched from an external policy registry.
Redirecting to a login page
-Customize response status code and headers on failed requests. E.g. redirect users of a web application protected with Authorino to a login page instead of a 401 Unauthorized; mask resources on access denied behind a 404 Not Found response instead of 403 Forbidden.
-
-
-
Mixing Envoy built-in filter for auth and Authorino
-Have JWT validation handled by Envoy beforehand and the JWT payload injected into the request to Authorino, to be used in custom authorization policies defined in a AuthConfig.
-
-
-
Host override via context extension
-Induce the lookup of an AuthConfig by supplying extended host context, for use cases such as of path prefix-based lookup and wildcard subdomains lookup.
-
-
-
Using Authorino as ValidatingWebhook service
-Use Authorino as a generic Kubernetes ValidatingWebhook service where the rules to validate a request to the Kubernetes API are written in an AuthConfig.
Caching
-Cache auth objects resolved at runtime for any configuration bit of an AuthConfig, for easy access in subsequent requests whenever an arbitrary cache key repeats, until the cache entry expires.
-
-
-
Observability
-Prometheus metrics exported by Authorino, readiness probe, logging, tracing, etc.
Kubernetes server with permissions to install cluster-scoped resources (operator, CRDs and RBAC)
-
-
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
-
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❺.
-
At step ❺, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
Create an Authorino AuthConfig custom resource declaring the auth rules to be enforced.
-
The following defines a JSON object to be injected as an added HTTP header into the request, named after the response config x-ext-auth-data. The object includes 3 properties:
-1. a static value authorized: true;
-2. a dynamic value request-time, from Envoy-supplied contextual data present in the Authorization JSON; and
-3. a greeting message geeting-message that interpolates a dynamic value read from an annotation of the Kubernetes Secret resource that represents the API key used to authenticate into a static string.
-
-
-
-
- Kuadrant users –
- Remember to create an AuthPolicy instead of an AuthConfig.
- For more, see Kuadrant auth.
-
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
User guide: Simple pattern-matching authorization policies¶
-
Write simple authorization rules based on JSON patterns matched against Authorino's Authorization JSON; check contextual information of the request, validate JWT claims, cross metadata fetched from external sources, etc.
-
-
- Authorino capabilities featured in this guide:
-
Authorino provides a built-in authorization module to check simple pattern-matching rules against the Authorization JSON. This is an alternative to OPA when all you want is to check for some simple rules, without complex logics, such as match the value of a JWT claim.
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
Deploy the identity provider and authentication server by executing the command below. For the examples in this guide, we are going to use a Keycloak server preloaded with all required realm settings.
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❺.
-
At step ❺, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
Create an Authorino AuthConfig custom resource declaring the auth rules to be enforced.
-
The email-verified-only authorization policy ensures that users consuming the API from a given network (IP range 192.168.1/24) must have their emails verified.
-
The email_verified claim is a property of the identity added to the JWT by the OpenID Connect issuer.
-
The implementation relies on the X-Forwarded-For HTTP header to read the client's IP address.
-
-
-
-
- Kuadrant users –
- Remember to create an AuthPolicy instead of an AuthConfig.
- For more, see Kuadrant auth.
-
Check out the docs for information about semantics and operators supported by the JSON pattern-matching authorization feature, as well the common feature JSON paths for reading from the Authorization JSON, including the description of the string modifier @extract used above. Check out as well the common feature Conditions about skipping parts of an AuthConfig in the auth pipeline based on context.
Obtain an access token and consume the API as Jane (email verified)¶
-
Obtain an access token with the Keycloak server for Jane:
-
The AuthConfig deployed in the previous step is suitable for validating access tokens requested inside the cluster. This is because Keycloak's iss claim added to the JWTs matches always the host used to request the token and Authorino will later try to match this host to the host that provides the OpenID Connect configuration.
-
Obtain an access token from within the cluster for the user Jane, whose e-mail has been verified:
If your Keycloak server is reachable from outside the cluster, feel free to obtain the token directly. Make sure the host name set in the OIDC issuer endpoint in the AuthConfig matches the one used to obtain the token and is as well reachable from within the cluster.
-
As Jane, consume the API outside the area where the policy applies:
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
User guide: Authorization with Keycloak Authorization Services¶
-
Keycloak provides a powerful set of tools (REST endpoints and administrative UIs), also known as Keycloak Authorization Services, to manage and enforce authorization, workflows for multiple access control mechanisms, including discretionary user access control and user-managed permissions.
-
This user guide is an example of how to use Authorino as an adapter to Keycloak Authorization Services while still relying on the reverse-proxy integration pattern, thus not involving importing an authorization library nor rebuilding the application's code.
-
-
- Authorino capabilities featured in this guide:
-
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
Deploy the identity provider and authentication server by executing the command below. For the examples in this guide, we are going to use a Keycloak server preloaded with all required realm settings.
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❺.
-
At step ❺, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
Create an Authorino AuthConfig custom resource declaring the auth rules to be enforced.
-
In this example, Authorino will accept access tokens (JWTs) issued by the Keycloak server. These JWTs can be either normal Keycloak ID tokens or Requesting Party Tokens (RPT).
-
RPTs include claims about the permissions of the user regarding protected resources and scopes associated with a Keycloak authorization client that the user can access.
-
When the supplied access token is an RPT, Authorino will just validate whether the user's granted permissions present in the token include the requested resource ID (translated from the path) and scope (inferred from the HTTP method). If the token does not contain a permissions claim (i.e. it is not an RPT), Authorino will negotiate a User-Managed Access (UMA) ticket on behalf of the user and try to obtain an RPT on that UMA ticket.
-
In cases of asynchronous user-managed permission control, the first request to the API using a normal Keycloak ID token is denied by Authorino. The user that owns the resource acknowledges the access request in the Keycloak UI. If access is granted, the new permissions will be reflected in subsequent RPTs obtained by Authorino on behalf of the requesting party.
-
Whenever an RPT with proper permissions is obtained by Authorino, the RPT is supplied back to the API consumer, so it can be used in subsequent requests thus skipping new negotiations of UMA tickets.
-
-
-
-
- Kuadrant users –
- Remember to create an AuthPolicy instead of an AuthConfig.
- For more, see Kuadrant auth.
-
❻ Obtain an access token with the Keycloak server¶
-
The AuthConfig deployed in the previous step is suitable for validating access tokens requested inside the cluster. This is because Keycloak's iss claim added to the JWTs matches always the host used to request the token and Authorino will later try to match this host to the host that provides the OpenID Connect configuration.
-
Obtain an access token from within the cluster for user Jane:
If your Keycloak server is reachable from outside the cluster, feel free to obtain the token directly. Make sure the host name set in the OIDC issuer endpoint in the AuthConfig matches the one used to obtain the token and is as well reachable from within the cluster.
As John, log in to http://localhost:8080/realms/kuadrant/account in the web browser (username: john / password: p), and grant access to the resource greeting-1 for Jane. A pending permission request by Jane shall exist in the list of John's Resources.
-
-
-
-
As Jane, try to consume the protected resource /greetings/1 again:
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
Authorino can delegate authorization decision to the Kubernetes authorization system, allowing permissions to be stored and managed using the Kubernetes Role-Based Access Control (RBAC) for example. The feature is based on the SubjectAccessReview API and can be used for resourceAttributes (parameters defined in the AuthConfig) or nonResourceAttributes (inferring HTTP path and verb from the original request).
Kubernetes server with permissions to install cluster-scoped resources (operator, CRDs and RBAC) and to create TokenRequests (to consume the protected service from outside the cluster)
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
-
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❺.
-
At step ❺, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
Create an Authorino AuthConfig custom resource declaring the auth rules to be enforced.
-
The AuthConfig below sets all Kubernetes service accounts as trusted users of the API, and relies on the Kubernetes RBAC to enforce authorization using Kubernetes SubjectAccessReview API for non-resource endpoints:
-
-
-
-
- Kuadrant users –
- Remember to create an AuthPolicy instead of an AuthConfig.
- For more, see Kuadrant auth.
-
Check out the spec for the Authorino Kubernetes SubjectAccessReview authorization feature, for resource attributes permission checks where SubjectAccessReviews issued by Authorino are modeled in terms of common attributes of operations on Kubernetes resources (namespace, API group, kind, name, subresource, verb).
-
❻ Create roles associated with endpoints of the API¶
-
Because the k8s-rbac policy defined in the AuthConfig in the previous step is for non-resource access review requests, the corresponding roles and role bindings have to be defined at cluster scope.
-
Create a talker-api-greeter role whose users and service accounts bound to this role can consume the non-resource endpoints POST /hello and POST /hi of the API:
Run a pod that consumes one of the greeting endpoints of the API from inside the cluster, as service account api-consumer-1, bound to the talker-api-greeter and talker-api-speaker cluster roles in the Kubernetes RBAC:
Run a pod that sends a POST request to /say/blah from within the cluster, as service account api-consumer-2, bound only to the talker-api-greeter cluster role in the Kubernetes RBAC:
-
-Extra: consume the API as service account api-consumer-2 from outside the cluster
-
-
Obtain a short-lived access token for service account api-consumer-2, bound to the talker-api-greeter cluster role in the Kubernetes RBAC, using the Kubernetes TokenRequest API:
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
Authorino can verify Kubernetes-valid access tokens (using Kubernetes TokenReview API).
-
These tokens can be either ServiceAccount tokens or any valid user access tokens issued to users of the Kubernetes server API.
-
The audiences claim of the token must include the requested host and port of the protected API (default), or all audiences specified in spec.identity.kubernetes.audiences of the AuthConfig.
-
For further details about Authorino features in general, check the docs.
Kubernetes server with permissions to install cluster-scoped resources (operator, CRDs and RBAC) and to create TokenRequests (to consume the protected service from outside the cluster)
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
-
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❺.
-
At step ❺, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
Deploy an application that consumes an endpoint of the Talker API, in a loop, every 10 seconds. The application uses a short-lived service account token mounted inside the container using Kubernetes Service Account Token Volume Projection to authenticate.
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
Authorino can verify x509 certificates presented by clients for authentication on the request to the protected APIs, at application level.
-
Trusted root Certificate Authorities (CA) are stored as Kubernetes kubernetes.io/tls Secrets labeled according to selectors specified in the AuthConfig, watched and cached by Authorino.
-
For further details about Authorino features in general, check the docs.
Kubernetes server with permissions to install cluster-scoped resources (operator, CRDs and RBAC)
-
-
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
-
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❻.
-
At step ❻, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following commands will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS enabled3.
-
Create the TLS certificates for the Authorino service:
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
Create a CA (Certificate Authority) certificate to issue the client certificates that will be used to authenticate clients that send requests to the Talker API:
The following command deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
Authorino can perform OAuth 2.0 token introspection (RFC 7662) on the access tokens supplied in the requests to protected APIs. This is particularly useful when using opaque tokens, for remote checking the token validity and resolving the identity object.
-
Important! Authorino does not implement OAuth2 grants nor OIDC authentication flows. As a common recommendation of good practice, obtaining and refreshing access tokens is for clients to negotiate directly with the auth servers and token issuers. Authorino will only validate those tokens using the parameters provided by the trusted issuer authorities.
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
Deploy a Keycloak server preloaded with the realm settings required for this guide:
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❺.
-
At step ❺, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
On every request, Authorino will try to verify the token remotely with the Keycloak server and the a12n-server server.
-
For authorization, whenever the introspected token data includes a privileges property (returned by a12n-server), Authorino will enforce only consumers whose privileges.talker-api includes the "read" permission are granted access.
-
Check out the docs for information about the common feature Conditions about skipping parts of an AuthConfig in the auth pipeline based on context.
Obtain an access token with Keycloak and consume the API¶
-
Obtain an access token with the Keycloak server for user Jane:
-
The AuthConfig deployed in the previous step is suitable for validating access tokens requested inside the cluster. This is because Keycloak's iss claim added to the JWTs matches always the host used to request the token and Authorino will later try to match this host to the host that provides the OpenID Connect configuration.
-
Obtain an access token from within the cluster for the user Jane, whose e-mail has been verified:
If your Keycloak server is reachable from outside the cluster, feel free to obtain the token directly. Make sure the host name set in the OIDC issuer endpoint in the AuthConfig matches the one used to obtain the token and is as well reachable from within the cluster.
-
As user Jane, consume the API:
-
curl-H"Authorization: Bearer $ACCESS_TOKEN"http://talker-api.127.0.0.1.nip.io:8000/hello
-# HTTP/1.1 200 OK
-
-
Revoke the access token and try to consume the API again:
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
Metrics of the controller-runtime about reconciliation (caching) of AuthConfigs and API key Secrets
-
-
-
/server-metrics
-
Metrics of the external authorization gRPC and OIDC/Festival Wristband validation built-in HTTP servers
-
-
-
-
The Authorino Operator creates a Kubernetes Service named <authorino-cr-name>-controller-metrics that exposes the endpoints on port 8080. The Authorino instance allows to modify the port number of the metrics endpoints, by setting the --metrics-addr command-line flag (default: :8080).
Status of HTTP response sent by the OIDC (Festival Wristband) server.
-
status=200|404
-
counter
-
-
-
-
-
1 Both endpoints export metrics about the Go runtime, such as number of goroutines (go_goroutines) and threads (go_threads), usage of CPU, memory and GC stats.
-
2 Opt-in metrics: auth_server_evaluator_* metrics require authconfig.spec.(identity|metadata|authorization|response).metrics: true (default: false). This can be enforced for the entire instance (all AuthConfigs and evaluators), by setting the --deep-metrics-enabled command-line flag in the Authorino deployment.
-
-Example of metrics exported at the /metrics endpoint
-
# HELP controller_runtime_active_workers Number of currently used workers per controller
-# TYPE controller_runtime_active_workers gauge
-controller_runtime_active_workers{controller="authconfig"} 0
-controller_runtime_active_workers{controller="secret"} 0
-# HELP controller_runtime_max_concurrent_reconciles Maximum number of concurrent reconciles per controller
-# TYPE controller_runtime_max_concurrent_reconciles gauge
-controller_runtime_max_concurrent_reconciles{controller="authconfig"} 1
-controller_runtime_max_concurrent_reconciles{controller="secret"} 1
-# HELP controller_runtime_reconcile_errors_total Total number of reconciliation errors per controller
-# TYPE controller_runtime_reconcile_errors_total counter
-controller_runtime_reconcile_errors_total{controller="authconfig"} 12
-controller_runtime_reconcile_errors_total{controller="secret"} 0
-# HELP controller_runtime_reconcile_time_seconds Length of time per reconciliation per controller
-# TYPE controller_runtime_reconcile_time_seconds histogram
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="0.005"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="0.01"} 11
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="0.025"} 17
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="0.05"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="0.1"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="0.15"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="0.2"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="0.25"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="0.3"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="0.35"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="0.4"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="0.45"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="0.5"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="0.6"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="0.7"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="0.8"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="0.9"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="1"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="1.25"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="1.5"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="1.75"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="2"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="2.5"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="3"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="3.5"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="4"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="4.5"} 18
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="5"} 19
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="6"} 19
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="7"} 19
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="8"} 19
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="9"} 19
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="10"} 19
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="15"} 19
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="20"} 19
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="25"} 19
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="30"} 19
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="40"} 19
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="50"} 19
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="60"} 19
-controller_runtime_reconcile_time_seconds_bucket{controller="authconfig",le="+Inf"} 19
-controller_runtime_reconcile_time_seconds_sum{controller="authconfig"} 5.171108321999999
-controller_runtime_reconcile_time_seconds_count{controller="authconfig"} 19
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="0.005"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="0.01"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="0.025"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="0.05"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="0.1"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="0.15"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="0.2"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="0.25"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="0.3"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="0.35"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="0.4"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="0.45"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="0.5"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="0.6"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="0.7"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="0.8"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="0.9"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="1"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="1.25"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="1.5"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="1.75"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="2"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="2.5"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="3"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="3.5"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="4"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="4.5"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="5"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="6"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="7"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="8"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="9"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="10"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="15"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="20"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="25"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="30"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="40"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="50"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="60"} 1
-controller_runtime_reconcile_time_seconds_bucket{controller="secret",le="+Inf"} 1
-controller_runtime_reconcile_time_seconds_sum{controller="secret"} 0.000138025
-controller_runtime_reconcile_time_seconds_count{controller="secret"} 1
-# HELP controller_runtime_reconcile_total Total number of reconciliations per controller
-# TYPE controller_runtime_reconcile_total counter
-controller_runtime_reconcile_total{controller="authconfig",result="error"} 12
-controller_runtime_reconcile_total{controller="authconfig",result="requeue"} 0
-controller_runtime_reconcile_total{controller="authconfig",result="requeue_after"} 0
-controller_runtime_reconcile_total{controller="authconfig",result="success"} 7
-controller_runtime_reconcile_total{controller="secret",result="error"} 0
-controller_runtime_reconcile_total{controller="secret",result="requeue"} 0
-controller_runtime_reconcile_total{controller="secret",result="requeue_after"} 0
-controller_runtime_reconcile_total{controller="secret",result="success"} 1
-# HELP go_gc_cycles_automatic_gc_cycles_total Count of completed GC cycles generated by the Go runtime.
-# TYPE go_gc_cycles_automatic_gc_cycles_total counter
-go_gc_cycles_automatic_gc_cycles_total 13
-# HELP go_gc_cycles_forced_gc_cycles_total Count of completed GC cycles forced by the application.
-# TYPE go_gc_cycles_forced_gc_cycles_total counter
-go_gc_cycles_forced_gc_cycles_total 0
-# HELP go_gc_cycles_total_gc_cycles_total Count of all completed GC cycles.
-# TYPE go_gc_cycles_total_gc_cycles_total counter
-go_gc_cycles_total_gc_cycles_total 13
-# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
-# TYPE go_gc_duration_seconds summary
-go_gc_duration_seconds{quantile="0"} 4.5971e-05
-go_gc_duration_seconds{quantile="0.25"} 5.69e-05
-go_gc_duration_seconds{quantile="0.5"} 0.000140699
-go_gc_duration_seconds{quantile="0.75"} 0.000313162
-go_gc_duration_seconds{quantile="1"} 0.001692423
-go_gc_duration_seconds_sum 0.003671076
-go_gc_duration_seconds_count 13
-# HELP go_gc_heap_allocs_by_size_bytes_total Distribution of heap allocations by approximate size. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks.
-# TYPE go_gc_heap_allocs_by_size_bytes_total histogram
-go_gc_heap_allocs_by_size_bytes_total_bucket{le="8.999999999999998"} 6357
-go_gc_heap_allocs_by_size_bytes_total_bucket{le="16.999999999999996"} 45065
-[...]
-go_gc_heap_allocs_by_size_bytes_total_bucket{le="32768.99999999999"} 128306
-go_gc_heap_allocs_by_size_bytes_total_bucket{le="+Inf"} 128327
-go_gc_heap_allocs_by_size_bytes_total_sum 1.5021512e+07
-go_gc_heap_allocs_by_size_bytes_total_count 128327
-# HELP go_gc_heap_allocs_bytes_total Cumulative sum of memory allocated to the heap by the application.
-# TYPE go_gc_heap_allocs_bytes_total counter
-go_gc_heap_allocs_bytes_total 1.5021512e+07
-# HELP go_gc_heap_allocs_objects_total Cumulative count of heap allocations triggered by the application. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks.
-# TYPE go_gc_heap_allocs_objects_total counter
-go_gc_heap_allocs_objects_total 128327
-# HELP go_gc_heap_frees_by_size_bytes_total Distribution of freed heap allocations by approximate size. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks.
-# TYPE go_gc_heap_frees_by_size_bytes_total histogram
-go_gc_heap_frees_by_size_bytes_total_bucket{le="8.999999999999998"} 3885
-go_gc_heap_frees_by_size_bytes_total_bucket{le="16.999999999999996"} 33418
-[...]
-go_gc_heap_frees_by_size_bytes_total_bucket{le="32768.99999999999"} 96417
-go_gc_heap_frees_by_size_bytes_total_bucket{le="+Inf"} 96425
-go_gc_heap_frees_by_size_bytes_total_sum 9.880944e+06
-go_gc_heap_frees_by_size_bytes_total_count 96425
-# HELP go_gc_heap_frees_bytes_total Cumulative sum of heap memory freed by the garbage collector.
-# TYPE go_gc_heap_frees_bytes_total counter
-go_gc_heap_frees_bytes_total 9.880944e+06
-# HELP go_gc_heap_frees_objects_total Cumulative count of heap allocations whose storage was freed by the garbage collector. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks.
-# TYPE go_gc_heap_frees_objects_total counter
-go_gc_heap_frees_objects_total 96425
-# HELP go_gc_heap_goal_bytes Heap size target for the end of the GC cycle.
-# TYPE go_gc_heap_goal_bytes gauge
-go_gc_heap_goal_bytes 9.356624e+06
-# HELP go_gc_heap_objects_objects Number of objects, live or unswept, occupying heap memory.
-# TYPE go_gc_heap_objects_objects gauge
-go_gc_heap_objects_objects 31902
-# HELP go_gc_heap_tiny_allocs_objects_total Count of small allocations that are packed together into blocks. These allocations are counted separately from other allocations because each individual allocation is not tracked by the runtime, only their block. Each block is already accounted for in allocs-by-size and frees-by-size.
-# TYPE go_gc_heap_tiny_allocs_objects_total counter
-go_gc_heap_tiny_allocs_objects_total 11750
-# HELP go_gc_pauses_seconds_total Distribution individual GC-related stop-the-world pause latencies.
-# TYPE go_gc_pauses_seconds_total histogram
-go_gc_pauses_seconds_total_bucket{le="9.999999999999999e-10"} 0
-go_gc_pauses_seconds_total_bucket{le="1.9999999999999997e-09"} 0
-[...]
-go_gc_pauses_seconds_total_bucket{le="206708.18602188796"} 26
-go_gc_pauses_seconds_total_bucket{le="+Inf"} 26
-go_gc_pauses_seconds_total_sum 0.003151488
-go_gc_pauses_seconds_total_count 26
-# HELP go_goroutines Number of goroutines that currently exist.
-# TYPE go_goroutines gauge
-go_goroutines 80
-# HELP go_info Information about the Go environment.
-# TYPE go_info gauge
-go_info{version="go1.18.7"} 1
-# HELP go_memory_classes_heap_free_bytes Memory that is completely free and eligible to be returned to the underlying system, but has not been. This metric is the runtime's estimate of free address space that is backed by physical memory.
-# TYPE go_memory_classes_heap_free_bytes gauge
-go_memory_classes_heap_free_bytes 589824
-# HELP go_memory_classes_heap_objects_bytes Memory occupied by live objects and dead objects that have not yet been marked free by the garbage collector.
-# TYPE go_memory_classes_heap_objects_bytes gauge
-go_memory_classes_heap_objects_bytes 5.140568e+06
-# HELP go_memory_classes_heap_released_bytes Memory that is completely free and has been returned to the underlying system. This metric is the runtime's estimate of free address space that is still mapped into the process, but is not backed by physical memory.
-# TYPE go_memory_classes_heap_released_bytes gauge
-go_memory_classes_heap_released_bytes 4.005888e+06
-# HELP go_memory_classes_heap_stacks_bytes Memory allocated from the heap that is reserved for stack space, whether or not it is currently in-use.
-# TYPE go_memory_classes_heap_stacks_bytes gauge
-go_memory_classes_heap_stacks_bytes 786432
-# HELP go_memory_classes_heap_unused_bytes Memory that is reserved for heap objects but is not currently used to hold heap objects.
-# TYPE go_memory_classes_heap_unused_bytes gauge
-go_memory_classes_heap_unused_bytes 2.0602e+06
-# HELP go_memory_classes_metadata_mcache_free_bytes Memory that is reserved for runtime mcache structures, but not in-use.
-# TYPE go_memory_classes_metadata_mcache_free_bytes gauge
-go_memory_classes_metadata_mcache_free_bytes 13984
-# HELP go_memory_classes_metadata_mcache_inuse_bytes Memory that is occupied by runtime mcache structures that are currently being used.
-# TYPE go_memory_classes_metadata_mcache_inuse_bytes gauge
-go_memory_classes_metadata_mcache_inuse_bytes 2400
-# HELP go_memory_classes_metadata_mspan_free_bytes Memory that is reserved for runtime mspan structures, but not in-use.
-# TYPE go_memory_classes_metadata_mspan_free_bytes gauge
-go_memory_classes_metadata_mspan_free_bytes 17104
-# HELP go_memory_classes_metadata_mspan_inuse_bytes Memory that is occupied by runtime mspan structures that are currently being used.
-# TYPE go_memory_classes_metadata_mspan_inuse_bytes gauge
-go_memory_classes_metadata_mspan_inuse_bytes 113968
-# HELP go_memory_classes_metadata_other_bytes Memory that is reserved for or used to hold runtime metadata.
-# TYPE go_memory_classes_metadata_other_bytes gauge
-go_memory_classes_metadata_other_bytes 5.544408e+06
-# HELP go_memory_classes_os_stacks_bytes Stack memory allocated by the underlying operating system.
-# TYPE go_memory_classes_os_stacks_bytes gauge
-go_memory_classes_os_stacks_bytes 0
-# HELP go_memory_classes_other_bytes Memory used by execution trace buffers, structures for debugging the runtime, finalizer and profiler specials, and more.
-# TYPE go_memory_classes_other_bytes gauge
-go_memory_classes_other_bytes 537777
-# HELP go_memory_classes_profiling_buckets_bytes Memory that is used by the stack trace hash map used for profiling.
-# TYPE go_memory_classes_profiling_buckets_bytes gauge
-go_memory_classes_profiling_buckets_bytes 1.455487e+06
-# HELP go_memory_classes_total_bytes All memory mapped by the Go runtime into the current process as read-write. Note that this does not include memory mapped by code called via cgo or via the syscall package. Sum of all metrics in /memory/classes.
-# TYPE go_memory_classes_total_bytes gauge
-go_memory_classes_total_bytes 2.026804e+07
-# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
-# TYPE go_memstats_alloc_bytes gauge
-go_memstats_alloc_bytes 5.140568e+06
-# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
-# TYPE go_memstats_alloc_bytes_total counter
-go_memstats_alloc_bytes_total 1.5021512e+07
-# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
-# TYPE go_memstats_buck_hash_sys_bytes gauge
-go_memstats_buck_hash_sys_bytes 1.455487e+06
-# HELP go_memstats_frees_total Total number of frees.
-# TYPE go_memstats_frees_total counter
-go_memstats_frees_total 108175
-# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
-# TYPE go_memstats_gc_cpu_fraction gauge
-go_memstats_gc_cpu_fraction 0
-# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
-# TYPE go_memstats_gc_sys_bytes gauge
-go_memstats_gc_sys_bytes 5.544408e+06
-# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
-# TYPE go_memstats_heap_alloc_bytes gauge
-go_memstats_heap_alloc_bytes 5.140568e+06
-# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
-# TYPE go_memstats_heap_idle_bytes gauge
-go_memstats_heap_idle_bytes 4.595712e+06
-# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
-# TYPE go_memstats_heap_inuse_bytes gauge
-go_memstats_heap_inuse_bytes 7.200768e+06
-# HELP go_memstats_heap_objects Number of allocated objects.
-# TYPE go_memstats_heap_objects gauge
-go_memstats_heap_objects 31902
-# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
-# TYPE go_memstats_heap_released_bytes gauge
-go_memstats_heap_released_bytes 4.005888e+06
-# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
-# TYPE go_memstats_heap_sys_bytes gauge
-go_memstats_heap_sys_bytes 1.179648e+07
-# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
-# TYPE go_memstats_last_gc_time_seconds gauge
-go_memstats_last_gc_time_seconds 1.6461572121033354e+09
-# HELP go_memstats_lookups_total Total number of pointer lookups.
-# TYPE go_memstats_lookups_total counter
-go_memstats_lookups_total 0
-# HELP go_memstats_mallocs_total Total number of mallocs.
-# TYPE go_memstats_mallocs_total counter
-go_memstats_mallocs_total 140077
-# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
-# TYPE go_memstats_mcache_inuse_bytes gauge
-go_memstats_mcache_inuse_bytes 2400
-# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
-# TYPE go_memstats_mcache_sys_bytes gauge
-go_memstats_mcache_sys_bytes 16384
-# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
-# TYPE go_memstats_mspan_inuse_bytes gauge
-go_memstats_mspan_inuse_bytes 113968
-# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
-# TYPE go_memstats_mspan_sys_bytes gauge
-go_memstats_mspan_sys_bytes 131072
-# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
-# TYPE go_memstats_next_gc_bytes gauge
-go_memstats_next_gc_bytes 9.356624e+06
-# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
-# TYPE go_memstats_other_sys_bytes gauge
-go_memstats_other_sys_bytes 537777
-# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
-# TYPE go_memstats_stack_inuse_bytes gauge
-go_memstats_stack_inuse_bytes 786432
-# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
-# TYPE go_memstats_stack_sys_bytes gauge
-go_memstats_stack_sys_bytes 786432
-# HELP go_memstats_sys_bytes Number of bytes obtained from system.
-# TYPE go_memstats_sys_bytes gauge
-go_memstats_sys_bytes 2.026804e+07
-# HELP go_sched_goroutines_goroutines Count of live goroutines.
-# TYPE go_sched_goroutines_goroutines gauge
-go_sched_goroutines_goroutines 80
-# HELP go_sched_latencies_seconds Distribution of the time goroutines have spent in the scheduler in a runnable state before actually running.
-# TYPE go_sched_latencies_seconds histogram
-go_sched_latencies_seconds_bucket{le="9.999999999999999e-10"} 244
-go_sched_latencies_seconds_bucket{le="1.9999999999999997e-09"} 244
-[...]
-go_sched_latencies_seconds_bucket{le="206708.18602188796"} 2336
-go_sched_latencies_seconds_bucket{le="+Inf"} 2336
-go_sched_latencies_seconds_sum 0.18509832400000004
-go_sched_latencies_seconds_count 2336
-# HELP go_threads Number of OS threads created.
-# TYPE go_threads gauge
-go_threads 8
-# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
-# TYPE process_cpu_seconds_total counter
-process_cpu_seconds_total 1.84
-# HELP process_max_fds Maximum number of open file descriptors.
-# TYPE process_max_fds gauge
-process_max_fds 1.048576e+06
-# HELP process_open_fds Number of open file descriptors.
-# TYPE process_open_fds gauge
-process_open_fds 14
-# HELP process_resident_memory_bytes Resident memory size in bytes.
-# TYPE process_resident_memory_bytes gauge
-process_resident_memory_bytes 4.3728896e+07
-# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
-# TYPE process_start_time_seconds gauge
-process_start_time_seconds 1.64615612779e+09
-# HELP process_virtual_memory_bytes Virtual memory size in bytes.
-# TYPE process_virtual_memory_bytes gauge
-process_virtual_memory_bytes 7.65362176e+08
-# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
-# TYPE process_virtual_memory_max_bytes gauge
-process_virtual_memory_max_bytes 1.8446744073709552e+19
-# HELP rest_client_requests_total Number of HTTP requests, partitioned by status code, method, and host.
-# TYPE rest_client_requests_total counter
-rest_client_requests_total{code="200",host="10.96.0.1:443",method="GET"} 114
-rest_client_requests_total{code="200",host="10.96.0.1:443",method="PUT"} 4
-# HELP workqueue_adds_total Total number of adds handled by workqueue
-# TYPE workqueue_adds_total counter
-workqueue_adds_total{name="authconfig"} 19
-workqueue_adds_total{name="secret"} 1
-# HELP workqueue_depth Current depth of workqueue
-# TYPE workqueue_depth gauge
-workqueue_depth{name="authconfig"} 0
-workqueue_depth{name="secret"} 0
-# HELP workqueue_longest_running_processor_seconds How many seconds has the longest running processor for workqueue been running.
-# TYPE workqueue_longest_running_processor_seconds gauge
-workqueue_longest_running_processor_seconds{name="authconfig"} 0
-workqueue_longest_running_processor_seconds{name="secret"} 0
-# HELP workqueue_queue_duration_seconds How long in seconds an item stays in workqueue before being requested
-# TYPE workqueue_queue_duration_seconds histogram
-workqueue_queue_duration_seconds_bucket{name="authconfig",le="1e-08"} 0
-workqueue_queue_duration_seconds_bucket{name="authconfig",le="1e-07"} 0
-workqueue_queue_duration_seconds_bucket{name="authconfig",le="1e-06"} 0
-workqueue_queue_duration_seconds_bucket{name="authconfig",le="9.999999999999999e-06"} 8
-workqueue_queue_duration_seconds_bucket{name="authconfig",le="9.999999999999999e-05"} 17
-workqueue_queue_duration_seconds_bucket{name="authconfig",le="0.001"} 17
-workqueue_queue_duration_seconds_bucket{name="authconfig",le="0.01"} 17
-workqueue_queue_duration_seconds_bucket{name="authconfig",le="0.1"} 18
-workqueue_queue_duration_seconds_bucket{name="authconfig",le="1"} 18
-workqueue_queue_duration_seconds_bucket{name="authconfig",le="10"} 19
-workqueue_queue_duration_seconds_bucket{name="authconfig",le="+Inf"} 19
-workqueue_queue_duration_seconds_sum{name="authconfig"} 4.969016371
-workqueue_queue_duration_seconds_count{name="authconfig"} 19
-workqueue_queue_duration_seconds_bucket{name="secret",le="1e-08"} 0
-workqueue_queue_duration_seconds_bucket{name="secret",le="1e-07"} 0
-workqueue_queue_duration_seconds_bucket{name="secret",le="1e-06"} 0
-workqueue_queue_duration_seconds_bucket{name="secret",le="9.999999999999999e-06"} 1
-workqueue_queue_duration_seconds_bucket{name="secret",le="9.999999999999999e-05"} 1
-workqueue_queue_duration_seconds_bucket{name="secret",le="0.001"} 1
-workqueue_queue_duration_seconds_bucket{name="secret",le="0.01"} 1
-workqueue_queue_duration_seconds_bucket{name="secret",le="0.1"} 1
-workqueue_queue_duration_seconds_bucket{name="secret",le="1"} 1
-workqueue_queue_duration_seconds_bucket{name="secret",le="10"} 1
-workqueue_queue_duration_seconds_bucket{name="secret",le="+Inf"} 1
-workqueue_queue_duration_seconds_sum{name="secret"} 4.67e-06
-workqueue_queue_duration_seconds_count{name="secret"} 1
-# HELP workqueue_retries_total Total number of retries handled by workqueue
-# TYPE workqueue_retries_total counter
-workqueue_retries_total{name="authconfig"} 12
-workqueue_retries_total{name="secret"} 0
-# HELP workqueue_unfinished_work_seconds How many seconds of work has been done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases.
-# TYPE workqueue_unfinished_work_seconds gauge
-workqueue_unfinished_work_seconds{name="authconfig"} 0
-workqueue_unfinished_work_seconds{name="secret"} 0
-# HELP workqueue_work_duration_seconds How long in seconds processing an item from workqueue takes.
-# TYPE workqueue_work_duration_seconds histogram
-workqueue_work_duration_seconds_bucket{name="authconfig",le="1e-08"} 0
-workqueue_work_duration_seconds_bucket{name="authconfig",le="1e-07"} 0
-workqueue_work_duration_seconds_bucket{name="authconfig",le="1e-06"} 0
-workqueue_work_duration_seconds_bucket{name="authconfig",le="9.999999999999999e-06"} 0
-workqueue_work_duration_seconds_bucket{name="authconfig",le="9.999999999999999e-05"} 0
-workqueue_work_duration_seconds_bucket{name="authconfig",le="0.001"} 0
-workqueue_work_duration_seconds_bucket{name="authconfig",le="0.01"} 11
-workqueue_work_duration_seconds_bucket{name="authconfig",le="0.1"} 18
-workqueue_work_duration_seconds_bucket{name="authconfig",le="1"} 18
-workqueue_work_duration_seconds_bucket{name="authconfig",le="10"} 19
-workqueue_work_duration_seconds_bucket{name="authconfig",le="+Inf"} 19
-workqueue_work_duration_seconds_sum{name="authconfig"} 5.171738079000001
-workqueue_work_duration_seconds_count{name="authconfig"} 19
-workqueue_work_duration_seconds_bucket{name="secret",le="1e-08"} 0
-workqueue_work_duration_seconds_bucket{name="secret",le="1e-07"} 0
-workqueue_work_duration_seconds_bucket{name="secret",le="1e-06"} 0
-workqueue_work_duration_seconds_bucket{name="secret",le="9.999999999999999e-06"} 0
-workqueue_work_duration_seconds_bucket{name="secret",le="9.999999999999999e-05"} 0
-workqueue_work_duration_seconds_bucket{name="secret",le="0.001"} 1
-workqueue_work_duration_seconds_bucket{name="secret",le="0.01"} 1
-workqueue_work_duration_seconds_bucket{name="secret",le="0.1"} 1
-workqueue_work_duration_seconds_bucket{name="secret",le="1"} 1
-workqueue_work_duration_seconds_bucket{name="secret",le="10"} 1
-workqueue_work_duration_seconds_bucket{name="secret",le="+Inf"} 1
-workqueue_work_duration_seconds_sum{name="secret"} 0.000150956
-workqueue_work_duration_seconds_count{name="secret"} 1
-
-
-
-Example of metrics exported at the /server-metrics endpoint
-
# HELP auth_server_authconfig_duration_seconds Response latency of authconfig enforced by the auth server (in seconds).
-# TYPE auth_server_authconfig_duration_seconds histogram
-auth_server_authconfig_duration_seconds_bucket{authconfig="edge-auth",namespace="authorino",le="0.001"} 0
-auth_server_authconfig_duration_seconds_bucket{authconfig="edge-auth",namespace="authorino",le="0.051000000000000004"} 1
-auth_server_authconfig_duration_seconds_bucket{authconfig="edge-auth",namespace="authorino",le="0.101"} 1
-auth_server_authconfig_duration_seconds_bucket{authconfig="edge-auth",namespace="authorino",le="0.15100000000000002"} 1
-auth_server_authconfig_duration_seconds_bucket{authconfig="edge-auth",namespace="authorino",le="0.201"} 1
-auth_server_authconfig_duration_seconds_bucket{authconfig="edge-auth",namespace="authorino",le="0.251"} 1
-auth_server_authconfig_duration_seconds_bucket{authconfig="edge-auth",namespace="authorino",le="0.301"} 1
-auth_server_authconfig_duration_seconds_bucket{authconfig="edge-auth",namespace="authorino",le="0.351"} 1
-auth_server_authconfig_duration_seconds_bucket{authconfig="edge-auth",namespace="authorino",le="0.40099999999999997"} 1
-auth_server_authconfig_duration_seconds_bucket{authconfig="edge-auth",namespace="authorino",le="0.45099999999999996"} 1
-auth_server_authconfig_duration_seconds_bucket{authconfig="edge-auth",namespace="authorino",le="0.501"} 1
-auth_server_authconfig_duration_seconds_bucket{authconfig="edge-auth",namespace="authorino",le="0.551"} 1
-auth_server_authconfig_duration_seconds_bucket{authconfig="edge-auth",namespace="authorino",le="0.6010000000000001"} 1
-auth_server_authconfig_duration_seconds_bucket{authconfig="edge-auth",namespace="authorino",le="0.6510000000000001"} 1
-auth_server_authconfig_duration_seconds_bucket{authconfig="edge-auth",namespace="authorino",le="0.7010000000000002"} 1
-auth_server_authconfig_duration_seconds_bucket{authconfig="edge-auth",namespace="authorino",le="0.7510000000000002"} 1
-auth_server_authconfig_duration_seconds_bucket{authconfig="edge-auth",namespace="authorino",le="0.8010000000000003"} 1
-auth_server_authconfig_duration_seconds_bucket{authconfig="edge-auth",namespace="authorino",le="0.8510000000000003"} 1
-auth_server_authconfig_duration_seconds_bucket{authconfig="edge-auth",namespace="authorino",le="0.9010000000000004"} 1
-auth_server_authconfig_duration_seconds_bucket{authconfig="edge-auth",namespace="authorino",le="0.9510000000000004"} 1
-auth_server_authconfig_duration_seconds_bucket{authconfig="edge-auth",namespace="authorino",le="+Inf"} 1
-auth_server_authconfig_duration_seconds_sum{authconfig="edge-auth",namespace="authorino"} 0.001701795
-auth_server_authconfig_duration_seconds_count{authconfig="edge-auth",namespace="authorino"} 1
-auth_server_authconfig_duration_seconds_bucket{authconfig="talker-api-protection",namespace="authorino",le="0.001"} 1
-auth_server_authconfig_duration_seconds_bucket{authconfig="talker-api-protection",namespace="authorino",le="0.051000000000000004"} 4
-auth_server_authconfig_duration_seconds_bucket{authconfig="talker-api-protection",namespace="authorino",le="0.101"} 4
-auth_server_authconfig_duration_seconds_bucket{authconfig="talker-api-protection",namespace="authorino",le="0.15100000000000002"} 5
-auth_server_authconfig_duration_seconds_bucket{authconfig="talker-api-protection",namespace="authorino",le="0.201"} 5
-auth_server_authconfig_duration_seconds_bucket{authconfig="talker-api-protection",namespace="authorino",le="0.251"} 5
-auth_server_authconfig_duration_seconds_bucket{authconfig="talker-api-protection",namespace="authorino",le="0.301"} 5
-auth_server_authconfig_duration_seconds_bucket{authconfig="talker-api-protection",namespace="authorino",le="0.351"} 5
-auth_server_authconfig_duration_seconds_bucket{authconfig="talker-api-protection",namespace="authorino",le="0.40099999999999997"} 5
-auth_server_authconfig_duration_seconds_bucket{authconfig="talker-api-protection",namespace="authorino",le="0.45099999999999996"} 5
-auth_server_authconfig_duration_seconds_bucket{authconfig="talker-api-protection",namespace="authorino",le="0.501"} 5
-auth_server_authconfig_duration_seconds_bucket{authconfig="talker-api-protection",namespace="authorino",le="0.551"} 5
-auth_server_authconfig_duration_seconds_bucket{authconfig="talker-api-protection",namespace="authorino",le="0.6010000000000001"} 5
-auth_server_authconfig_duration_seconds_bucket{authconfig="talker-api-protection",namespace="authorino",le="0.6510000000000001"} 5
-auth_server_authconfig_duration_seconds_bucket{authconfig="talker-api-protection",namespace="authorino",le="0.7010000000000002"} 5
-auth_server_authconfig_duration_seconds_bucket{authconfig="talker-api-protection",namespace="authorino",le="0.7510000000000002"} 5
-auth_server_authconfig_duration_seconds_bucket{authconfig="talker-api-protection",namespace="authorino",le="0.8010000000000003"} 5
-auth_server_authconfig_duration_seconds_bucket{authconfig="talker-api-protection",namespace="authorino",le="0.8510000000000003"} 5
-auth_server_authconfig_duration_seconds_bucket{authconfig="talker-api-protection",namespace="authorino",le="0.9010000000000004"} 5
-auth_server_authconfig_duration_seconds_bucket{authconfig="talker-api-protection",namespace="authorino",le="0.9510000000000004"} 5
-auth_server_authconfig_duration_seconds_bucket{authconfig="talker-api-protection",namespace="authorino",le="+Inf"} 5
-auth_server_authconfig_duration_seconds_sum{authconfig="talker-api-protection",namespace="authorino"} 0.26967658299999997
-auth_server_authconfig_duration_seconds_count{authconfig="talker-api-protection",namespace="authorino"} 5
-# HELP auth_server_authconfig_response_status Response status of authconfigs sent by the auth server, partitioned by authconfig.
-# TYPE auth_server_authconfig_response_status counter
-auth_server_authconfig_response_status{authconfig="edge-auth",namespace="authorino",status="OK"} 1
-auth_server_authconfig_response_status{authconfig="talker-api-protection",namespace="authorino",status="OK"} 2
-auth_server_authconfig_response_status{authconfig="talker-api-protection",namespace="authorino",status="PERMISSION_DENIED"} 2
-auth_server_authconfig_response_status{authconfig="talker-api-protection",namespace="authorino",status="UNAUTHENTICATED"} 1
-# HELP auth_server_authconfig_total Total number of authconfigs enforced by the auth server, partitioned by authconfig.
-# TYPE auth_server_authconfig_total counter
-auth_server_authconfig_total{authconfig="edge-auth",namespace="authorino"} 1
-auth_server_authconfig_total{authconfig="talker-api-protection",namespace="authorino"} 5
-# HELP auth_server_evaluator_duration_seconds Response latency of individual authconfig rule evaluated by the auth server (in seconds).
-# TYPE auth_server_evaluator_duration_seconds histogram
-auth_server_evaluator_duration_seconds_bucket{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino",le="0.001"} 0
-auth_server_evaluator_duration_seconds_bucket{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino",le="0.051000000000000004"} 3
-auth_server_evaluator_duration_seconds_bucket{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino",le="0.101"} 3
-auth_server_evaluator_duration_seconds_bucket{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino",le="0.15100000000000002"} 4
-auth_server_evaluator_duration_seconds_bucket{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino",le="0.201"} 4
-auth_server_evaluator_duration_seconds_bucket{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino",le="0.251"} 4
-auth_server_evaluator_duration_seconds_bucket{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino",le="0.301"} 4
-auth_server_evaluator_duration_seconds_bucket{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino",le="0.351"} 4
-auth_server_evaluator_duration_seconds_bucket{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino",le="0.40099999999999997"} 4
-auth_server_evaluator_duration_seconds_bucket{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino",le="0.45099999999999996"} 4
-auth_server_evaluator_duration_seconds_bucket{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino",le="0.501"} 4
-auth_server_evaluator_duration_seconds_bucket{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino",le="0.551"} 4
-auth_server_evaluator_duration_seconds_bucket{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino",le="0.6010000000000001"} 4
-auth_server_evaluator_duration_seconds_bucket{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino",le="0.6510000000000001"} 4
-auth_server_evaluator_duration_seconds_bucket{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino",le="0.7010000000000002"} 4
-auth_server_evaluator_duration_seconds_bucket{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino",le="0.7510000000000002"} 4
-auth_server_evaluator_duration_seconds_bucket{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino",le="0.8010000000000003"} 4
-auth_server_evaluator_duration_seconds_bucket{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino",le="0.8510000000000003"} 4
-auth_server_evaluator_duration_seconds_bucket{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino",le="0.9010000000000004"} 4
-auth_server_evaluator_duration_seconds_bucket{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino",le="0.9510000000000004"} 4
-auth_server_evaluator_duration_seconds_bucket{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino",le="+Inf"} 4
-auth_server_evaluator_duration_seconds_sum{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino"} 0.25800055
-auth_server_evaluator_duration_seconds_count{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino"} 4
-# HELP auth_server_evaluator_total Total number of evaluations of individual authconfig rule performed by the auth server.
-# TYPE auth_server_evaluator_total counter
-auth_server_evaluator_total{authconfig="talker-api-protection",evaluator_name="geo",evaluator_type="METADATA_GENERIC_HTTP",namespace="authorino"} 4
-# HELP auth_server_response_status Response status of authconfigs sent by the auth server.
-# TYPE auth_server_response_status counter
-auth_server_response_status{status="NOT_FOUND"} 1
-auth_server_response_status{status="OK"} 3
-auth_server_response_status{status="PERMISSION_DENIED"} 2
-auth_server_response_status{status="UNAUTHENTICATED"} 1
-# HELP go_gc_cycles_automatic_gc_cycles_total Count of completed GC cycles generated by the Go runtime.
-# TYPE go_gc_cycles_automatic_gc_cycles_total counter
-go_gc_cycles_automatic_gc_cycles_total 11
-# HELP go_gc_cycles_forced_gc_cycles_total Count of completed GC cycles forced by the application.
-# TYPE go_gc_cycles_forced_gc_cycles_total counter
-go_gc_cycles_forced_gc_cycles_total 0
-# HELP go_gc_cycles_total_gc_cycles_total Count of all completed GC cycles.
-# TYPE go_gc_cycles_total_gc_cycles_total counter
-go_gc_cycles_total_gc_cycles_total 11
-# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
-# TYPE go_gc_duration_seconds summary
-go_gc_duration_seconds{quantile="0"} 4.5971e-05
-go_gc_duration_seconds{quantile="0.25"} 5.69e-05
-go_gc_duration_seconds{quantile="0.5"} 0.000158594
-go_gc_duration_seconds{quantile="0.75"} 0.000324091
-go_gc_duration_seconds{quantile="1"} 0.001692423
-go_gc_duration_seconds_sum 0.003546711
-go_gc_duration_seconds_count 11
-# HELP go_gc_heap_allocs_by_size_bytes_total Distribution of heap allocations by approximate size. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks.
-# TYPE go_gc_heap_allocs_by_size_bytes_total histogram
-go_gc_heap_allocs_by_size_bytes_total_bucket{le="8.999999999999998"} 6261
-go_gc_heap_allocs_by_size_bytes_total_bucket{le="16.999999999999996"} 42477
-[...]
-go_gc_heap_allocs_by_size_bytes_total_bucket{le="32768.99999999999"} 122133
-go_gc_heap_allocs_by_size_bytes_total_bucket{le="+Inf"} 122154
-go_gc_heap_allocs_by_size_bytes_total_sum 1.455944e+07
-go_gc_heap_allocs_by_size_bytes_total_count 122154
-# HELP go_gc_heap_allocs_bytes_total Cumulative sum of memory allocated to the heap by the application.
-# TYPE go_gc_heap_allocs_bytes_total counter
-go_gc_heap_allocs_bytes_total 1.455944e+07
-# HELP go_gc_heap_allocs_objects_total Cumulative count of heap allocations triggered by the application. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks.
-# TYPE go_gc_heap_allocs_objects_total counter
-go_gc_heap_allocs_objects_total 122154
-# HELP go_gc_heap_frees_by_size_bytes_total Distribution of freed heap allocations by approximate size. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks.
-# TYPE go_gc_heap_frees_by_size_bytes_total histogram
-go_gc_heap_frees_by_size_bytes_total_bucket{le="8.999999999999998"} 3789
-go_gc_heap_frees_by_size_bytes_total_bucket{le="16.999999999999996"} 31067
-[...]
-go_gc_heap_frees_by_size_bytes_total_bucket{le="32768.99999999999"} 91013
-go_gc_heap_frees_by_size_bytes_total_bucket{le="+Inf"} 91021
-go_gc_heap_frees_by_size_bytes_total_sum 9.399936e+06
-go_gc_heap_frees_by_size_bytes_total_count 91021
-# HELP go_gc_heap_frees_bytes_total Cumulative sum of heap memory freed by the garbage collector.
-# TYPE go_gc_heap_frees_bytes_total counter
-go_gc_heap_frees_bytes_total 9.399936e+06
-# HELP go_gc_heap_frees_objects_total Cumulative count of heap allocations whose storage was freed by the garbage collector. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks.
-# TYPE go_gc_heap_frees_objects_total counter
-go_gc_heap_frees_objects_total 91021
-# HELP go_gc_heap_goal_bytes Heap size target for the end of the GC cycle.
-# TYPE go_gc_heap_goal_bytes gauge
-go_gc_heap_goal_bytes 9.601744e+06
-# HELP go_gc_heap_objects_objects Number of objects, live or unswept, occupying heap memory.
-# TYPE go_gc_heap_objects_objects gauge
-go_gc_heap_objects_objects 31133
-# HELP go_gc_heap_tiny_allocs_objects_total Count of small allocations that are packed together into blocks. These allocations are counted separately from other allocations because each individual allocation is not tracked by the runtime, only their block. Each block is already accounted for in allocs-by-size and frees-by-size.
-# TYPE go_gc_heap_tiny_allocs_objects_total counter
-go_gc_heap_tiny_allocs_objects_total 9866
-# HELP go_gc_pauses_seconds_total Distribution individual GC-related stop-the-world pause latencies.
-# TYPE go_gc_pauses_seconds_total histogram
-go_gc_pauses_seconds_total_bucket{le="9.999999999999999e-10"} 0
-go_gc_pauses_seconds_total_bucket{le="1.9999999999999997e-09"} 0
-[...]
-go_gc_pauses_seconds_total_bucket{le="206708.18602188796"} 22
-go_gc_pauses_seconds_total_bucket{le="+Inf"} 22
-go_gc_pauses_seconds_total_sum 0.0030393599999999996
-go_gc_pauses_seconds_total_count 22
-# HELP go_goroutines Number of goroutines that currently exist.
-# TYPE go_goroutines gauge
-go_goroutines 79
-# HELP go_info Information about the Go environment.
-# TYPE go_info gauge
-go_info{version="go1.18.7"} 1
-# HELP go_memory_classes_heap_free_bytes Memory that is completely free and eligible to be returned to the underlying system, but has not been. This metric is the runtime's estimate of free address space that is backed by physical memory.
-# TYPE go_memory_classes_heap_free_bytes gauge
-go_memory_classes_heap_free_bytes 630784
-# HELP go_memory_classes_heap_objects_bytes Memory occupied by live objects and dead objects that have not yet been marked free by the garbage collector.
-# TYPE go_memory_classes_heap_objects_bytes gauge
-go_memory_classes_heap_objects_bytes 5.159504e+06
-# HELP go_memory_classes_heap_released_bytes Memory that is completely free and has been returned to the underlying system. This metric is the runtime's estimate of free address space that is still mapped into the process, but is not backed by physical memory.
-# TYPE go_memory_classes_heap_released_bytes gauge
-go_memory_classes_heap_released_bytes 3.858432e+06
-# HELP go_memory_classes_heap_stacks_bytes Memory allocated from the heap that is reserved for stack space, whether or not it is currently in-use.
-# TYPE go_memory_classes_heap_stacks_bytes gauge
-go_memory_classes_heap_stacks_bytes 786432
-# HELP go_memory_classes_heap_unused_bytes Memory that is reserved for heap objects but is not currently used to hold heap objects.
-# TYPE go_memory_classes_heap_unused_bytes gauge
-go_memory_classes_heap_unused_bytes 2.14776e+06
-# HELP go_memory_classes_metadata_mcache_free_bytes Memory that is reserved for runtime mcache structures, but not in-use.
-# TYPE go_memory_classes_metadata_mcache_free_bytes gauge
-go_memory_classes_metadata_mcache_free_bytes 13984
-# HELP go_memory_classes_metadata_mcache_inuse_bytes Memory that is occupied by runtime mcache structures that are currently being used.
-# TYPE go_memory_classes_metadata_mcache_inuse_bytes gauge
-go_memory_classes_metadata_mcache_inuse_bytes 2400
-# HELP go_memory_classes_metadata_mspan_free_bytes Memory that is reserved for runtime mspan structures, but not in-use.
-# TYPE go_memory_classes_metadata_mspan_free_bytes gauge
-go_memory_classes_metadata_mspan_free_bytes 16696
-# HELP go_memory_classes_metadata_mspan_inuse_bytes Memory that is occupied by runtime mspan structures that are currently being used.
-# TYPE go_memory_classes_metadata_mspan_inuse_bytes gauge
-go_memory_classes_metadata_mspan_inuse_bytes 114376
-# HELP go_memory_classes_metadata_other_bytes Memory that is reserved for or used to hold runtime metadata.
-# TYPE go_memory_classes_metadata_other_bytes gauge
-go_memory_classes_metadata_other_bytes 5.544408e+06
-# HELP go_memory_classes_os_stacks_bytes Stack memory allocated by the underlying operating system.
-# TYPE go_memory_classes_os_stacks_bytes gauge
-go_memory_classes_os_stacks_bytes 0
-# HELP go_memory_classes_other_bytes Memory used by execution trace buffers, structures for debugging the runtime, finalizer and profiler specials, and more.
-# TYPE go_memory_classes_other_bytes gauge
-go_memory_classes_other_bytes 537777
-# HELP go_memory_classes_profiling_buckets_bytes Memory that is used by the stack trace hash map used for profiling.
-# TYPE go_memory_classes_profiling_buckets_bytes gauge
-go_memory_classes_profiling_buckets_bytes 1.455487e+06
-# HELP go_memory_classes_total_bytes All memory mapped by the Go runtime into the current process as read-write. Note that this does not include memory mapped by code called via cgo or via the syscall package. Sum of all metrics in /memory/classes.
-# TYPE go_memory_classes_total_bytes gauge
-go_memory_classes_total_bytes 2.026804e+07
-# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
-# TYPE go_memstats_alloc_bytes gauge
-go_memstats_alloc_bytes 5.159504e+06
-# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
-# TYPE go_memstats_alloc_bytes_total counter
-go_memstats_alloc_bytes_total 1.455944e+07
-# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
-# TYPE go_memstats_buck_hash_sys_bytes gauge
-go_memstats_buck_hash_sys_bytes 1.455487e+06
-# HELP go_memstats_frees_total Total number of frees.
-# TYPE go_memstats_frees_total counter
-go_memstats_frees_total 100887
-# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
-# TYPE go_memstats_gc_cpu_fraction gauge
-go_memstats_gc_cpu_fraction 0
-# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
-# TYPE go_memstats_gc_sys_bytes gauge
-go_memstats_gc_sys_bytes 5.544408e+06
-# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
-# TYPE go_memstats_heap_alloc_bytes gauge
-go_memstats_heap_alloc_bytes 5.159504e+06
-# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
-# TYPE go_memstats_heap_idle_bytes gauge
-go_memstats_heap_idle_bytes 4.489216e+06
-# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
-# TYPE go_memstats_heap_inuse_bytes gauge
-go_memstats_heap_inuse_bytes 7.307264e+06
-# HELP go_memstats_heap_objects Number of allocated objects.
-# TYPE go_memstats_heap_objects gauge
-go_memstats_heap_objects 31133
-# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
-# TYPE go_memstats_heap_released_bytes gauge
-go_memstats_heap_released_bytes 3.858432e+06
-# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
-# TYPE go_memstats_heap_sys_bytes gauge
-go_memstats_heap_sys_bytes 1.179648e+07
-# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
-# TYPE go_memstats_last_gc_time_seconds gauge
-go_memstats_last_gc_time_seconds 1.6461569717723043e+09
-# HELP go_memstats_lookups_total Total number of pointer lookups.
-# TYPE go_memstats_lookups_total counter
-go_memstats_lookups_total 0
-# HELP go_memstats_mallocs_total Total number of mallocs.
-# TYPE go_memstats_mallocs_total counter
-go_memstats_mallocs_total 132020
-# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
-# TYPE go_memstats_mcache_inuse_bytes gauge
-go_memstats_mcache_inuse_bytes 2400
-# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
-# TYPE go_memstats_mcache_sys_bytes gauge
-go_memstats_mcache_sys_bytes 16384
-# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
-# TYPE go_memstats_mspan_inuse_bytes gauge
-go_memstats_mspan_inuse_bytes 114376
-# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
-# TYPE go_memstats_mspan_sys_bytes gauge
-go_memstats_mspan_sys_bytes 131072
-# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
-# TYPE go_memstats_next_gc_bytes gauge
-go_memstats_next_gc_bytes 9.601744e+06
-# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
-# TYPE go_memstats_other_sys_bytes gauge
-go_memstats_other_sys_bytes 537777
-# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
-# TYPE go_memstats_stack_inuse_bytes gauge
-go_memstats_stack_inuse_bytes 786432
-# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
-# TYPE go_memstats_stack_sys_bytes gauge
-go_memstats_stack_sys_bytes 786432
-# HELP go_memstats_sys_bytes Number of bytes obtained from system.
-# TYPE go_memstats_sys_bytes gauge
-go_memstats_sys_bytes 2.026804e+07
-# HELP go_sched_goroutines_goroutines Count of live goroutines.
-# TYPE go_sched_goroutines_goroutines gauge
-go_sched_goroutines_goroutines 79
-# HELP go_sched_latencies_seconds Distribution of the time goroutines have spent in the scheduler in a runnable state before actually running.
-# TYPE go_sched_latencies_seconds histogram
-go_sched_latencies_seconds_bucket{le="9.999999999999999e-10"} 225
-go_sched_latencies_seconds_bucket{le="1.9999999999999997e-09"} 225
-[...]
-go_sched_latencies_seconds_bucket{le="206708.18602188796"} 1916
-go_sched_latencies_seconds_bucket{le="+Inf"} 1916
-go_sched_latencies_seconds_sum 0.18081453600000003
-go_sched_latencies_seconds_count 1916
-# HELP go_threads Number of OS threads created.
-# TYPE go_threads gauge
-go_threads 8
-# HELP grpc_server_handled_total Total number of RPCs completed on the server, regardless of success or failure.
-# TYPE grpc_server_handled_total counter
-grpc_server_handled_total{grpc_code="Aborted",grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="Aborted",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="Aborted",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
-grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
-grpc_server_handled_total{grpc_code="Canceled",grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="Canceled",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="Canceled",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
-grpc_server_handled_total{grpc_code="DataLoss",grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="DataLoss",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="DataLoss",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
-grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
-grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
-grpc_server_handled_total{grpc_code="Internal",grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="Internal",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="Internal",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
-grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
-grpc_server_handled_total{grpc_code="NotFound",grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="NotFound",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="NotFound",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
-grpc_server_handled_total{grpc_code="OK",grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 7
-grpc_server_handled_total{grpc_code="OK",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="OK",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
-grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
-grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
-grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
-grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
-grpc_server_handled_total{grpc_code="Unavailable",grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="Unavailable",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="Unavailable",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
-grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
-grpc_server_handled_total{grpc_code="Unknown",grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="Unknown",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
-grpc_server_handled_total{grpc_code="Unknown",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
-# HELP grpc_server_handling_seconds Histogram of response latency (seconds) of gRPC that had been application-level handled by the server.
-# TYPE grpc_server_handling_seconds histogram
-grpc_server_handling_seconds_bucket{grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary",le="0.005"} 3
-grpc_server_handling_seconds_bucket{grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary",le="0.01"} 3
-grpc_server_handling_seconds_bucket{grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary",le="0.025"} 3
-grpc_server_handling_seconds_bucket{grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary",le="0.05"} 6
-grpc_server_handling_seconds_bucket{grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary",le="0.1"} 6
-grpc_server_handling_seconds_bucket{grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary",le="0.25"} 7
-grpc_server_handling_seconds_bucket{grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary",le="0.5"} 7
-grpc_server_handling_seconds_bucket{grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary",le="1"} 7
-grpc_server_handling_seconds_bucket{grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary",le="2.5"} 7
-grpc_server_handling_seconds_bucket{grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary",le="5"} 7
-grpc_server_handling_seconds_bucket{grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary",le="10"} 7
-grpc_server_handling_seconds_bucket{grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary",le="+Inf"} 7
-grpc_server_handling_seconds_sum{grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 0.277605516
-grpc_server_handling_seconds_count{grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 7
-# HELP grpc_server_msg_received_total Total number of RPC stream messages received on the server.
-# TYPE grpc_server_msg_received_total counter
-grpc_server_msg_received_total{grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 7
-grpc_server_msg_received_total{grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
-grpc_server_msg_received_total{grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
-# HELP grpc_server_msg_sent_total Total number of gRPC stream messages sent by the server.
-# TYPE grpc_server_msg_sent_total counter
-grpc_server_msg_sent_total{grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 7
-grpc_server_msg_sent_total{grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
-grpc_server_msg_sent_total{grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
-# HELP grpc_server_started_total Total number of RPCs started on the server.
-# TYPE grpc_server_started_total counter
-grpc_server_started_total{grpc_method="Check",grpc_service="envoy.service.auth.v3.Authorization",grpc_type="unary"} 7
-grpc_server_started_total{grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
-grpc_server_started_total{grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
-# HELP oidc_server_requests_total Number of get requests received on the OIDC (Festival Wristband) server.
-# TYPE oidc_server_requests_total counter
-oidc_server_requests_total{authconfig="edge-auth",namespace="authorino",path="/.well-known/openid-configuration",wristband="wristband"} 1
-oidc_server_requests_total{authconfig="edge-auth",namespace="authorino",path="/.well-known/openid-connect/certs",wristband="wristband"} 1
-# HELP oidc_server_response_status Status of HTTP response sent by the OIDC (Festival Wristband) server.
-# TYPE oidc_server_response_status counter
-oidc_server_response_status{status="200"} 2
-# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
-# TYPE process_cpu_seconds_total counter
-process_cpu_seconds_total 1.42
-# HELP process_max_fds Maximum number of open file descriptors.
-# TYPE process_max_fds gauge
-process_max_fds 1.048576e+06
-# HELP process_open_fds Number of open file descriptors.
-# TYPE process_open_fds gauge
-process_open_fds 14
-# HELP process_resident_memory_bytes Resident memory size in bytes.
-# TYPE process_resident_memory_bytes gauge
-process_resident_memory_bytes 4.370432e+07
-# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
-# TYPE process_start_time_seconds gauge
-process_start_time_seconds 1.64615612779e+09
-# HELP process_virtual_memory_bytes Virtual memory size in bytes.
-# TYPE process_virtual_memory_bytes gauge
-process_virtual_memory_bytes 7.65362176e+08
-# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
-# TYPE process_virtual_memory_max_bytes gauge
-process_virtual_memory_max_bytes 1.8446744073709552e+19
-# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
-# TYPE promhttp_metric_handler_requests_in_flight gauge
-promhttp_metric_handler_requests_in_flight 1
-# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
-# TYPE promhttp_metric_handler_requests_total counter
-promhttp_metric_handler_requests_total{code="200"} 1
-promhttp_metric_handler_requests_total{code="500"} 0
-promhttp_metric_handler_requests_total{code="503"} 0
-
Authorino exposes two main endpoints for health and readiness check of the AuthConfig controller:
-
-
/healthz: Health probe (ping) – reports "ok" if the controller is healthy.
-
/readyz: Readiness probe – reports "ok" if the controller is ready to reconcile AuthConfig-related events.
-
-
In general, the endpoints return either 200 ("ok", i.e. all checks have passed) or 500 (when one or more checks failed).
-
The default binding network address is :8081, which can be changed by setting the command-line flag --health-probe-addr.
-
The following additional subpath is available and its corresponding check can be aggregated into the response from the main readiness probe:
-- /readyz/authconfigs: Aggregated readiness status of the AuthConfigs – reports "ok" if all AuthConfigs watched by the reconciler have been marked as ready.
-
-
-
-
Important! The AuthConfig readiness check within the scope of the aggregated readiness probe endpoint is deactivated by default – i.e. this check is an opt-in check. Sending a request to the /readyz endpoint without explicitly opting-in for the AuthConfigs check, by using the include parameter, will result in a response message that disregards the actual status of the watched AuthConfigs, possibly an "ok" message. To read the aggregated status of the watched AuthConfigs, either use the specific endpoint /readyz/authconfigs or opt-in for the check in the aggregated endpoint by sending a request to /readyz?include=authconfigs
-
-
-
-
-
Apart from include to add the aggregated status of the AuthConfigs, the following additional query string parameters are available:
-
-
verbose=true|false - provides more verbose response messages;
-
exclude=(check name) – to exclude a particular readiness check (for future usage).
Authorino provides structured log messages ("production") or more log messages output to stdout in a more user-friendly format ("development" mode) and different level of logging.
Authorino outputs 3 levels of log messages: (from lowest to highest level)
-
-
debug
-
info (default)
-
error
-
-
info logging is restricted to high-level information of the gRPC and HTTP authorization services, limiting messages to incoming request and respective outgoing response logs, with reduced details about the corresponding objects (request payload and authorization result), and without any further detailed logs of the steps in between, except for errors.
-
Only debug logging will include processing details of each Auth Pipeline, such as intermediary requests to validate identities with external auth servers, requests to external sources of auth metadata or authorization policies.
-
To configure the desired log level, set the spec.logLevel field of the Authorino custom resource (or --log-level command-line flag in the Authorino deployment), to one of the supported values listed above. Default log level is info.
-
Apart from log level, Authorino can output messages to the logs in 2 different formats:
-
-
production (default): each line is a parseable JSON object with properties {"level":string, "ts":int, "msg":string, "logger":string, extra values...}
-
development: more human-readable outputs, extra stack traces and logging info, plus extra values output as JSON, in the format: <timestamp-iso-8601>\t<log-level>\t<logger>\t<message>\t{extra-values-as-json}
-
-
To configure the desired log mode, set the spec.logMode field of the Authorino custom resource (or --log-mode command-line flag in the Authorino deployment), to one of the supported values listed above. Default log level is production.
-
Example of Authorino custom resource with log level debug and log mode production:
Authorino will never output HTTP headers and query string parameters to info log messages, as such values usually include sensitive data (e.g. access tokens, API keys and Authorino Festival Wristbands). However, debug log messages may include such sensitive information and those are not redacted.
-
Therefore, DO NOT USE debug LOG LEVEL IN PRODUCTION! Instead, use either info or error.
-
-
-Reconciling an AuthConfig and 2 related API key secrets
-
{"level":"debug","ts":1669221208.7473805,"logger":"authorino.controller-runtime.manager.controller.authconfig.statusupdater","msg":"resource status changed","authconfig":"default/talker-api-protection","authconfig/status":{"conditions":[{"type":"Available","status":"False","lastTransitionTime":"2022-11-23T16:33:28Z","reason":"HostsNotLinked","message":"No hosts linked to the resource"},{"type":"Ready","status":"False","lastTransitionTime":"2022-11-23T16:33:28Z","reason":"Unknown"}],"summary":{"ready":false,"hostsReady":[],"numHostsReady":"0/1","numIdentitySources":1,"numMetadataSources":0,"numAuthorizationPolicies":0,"numResponseItems":0,"festivalWristbandEnabled":false}}}
-{"level":"info","ts":1669221208.7496614,"logger":"authorino.controller-runtime.manager.controller.authconfig","msg":"resource reconciled","authconfig":"default/talker-api-protection"}
-{"level":"info","ts":1669221208.7532616,"logger":"authorino.controller-runtime.manager.controller.authconfig","msg":"resource reconciled","authconfig":"default/talker-api-protection"}
-{"level":"debug","ts":1669221208.7535005,"logger":"authorino.controller.secret","msg":"adding k8s secret to the index","reconciler group":"","reconciler kind":"Secret","name":"api-key-1","namespace":"default","authconfig":"default/talker-api-protection","config":"friends"}
-{"level":"debug","ts":1669221208.7535596,"logger":"authorino.controller.secret.apikey","msg":"api key added","reconciler group":"","reconciler kind":"Secret","name":"api-key-1","namespace":"default"}
-{"level":"info","ts":1669221208.7536132,"logger":"authorino.controller-runtime.manager.controller.secret","msg":"resource reconciled","secret":"default/api-key-1"}
-{"level":"info","ts":1669221208.753772,"logger":"authorino.controller-runtime.manager.controller.authconfig.statusupdater","msg":"resource status updated","authconfig":"default/talker-api-protection"}
-{"level":"debug","ts":1669221208.753835,"logger":"authorino.controller-runtime.manager.controller.authconfig.statusupdater","msg":"resource status changed","authconfig":"default/talker-api-protection","authconfig/status":{"conditions":[{"type":"Available","status":"True","lastTransitionTime":"2022-11-23T16:33:28Z","reason":"HostsLinked"},{"type":"Ready","status":"True","lastTransitionTime":"2022-11-23T16:33:28Z","reason":"Reconciled"}],"summary":{"ready":true,"hostsReady":["talker-api.127.0.0.1.nip.io"],"numHostsReady":"1/1","numIdentitySources":1,"numMetadataSources":0,"numAuthorizationPolicies":0,"numResponseItems":0,"festivalWristbandEnabled":false}}}
-{"level":"info","ts":1669221208.7571108,"logger":"authorino.controller-runtime.manager.controller.authconfig","msg":"resource reconciled","authconfig":"default/talker-api-protection"}
-{"level":"info","ts":1669221208.7573664,"logger":"authorino.controller-runtime.manager.controller.authconfig.statusupdater","msg":"resource status updated","authconfig":"default/talker-api-protection"}
-{"level":"debug","ts":1669221208.757429,"logger":"authorino.controller-runtime.manager.controller.authconfig.statusupdater","msg":"resource status did not change","authconfig":"default/talker-api-protection"}
-{"level":"debug","ts":1669221208.7586699,"logger":"authorino.controller.secret","msg":"adding k8s secret to the index","reconciler group":"","reconciler kind":"Secret","name":"api-key-2","namespace":"default","authconfig":"default/talker-api-protection","config":"friends"}
-{"level":"debug","ts":1669221208.7586884,"logger":"authorino.controller.secret.apikey","msg":"api key added","reconciler group":"","reconciler kind":"Secret","name":"api-key-2","namespace":"default"}
-{"level":"info","ts":1669221208.7586913,"logger":"authorino.controller-runtime.manager.controller.secret","msg":"resource reconciled","secret":"default/api-key-2"}
-{"level":"debug","ts":1669221208.7597604,"logger":"authorino.controller-runtime.manager.controller.authconfig.statusupdater","msg":"resource status did not change","authconfig":"default/talker-api-protection"}
-
-
-
-Enforcing an AuthConfig with authentication based on Kubernetes tokens:
-
{"level":"info","ts":1669221635.0135982,"logger":"authorino","msg":"Stopping and waiting for non leader election runnables"}
-{"level":"info","ts":1669221635.0136683,"logger":"authorino","msg":"Stopping and waiting for leader election runnables"}
-{"level":"info","ts":1669221635.0135982,"logger":"authorino","msg":"Stopping and waiting for non leader election runnables"}
-{"level":"info","ts":1669221635.0136883,"logger":"authorino","msg":"Stopping and waiting for leader election runnables"}
-{"level":"info","ts":1669221635.0137057,"logger":"authorino.controller.secret","msg":"Shutdown signal received, waiting for all workers to finish","reconciler group":"","reconciler kind":"Secret"}
-{"level":"info","ts":1669221635.013724,"logger":"authorino.controller.authconfig","msg":"Shutdown signal received, waiting for all workers to finish","reconciler group":"authorino.kuadrant.io","reconciler kind":"AuthConfig"}
-{"level":"info","ts":1669221635.01375,"logger":"authorino.controller.authconfig","msg":"All workers finished","reconciler group":"authorino.kuadrant.io","reconciler kind":"AuthConfig"}
-{"level":"info","ts":1669221635.013752,"logger":"authorino.controller.secret","msg":"All workers finished","reconciler group":"","reconciler kind":"Secret"}
-{"level":"info","ts":1669221635.0137632,"logger":"authorino","msg":"Stopping and waiting for caches"}
-{"level":"info","ts":1669221635.013751,"logger":"authorino.controller.authconfig","msg":"Shutdown signal received, waiting for all workers to finish","reconciler group":"authorino.kuadrant.io","reconciler kind":"AuthConfig"}
-{"level":"info","ts":1669221635.0137684,"logger":"authorino.controller.authconfig","msg":"All workers finished","reconciler group":"authorino.kuadrant.io","reconciler kind":"AuthConfig"}
-{"level":"info","ts":1669221635.0137722,"logger":"authorino","msg":"Stopping and waiting for caches"}
-{"level":"info","ts":1669221635.0138857,"logger":"authorino","msg":"Stopping and waiting for webhooks"}
-{"level":"info","ts":1669221635.0138955,"logger":"authorino","msg":"Wait completed, proceeding to shutdown the manager"}
-{"level":"info","ts":1669221635.0138893,"logger":"authorino","msg":"Stopping and waiting for webhooks"}
-{"level":"info","ts":1669221635.0139785,"logger":"authorino","msg":"Wait completed, proceeding to shutdown the manager"}
-
Processes related to the authorization request are identified and linked together by a request ID. The request ID can be:
-
-
generated outside Authorino and passed in the authorization request – this is essentially the case of requests via GRPC authorization interface initiated by the Envoy;
Most log messages associated with an authorization request include the request id value. This value can be used to match incoming request and corresponding outgoing response log messages, including at deep level when more fine-grained log details are enabled (debug level level).
Integration with an OpenTelemetry collector can be enabled by supplying the --tracing-service-endpoint command-line flag (e.g. authorino server --tracing-service-endpoint=http://jaeger:14268/api/traces).
-
The additional --tracing-service-tags command-line flag allow to specify fixed agent-level key-value tags for the trace signals emitted by Authorino (e.g. authorino server --tracing-service-endpoint=... --tracing-service-tag=key1=value1 --tracing-service-tag=key2=value2).
-
Traces related to authorization requests are additionally tagged with the authorino.request_id attribute.
User guide: OpenID Connect Discovery and authentication with JWTs¶
-
Validate JSON Web Tokens (JWT) issued and signed by an OpenID Connect server; leverage OpenID Connect Discovery to automatically fetch JSON Web Key Sets (JWKS).
-
-
- Authorino capabilities featured in this guide:
-
Authorino validates JSON Web Tokens (JWT) issued by an OpenID Connect server that implements OpenID Connect Discovery. Authorino fetches the OpenID Connect configuration and JSON Web Key Set (JWKS) from the issuer endpoint, and verifies the JSON Web Signature (JWS) and time validity of the token.
-
Important! Authorino does not implement OAuth2 grants nor OIDC authentication flows. As a common recommendation of good practice, obtaining and refreshing access tokens is for clients to negotiate directly with the auth servers and token issuers. Authorino will only validate those tokens using the parameters provided by the trusted issuer authorities.
-
For further details about Authorino features in general, check the docs.
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
Deploy the identity provider and authentication server by executing the command below. For the examples in this guide, we are going to use a Keycloak server preloaded with all required realm settings.
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❺.
-
At step ❺, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
❻ Obtain an access token with the Keycloak server¶
-
The AuthConfig deployed in the previous step is suitable for validating access tokens requested inside the cluster. This is because Keycloak's iss claim added to the JWTs matches always the host used to request the token and Authorino will later try to match this host to the host that provides the OpenID Connect configuration.
If your Keycloak server is reachable from outside the cluster, feel free to obtain the token directly. Make sure the host name set in the OIDC issuer endpoint in the AuthConfig matches the one used to obtain the token and is as well reachable from within the cluster.
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
User guide: OpenID Connect (OIDC) and Role-Based Access Control (RBAC) with Authorino and Keycloak¶
-
Combine OpenID Connect (OIDC) authentication and Role-Based Access Control (RBAC) authorization rules leveraging Keycloak and Authorino working together.
-
In this user guide, you will learn via example how to implement a simple Role-Based Access Control (RBAC) system to protect endpoints of an API, with roles assigned to users of an Identity Provider (Keycloak) and carried within the access tokens as JSON Web Token (JWT) claims. Users authenticate with the IdP via OAuth2/OIDC flow and get their access tokens verified and validated by Authorino on every request. Moreover, Authorino reads the role bindings of the user and enforces the proper RBAC rules based upon the context.
-
-
- Authorino capabilities featured in this guide:
-
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
Deploy the identity provider and authentication server by executing the command below. For the examples in this guide, we are going to use a Keycloak server preloaded with all required realm settings.
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❺.
-
At step ❺, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
Create an Authorino AuthConfig custom resource declaring the auth rules to be enforced.
-
In this example, the Keycloak realm defines a few users and 2 realm roles: 'member' and 'admin'. When users authenticate to the Keycloak server by any of the supported OAuth2/OIDC flows, Keycloak adds to the access token JWT a claim "realm_access": { "roles": array } that holds the list of roles assigned to the user. Authorino will verify the JWT on requests to the API and read from that claim to enforce the following RBAC rules:
-
-
-
-
Path
-
Method
-
Role
-
-
-
-
-
/resources[/*]
-
GET / POST / PUT
-
member
-
-
-
/resources/{id}
-
DELETE
-
admin
-
-
-
/admin[/*]
-
*
-
admin
-
-
-
-
-
-
-
- Kuadrant users –
- Remember to create an AuthPolicy instead of an AuthConfig.
- For more, see Kuadrant auth.
-
Obtain an access token and consume the API as John (member)¶
-
Obtain an access token with the Keycloak server for John:
-
The AuthConfig deployed in the previous step is suitable for validating access tokens requested inside the cluster. This is because Keycloak's iss claim added to the JWTs matches always the host used to request the token and Authorino will later try to match this host to the host that provides the OpenID Connect configuration.
-
Obtain an access token from within the cluster for the user John, who is assigned to the 'member' role:
If your Keycloak server is reachable from outside the cluster, feel free to obtain the token directly. Make sure the host name set in the OIDC issuer endpoint in the AuthConfig matches the one used to obtain the token and is as well reachable from within the cluster.
-
As John, send a GET request to /resources:
-
curl-H"Authorization: Bearer $ACCESS_TOKEN"http://talker-api.127.0.0.1.nip.io:8000/resources-i
-# HTTP/1.1 200 OK
-
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
Apart from possibly complementing information of the JWT, fetching OpenID Connect UserInfo in request-time can be particularly useful for remote checking the state of the session, as opposed to only verifying the JWT/JWS offline. Implementation requires an OpenID Connect issuer (spec.identity.oidc) configured in the same AuthConfig.
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
Deploy the identity provider and authentication server by executing the command below. For the examples in this guide, we are going to use a Keycloak server preloaded with all required realm settings.
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❺.
-
At step ❺, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
❻ Obtain an access token with the Keycloak server¶
-
The AuthConfig deployed in the previous step is suitable for validating access tokens requested inside the cluster. This is because Keycloak's iss claim added to the JWTs matches always the host used to request the token and Authorino will later try to match this host to the host that provides the OpenID Connect configuration.
If your Keycloak server is reachable from outside the cluster, feel free to obtain the token directly. Make sure the host name set in the OIDC issuer endpoint in the AuthConfig matches the one used to obtain the token and is as well reachable from within the cluster.
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
User guide: Open Policy Agent (OPA) Rego policies¶
-
Leverage the power of Open Policy Agent (OPA) policies, evaluated against Authorino's Authorization JSON in a built-in runtime compiled together with Authorino; pre-cache policies defined in Rego language inline or fetched from an external policy registry.
-
-
- Authorino capabilities featured in this guide:
-
Authorino supports Open Policy Agent policies, either inline defined in Rego language as part of the AuthConfig or fetched from an external endpoint, such as an OPA Policy Registry.
-
Authorino's built-in OPA module precompiles the policies in reconciliation-time and cache them for fast evaluation in request-time, where they receive the Authorization JSON as input.
Kubernetes server with permissions to install cluster-scoped resources (operator, CRDs and RBAC)
-
-
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
-
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❺.
-
At step ❺, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
-
-
-
You can also set use_remote_address: true in the Envoy route configuration, so the proxy will append its IP address instead of run in transparent mode. This setting will also ensure real remote address of the client connection passed in the x-envoy-external-address HTTP header, which can be used to simplify the read-only policy in remote environment. ↩
Kubernetes server with permissions to install cluster-scoped resources (operator, CRDs and RBAC)
-
-
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
-
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❺.
-
At step ❺, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
Create an Authorino AuthConfig custom resource declaring the auth rules to be enforced.
-
In this example, member users can authenticate supplying the API key in any of 4 different ways:
-- HTTP header Authorization: APIKEY <api-key>
-- HTTP header X-API-Key: <api-key>
-- Query string parameter api_key=<api-key>
-- Cookie Cookie: APIKEY=<api-key>;
-
admin API keys are only accepted in the (default) HTTP header Authorization: Bearer <api-key>.
-
-
-
-
- Kuadrant users –
- Remember to create an AuthPolicy instead of an AuthConfig.
- For more, see Kuadrant auth.
-
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
Deploy the identity provider and authentication server by executing the command below. For the examples in this guide, we are going to use a Keycloak server preloaded with all required realm settings.
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❺.
-
At step ❺, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
Create an Authorino AuthConfig custom resource declaring the auth rules to be enforced:
-
This example of resource-level authorization leverages part of Keycloak's User-Managed Access (UMA) support. Authorino will fetch resource attributes stored in a Keycloak resource server client.
-
The Keycloak server also provides the identities. The sub claim of the Keycloak-issued ID tokens must match the owner of the requested resource, identified by the URI of the request.
-
-
-
-
- Kuadrant users –
- Remember to create an AuthPolicy instead of an AuthConfig.
- For more, see Kuadrant auth.
-
-
-
-
-
-
Create a required secret that will be used by Authorino to initiate the authentication with the UMA registry.
The OPA policy owned-resource above enforces that all users can send GET and POST requests to /greetings, while only resource owners can send GET, PUT and DELETE requests to /greetings/{resource-id}.
-
❻ Obtain access tokens with the Keycloak server and consume the API¶
-
Obtain an access token as John and consume the API¶
-
Obtain an access token for user John (owner of the resource /greetings/1 in the UMA registry):
-
The AuthConfig deployed in the previous step is suitable for validating access tokens requested inside the cluster. This is because Keycloak's iss claim added to the JWTs matches always the host used to request the token and Authorino will later try to match this host to the host that provides the OpenID Connect configuration.
If your Keycloak server is reachable from outside the cluster, feel free to obtain the token directly. Make sure the host name set in the OIDC issuer endpoint in the AuthConfig matches the one used to obtain the token and is as well reachable from within the cluster.
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
By default, Authorino will watch events related to all AuthConfig custom resources in the reconciliation space (namespace or entire cluster). Instances can be configured though to only watch a subset of the resources, thus allowing such as:
-
-
to reduce noise and lower memory usage inside instances meant for restricted scope (e.g. Authorino deployed as a dedicated sidecar to protect only one host);
-
sharding auth config data across multiple instances;
-
multiple environments (e.g. staging, production) inside of a same cluster/namespace;
-
providing managed instances of Authorino that all watch CRs cluster-wide, yet dedicated to organizations allowed to create and operate their own AuthConfigs across multiple namespaces.
-
-
-
-
-
- ⚠️ Important:
- This feature may not be available to users of Authorino via Kuadrant.
-
-
-
-
-
-
-
- Authorino capabilities featured in this guide:
-
Kubernetes server with permissions to install cluster-scoped resources (operator, CRDs and RBAC)
-
-
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
Broadly, the term token normalization in authentication systems usually implies the exchange of an authentication token, as provided by the user in a given format, and/or its associated identity claims, for another freshly issued token/set of claims, of a given (normalized) structure or format.
-
The most typical use-case for token normalization involves accepting tokens issued by multiple trusted sources and of often varied authentication protocols, while ensuring that the eventual different data structures adopted by each of those sources are normalized, thus allowing to simplify policies and authorization checks that depend on those values. In general, however, any modification to the identity claims can be for the purpose of normalization.
-
This user guide focuses on the aspect of mutation of the identity claims resolved from an authentication token, to a certain data format and/or by extending them, so that required attributes can thereafter be trusted to be present among the claims, in a desired form. For such, Authorino allows to extend resolved identity objects with custom attributes (custom claims) of either static values or with values fetched from the Authorization JSON.
-
For not only normalizing the identity claims for purpose of writing simpler authorization checks and policies, but also getting Authorino to issue a new token in a normalized format, check the Festival Wristband tokens feature.
-
-
- Authorino capabilities featured in this guide:
-
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
Deploy the identity provider and authentication server by executing the command below. For the examples in this guide, we are going to use a Keycloak server preloaded with all required realm settings.
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant and already have your workload cluster configured and sample service application deployed, as well as your Gateway API network resources applied to route traffic to your service, skip straight to step ❺.
-
At step ❺, instead of creating an AuthConfig custom resource, create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources in the default namespace2, with TLS disabled3.
The Talker API is a simple HTTP service that echoes back in the response whatever it gets in the request. We will use it in this guide as the sample service to be protected by Authorino.
The following bundle from the Authorino examples deploys the Envoy proxy and configuration to wire up the Talker API behind the reverse-proxy, with external authorization enabled with the Authorino instance.4
The command above creates an Ingress with host name talker-api.127.0.0.1.nip.io. If you are using a local Kubernetes cluster created with Kind, forward requests from your local port 8000 to the Envoy service running inside the cluster:
Create an Authorino AuthConfig custom resource declaring the auth rules to be enforced.
-
This example implements a policy that only users bound to the admin role can send DELETE requests.
-
The config trusts access tokens issued by a Keycloak realm as well as API keys labeled specifically to a selected group (friends). The roles of the identities handled by Keycloak are managed in Keycloak, as realm roles. Particularly, users john and peter are bound to the member role, while user jane is bound to roles member and admin. As for the users authenticating with API key, they are all bound to the admin role.
-
Without normalizing identity claims from these two different sources, the policy would have to handle the differences of data formats with additional ifs-and-elses. Instead, the config here uses the identity.extendedProperties option to ensure a custom roles (Array) claim is always present in the identity object. In the case of Keycloak ID tokens, the value is extracted from the realm_access.roles claim; for API key-resolved objects, the custom claim is set to the static value ["admin"].
-
-
-
-
- Kuadrant users –
- Remember to create an AuthPolicy instead of an AuthConfig.
- For more, see Kuadrant auth.
-
Obtain an access token and consume the API as Jane (admin)¶
-
Obtain an access token with the Keycloak server for Jane:
-
The AuthConfig deployed in the previous step is suitable for validating access tokens requested inside the cluster. This is because Keycloak's iss claim added to the JWTs matches always the host used to request the token and Authorino will later try to match this host to the host that provides the OpenID Connect configuration.
-
Obtain an access token from within the cluster for the user Jane, whose e-mail has been verified:
If your Keycloak server is reachable from outside the cluster, feel free to obtain the token directly. Make sure the host name set in the OIDC issuer endpoint in the AuthConfig matches the one used to obtain the token and is as well reachable from within the cluster.
-
Consume the API as Jane:
-
curl-H"Authorization: Bearer $ACCESS_TOKEN"-XDELETEhttp://talker-api.127.0.0.1.nip.io:8000/hello-i
-# HTTP/1.1 200 OK
-
-
Obtain an access token and consume the API as John (member)¶
-
Obtain an access token with the Keycloak server for John:
For other variants and deployment options, check out Getting Started, as well as the Authorino CRD specification. ↩
-
-
-
For details and instructions to setup Envoy manually, see Protect a service > Setup Envoy in the Getting Started page. If you are running your ingress gateway in Kubernetes and wants to avoid setting up and configuring your proxy manually, check out Kuadrant. ↩
User guide: Using Authorino as ValidatingWebhook service¶
-
Authorino provides an interface for raw HTTP external authorization requests. This interface can be used for integrations other than the typical Envoy gRPC protocol, such as (though not limited to) using Authorino as a generic Kubernetes ValidatingWebhook service.
-
The rules to validate a request to the Kubernetes API – typically a POST, PUT or DELETE request targeting a particular Kubernetes resource or collection –, according to which either the change will be deemed accepted or not, are written in an Authorino AuthConfig custom resource. Authentication and authorization are performed by the Kubernetes API server as usual, with auth features of Authorino implementing the additional validation within the scope of an AdmissionReview request.
-
This user guide provides an example of using Authorino as a Kubernetes ValidatingWebhook service that validates requests to CREATE and UPDATE Authorino AuthConfig resources. In other words, we will use Authorino as a validator inside the cluster that decides what is a valid AuthConfig for any application which wants to rely on Authorino to protect itself.
-
-
- Authorino capabilities featured in this guide:
-
Kubernetes server with permissions to install cluster-scoped resources (operator, CRDs and RBAC)
-
Identity Provider (IdP) that implements OpenID Connect authentication and OpenID Connect Discovery (e.g. Keycloak)
-
-
If you do not own a Kubernetes server already and just want to try out the steps in this guide, you can create a local containerized cluster by executing the command below. In this case, the main requirement is having Kind installed, with either Docker or Podman.
-
kindcreatecluster--nameauthorino-tutorial
-
-
Deploy the identity provider and authentication server. For the examples in this guide, we are going to use a Keycloak server preloaded with all required realm settings.
-
The Keycloak server is only needed for trying out validating AuthConfig resources that use the authentication server.
The next steps walk you through installing Authorino, deploying and configuring a sample service called Talker API to be protected by the authorization service.
-
-
-
-
Using Kuadrant
-
-
-
-
-
-
If you are a user of Kuadrant you may already have Authorino installed and running. In this case, skip straight to step ❸.
-
At step ❺, alternatively to creating an AuthConfig custom resource, you may create a Kuadrant AuthPolicy one. The schema of the AuthConfig's spec matches the one of the AuthPolicy's, except spec.host, which is not available in the Kuadrant AuthPolicy. Host names in a Kuadrant AuthPolicy are inferred automatically from the Kubernetes network object referred in spec.targetRef and route selectors declared in the policy.
-
For more about using Kuadrant to enforce authorization, check out Kuadrant auth.
-
-
-
-
-
-
-
❶ Install the Authorino Operator (cluster admin required)¶
-
The following command will install the Authorino Operator in the Kubernetes cluster. The operator manages instances of the Authorino authorization service.
The following command will request an instance of Authorino as a separate service1 that watches for AuthConfig resources cluster-wide2, with TLS enabled3.
-
kubectl-nauthorinoapply-f-<<EOF
-apiVersion: operator.authorino.kuadrant.io/v1beta1
-kind: Authorino
-metadata:
- name: authorino
-spec:
- clusterWide: true
- listener:
- ports:
- grpc: 50051
- http: 5001 # for admissionreview requests sent by the kubernetes api server
- tls:
- certSecretRef:
- name: authorino-server-cert
- oidcServer:
- tls:
- certSecretRef:
- name: authorino-oidc-server-cert
-EOF
-
-
For convenience, the same instance of Authorino pointed as the validating webhook will also be targeted for the sample AuthConfigs created to test the validation. For using different instances of Authorino for the validating webhook and for protecting applications behind a proxy, check out the section about sharding in the docs. There is also a user guide on the topic, with concrete examples.
Create the AuthConfig with the auth rules to validate other AuthConfig resources applied to the cluster.
-
The AuthConfig to validate other AuthConfigs will enforce the following rules:
-- Authorino features that cannot be used by any application in their security schemes:
- - Anonymous Access
- - Plain identity object extracted from context
- - Kubernetes authentication (TokenReview)
- - Kubernetes authorization (SubjectAccessReview)
- - Festival Wristband tokens
-- Authorino features that require a RoleBinding to a specific ClusterRole in the 'authorino' namespace, to be used in a AuthConfig:
- - Authorino API key authentication
-- All metadata pulled from external sources must be cached for precisely 5 minutes (300 seconds)
-
kubectl-nauthorinoapply-f-<<EOF
-apiVersion: authorino.kuadrant.io/v1beta2
-kind: AuthConfig
-metadata:
- name: authconfig-validator
-spec:
- # admissionreview requests will be sent to this host name
- hosts:
- - authorino-authorino-authorization.authorino.svc
-
- # because we're using a single authorino instance for the validating webhook and to protect the user applications,
- # skip operations related to this one authconfig in the 'authorino' namespace
- when:
- - selector: context.request.http.body.@fromstr|request.object.metadata.namespace
- operator: neq
- value: authorino
-
- # kubernetes admissionreviews carry info about the authenticated user
- authentication:
- "k8s-userinfo":
- plain:
- selector: context.request.http.body.@fromstr|request.userInfo
-
- authorization:
- "features":
- opa:
- rego: |
- authconfig = json.unmarshal(input.context.request.http.body).request.object
-
- forbidden { count(object.get(authconfig.spec, "authentication", [])) == 0 }
- forbidden { authconfig.spec.authentication[_].anonymous }
- forbidden { authconfig.spec.authentication[_].kubernetesTokenReview }
- forbidden { authconfig.spec.authentication[_].plain }
- forbidden { authconfig.spec.authorization[_].kubernetesSubjectAccessReview }
- forbidden { authconfig.spec.response.success.headers[_].wristband }
-
- apiKey { authconfig.spec.authentication[_].apiKey }
-
- allow { count(authconfig.spec.authentication) > 0; not forbidden }
- allValues: true
-
- "apikey-authn-requires-k8s-role-binding":
- priority: 1
- when:
- - selector: auth.authorization.features.apiKey
- operator: eq
- value: "true"
- kubernetesSubjectAccessReview:
- user:
- selector: auth.identity.username
- resourceAttributes:
- namespace: { value: authorino }
- group: { value: authorino.kuadrant.io }
- resource: { value: authconfigs-with-apikeys }
- verb: { value: create }
-
- "metadata-cache-ttl":
- priority: 1
- opa:
- rego: |
- invalid_ttl = input.auth.authorization.features.authconfig.spec.metadata[_].cache.ttl != 300
- allow { not invalid_ttl }
-EOF
-
-
Define a ClusterRole to control the usage of protected features of Authorino:
-
kubectlapply-f-<<EOF
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
- name: authorino-apikey
-rules:
-- apiGroups: ["authorino.kuadrant.io"]
- resources: ["authconfigs-with-apikeys"] # not a real k8s resource
- verbs: ["create"]
-EOF
-
- Kuadrant users –
- For this and other example AuthConfigs below, if you create a Kuadrant AuthPolicy instead, the output of the commands shall differ. The requested AuthPolicy may be initially accepted, but its state will turn ready or not ready depending on whether the corresponding AuthConfig requested by Kuadrant is accepted or rejected, according to the validating webhook rules. Check the state of the resources to confirm.
- For more, see Kuadrant auth.
-
Kubernetes-native authorization service for tailor-made Zero Trust API security.
-
A lightweight Envoy external authorization server fully manageable via Kubernetes Custom Resources.
-JWT authentication, API key, mTLS, pattern-matching authz, OPA, K8s SA tokens, K8s RBAC, external metadata fetching, and more, with minimum to no coding at all, no rebuilding of your applications.
-
Authorino is not about inventing anything new. It's about making the best things about auth out there easy and simple to use. Authorino is multi-tenant, it's cloud-native and it's open source.
The User guides section of the docs gathers several AuthN/AuthZ use-cases as well as the instructions to implement them using Authorino. A few examples are:
Authorino enables hybrid API security, with usually no code changes required to your application, tailor-made for your own combination of authentication standards and protocols and authorization policies of choice.
Deploy the service/API to be protected ("Upstream"), Authorino and Envoy
-
Write and apply an Authorino AuthConfig Custom Resource associated to the public host of the service
-
-
Request-time:
-
-
-
-
-
-
A user or service account ("Consumer") obtains an access token to consume resources of the Upstream service, and sends a request to the Envoy ingress endpoint
-
The Envoy proxy establishes fast gRPC connection with Authorino carrying data of the HTTP request (context info), which causes Authorino to lookup for an AuthConfig Custom Resource to enforce (pre-cached)
-
Identity verification (authentication) phase - Authorino verifies the identity of the consumer, where at least one authentication method/identity provider must go through
-
External metadata phase - Authorino fetches additional metadata for the authorization from external sources (optional)
-
Policy enforcement (authorization) phase - Authorino takes as input a JSON composed out of context data, resolved identity object and fetched additional metadata from previous phases, and triggers the evaluation of user-defined authorization policies
-
Response (metadata-out) phase – Authorino builds user-defined custom responses (dynamic JSON objects and/or Festival Wristband OIDC tokens), to be supplied back to the client and/or upstream service within added HTTP headers or as Envoy Dynamic Metadata (optional)
Authorino and Envoy settle the authorization protocol with either OK/NOK response
-
If authorized, Envoy triggers other HTTP filters in the chain (if any), pre-injecting eventual dynamic metadata returned by Authorino, and ultimately redirects the request to the Upstream
-
The Upstream serves the requested resource to the consumer
-
-
-More
-
The Architecture section of the docs covers details of protecting your APIs with Envoy and Authorino, including information about topology (centralized gateway, centralized authorization service or sidecars), deployment modes (cluster-wide reconciliation vs. namespaced instances), an specification of Authorino's AuthConfig Custom Resource Definition (CRD) and more.
-
You will also find in that section information about what happens in request-time (aka Authorino's Auth Pipeline) and how to leverage the Authorization JSON for writing policies, dynamic responses and other features of Authorino.
Authorino is built from the ground up to work well with Envoy. It is strongly recommended that you leverage Envoy along side Authorino. That said, it is possible to use Authorino without Envoy.
-
Authorino implements Envoy's external authorization gRPC protocol and therefore will accept any client request that complies.
-
Authorino also provides a second interface for raw HTTP authorization, suitable for using with Kubernetes ValidatingWebhook and other integrations (e.g. other proxies).
-
The only attribute of the authorization request that is strictly required is the host name. (See Host lookup for more information.) The other attributes, such as method, path, headers, etc, might as well be required, depending on each AuthConfig. In the case of the gRPC CheckRequest method, the host is supplied in Attributes.Request.Http.Host and alternatively in Attributes.ContextExtensions["host"]. For raw HTTP authorization requests, the host must be supplied in Host HTTP header.
-
Check out Kuadrant for easy-to-use Envoy and Authorino deployment & configuration for API management use-cases, using Kubernetes Custom Resources.
-
-
-Is Authorino an Identity Provider (IdP)?
-
No, Authorino is not an Identity Provider (IdP). Neither it is an auth server of any kind, such as an OAuth2 server, an OpenID Connect (OIDC) server, a Single Sign On (SSO) server.
-
Authorino is not an identity broker either. It can verify access tokens from multiple trusted sources of identity and protocols, but it will not negotiate authentication flows for non-authenticated access requests. Some tricks nonetheless can be done, for example, to redirect unauthenticated users to a login page.
-
For an excellent auth server that checks all the boxes above, check out Keycloak.
-
-
-How does Authorino compare to Keycloak?
-
Keycloak is a proper auth server and identity provider (IdP). It offers a huge set of features for managing identities, identity sources with multiple user federation options, and a platform for authentication and authorization services.
-
Keycloak exposes authenticators that implement protocols such as OpenID Connect. The is a one-time flow that establishes the delegation of power to a client, for a short period of time. To be consistent with Zero Trust security, you want a validator to verify the short-lived tokens in every request that tries to reach your protected service/resource. This step that will repeat everytime could save heavy looking up into big tables of tokens and leverage cached authorization policies for fast in-memory evaluation. This is where Authorino comes in.
-
Authorino verifies and validates Keycloak-issued ID tokens. OpenID Connect Discovery is used to request and cache JSON Web Key Sets (JWKS), used to verify the signature of the tokens without having to contact again with the Keycloak server, or looking in a table of credentials. Moreover, user long-lived credentials are safe, rather than spread in hops across the network.
-
You can also use Keycloak for storing auth-relevant resource metadata. These can be fetched by Authorino in request-time, to be combined into your authorization policies. See Keycloak Authorization Services and User-Managed Access (UMA) support, as well as Authorino UMA external metadata counter-part.
-
-
-Why doesn't Authorino handle OAuth flows?
-
It has to do with trust. OAuth grants are supposed to be negotiated directly between whoever owns the long-lived credentials in one hand (user, service accounts), and the trustworthy auth server that receives those credentials – ideally with minimum number of hops in the middle – and exchanges them for short-lived access tokens, on the other end.
-
There are use-cases for Authorino running in the edge (e.g. Edge Authentication Architecture and token normalization), but in most cases Authorino should be seen as a last-mile component that provides decoupled identity verification and authorization policy enforcement to protected services in request-time. In this sense, the OAuth grant is a pre-flight exchange that happens once and as direct and safe as possible, whereas auth enforcement is kept lightweight and efficient.
-
-
-Where does Authorino store users and roles?
-
Authorino does not store users, roles, role bindings, access control lists, or any raw authorization data. Authorino handles policies, where even these policies can be stored elsewhere (as opposed to stated inline inside of an Authorino AuthConfig CR).
-
Authorino evaluates policies for stateless authorization requests. Any additional context is either resolved from the provided payload or static definitions inside the policies. That includes extracting user information from a JWT or client TLS certificate, requesting user metadata from opaque authentication tokens (e.g. API keys) to the trusted sources actually storing that content, obtaining synchronous HTTP metadata from services, etc.
-
In the case of authentication with API keys, as well as its derivative to model HTTP Basic Auth, user data are stored in Kubernetes Secrets. The secret's keys, annotations and labels are usually the structures used to organize the data that later a policy evaluated in Authorino may require. Strictly, those are not Authorino data structures.
-
-
-Can't I just use Envoy JWT Authentication and RBAC filters?
-
Envoy's JWT Authentication works pretty much similar to Authorino's JOSE/JWT verification and validation for OpenID Connect. In both cases, the JSON Web Key Sets (JWKS) to verify the JWTs are auto-loaded and cached to be used in request-time. Moreover, you can configure for details such as where to extract the JWT from the HTTP request (header, param or cookie) and do some cool tricks regarding how dynamic metadata based on JWT claims can be injected to consecutive filters in the chain.
-
However, in terms of authorization, while Envoy's implementation essentially allows to check for the list of audiences (aud JWT claim), Authorino opens up for a lot more options such as pattern-matching rules with operators and conditionals, built-in OPA and other methods of evaluating authorization policies.
-
Authorino also allows to combine JWT authentication with other types of authentication to support different sources of identity and groups of users such as API keys, Kubernetes tokens, OAuth opaque tokens , etc.
-
In summary, Envoy's JWT Authentication and Envoy RBAC filter are excellent solutions for simple use-cases where JWTs from one single issuer is the only authentication method you are planning to support and limited to no authorization rules suffice. On the other hand, if you need to integrate more identity sources, different types of authentication, authorization policies, etc, you might to consider Authorino.
-
-
-Should I use Authorino if I already have Istio configured?
-
Istio is a great solution for managing service meshes. It delivers an excellent platform with an interesting layer of abstraction on top of Envoy proxy's virtual omnipresence within the mesh.
-
There are lots of similarities, but also complementarity between Authorino and Istio and Istio Authorization in special.
-
Istio provides a simple way to enable features that are, in many cases, features of Envoy, such as authorization based on JWTs, authorization based on attributes of the request, and activation of external authorization services, without having to deal with complex Envoy config files. See Kuadrant for a similar approach, nonetheless leveraging features of Istio as well.
-
Authorino is an Envoy-compatible external authorization service. One can use Authorino with or without Istio.
-
In particular, Istio Authorization Policies can be seen, in terms of functionality and expressiveness, as a subset of one type of authorization policies supported by Authorino, the pattern-matching authorization policies. While Istio, however, is heavily focused on specific use cases of API Management, offering a relatively limited list of supported attribute conditions, Authorino is more generic, allowing to express authorization rules for a wider spectrum of use cases – ACLs, RBAC, ABAC, etc, pretty much counting on any attribute of the Envoy payload, identity object and external metadata available.
-
Authorino also provides built-in OPA authorization, several other methods of authentication and identity verification (e.g. Kubernetes token validation, API key-based authentication, OAuth token introspection, OIDC-discoverable JWT verification, etc), and features like fetching of external metadata (HTTP services, OIDC userinfo, UMA resource data), token normalization, wristband tokens and dynamic responses. These all can be used independently or combined, in a simple and straightforward Kubernetes-native fashion.
-
In summary, one might value Authorino when looking for a policy enforcer that offers:
-
-
multiple supported methods and protocols for rather hybrid authentication, encompassing future and legacy auth needs;
-
broader expressiveness and more functionalities for the authorization rules;
-
authentication and authorization in one single declarative manifest;
-
capability to fetch auth metadata from external sources on-the-fly;
-
built-in OPA module;
-
easy token normalization and/or aiming for Edge Authentication Architecture (EAA).
-
-
The good news is that, if you have Istio configured, then you have Envoy and the whole platform for wiring Authorino up if you want to. 😉
-
-
-Do I have to learn OPA/Rego language to use Authorino?
-
No, you do not. However, if you are comfortable with Rego from Open Policy Agent (OPA), there are some quite interesting things you can do in Authorino, just as you would in any OPA server or OPA plugin, but leveraging Authorino's built-in OPA module instead. Authorino's OPA module is compiled as part of Authorino's code directly from the Golang packages, and imposes no extra latency to the evaluation of your authorization policies. Even the policies themselves are pre-compiled in reconciliation-time, for fast evaluation afterwards, in request-time.
-
On the other hand, if you do not want to learn Rego or in any case would like to combine it with declarative and Kubernetes-native authN/authZ spec for your services, Authorino does complement OPA with at least two other methods for expressing authorization policies – i.e. pattern-matching authorization and Kubernetes SubjectAccessReview, the latter allowing to rely completely on the Kubernetes RBAC.
-
You break down, mix and combine these methods and technolgies in as many authorization policies as you want, potentially applying them according to specific conditions. Authorino will trigger the evaluation of concurrent policies in parallel, aborting the context if any of the processes denies access.
-
Authorino also packages well-established industry standards and protocols for identity verification (JOSE/JWT validation, OAuth token introspection, Kubernetes TokenReview) and ad-hoc request-time metadata fetching (OIDC userinfo, User-Managed Access (UMA)), and corresponding layers of caching, without which such functionalities would have to be implemented by code.
-
-
-Can I use Authorino to protect non-REST APIs?
-
Yes, you can. In principle, the API format (REST, gRPC, GraphQL, etc) should not matter for the authN/authZ enforcer. There are a couple points to consider though.
-
While REST APIs are designed in a way that, in most cases, information usually needed for the evaluation of authorization policies are available in the metadata of the HTTP request (method, path, headers), other API formats quite often will require processing of the HTTP body. By default, Envoy's external authorization HTTP filter will not forward the body of the request to Authorino; to change that, enable the with_request_body option in the Envoy configuration for the external authorization filter. E.g.:
Additionally, when enabling the request body passed in the payload to Authorino, parsing of the content should be of concern as well. Authorino provides easy access to attributes of the HTTP request, parsed as part of the Authorization JSON, however the body of the request is passed as string and should be parsed by the user according to each case.
-
Check out Authorino OPA authorization and the Rego Encoding functions for options to parse serialized JSON, YAML and URL-encoded params. For XML transformation, an external parsing service connected via Authorino's HTTP GET/GET-by-POST external metadata might be required.
-
-
-Can I run Authorino other than on Kubernetes?
-
As of today, no, you cannot, or at least it wouldn't suit production requirements.
-
-
-Do I have to be admin of the cluster to install Authorino?
-
To install the Authorino Custom Resource Definition (CRD) and to define cluster roles required by the Authorino service, admin privilege to the Kubernetes cluster is required. This step happens only once per cluster and is usually equivalent to installing the Authorino Operator.
-
Thereafter, deploying instances of the Authorino service and applying AuthConfig custom resources to a namespace depend on the permissions set by the cluster administrator – either directly by editing the bindings in the cluster's RBAC, or via options of the operator. In most cases, developers will be granted permissions to create and manage AuthConfigs, and sometimes to deploy their own instances of Authorino.
-
-
-Is it OK to store AuthN/AuthZ configs as Kubernetes objects?
-
By merging the definitions of service authN/authZ to the control plane, Authorino AuthConfig resources can be thought as extensions of the specs of the desired state of services regarding the data flow security. The Authorino custom controllers, built-in into the authorization service, are the agents that read from that desired state and reconcile the processes operating in the data plane.
-
Authorino is declarative and seamless for developers and cluster administrators managing the state of security of the applications running in the server, used to tools such as kubectl, the Kubernetes UI and its dashboards. Instead of learning about yet another configuration API format, Authorino users can jump straight to applying and editing YAML or JSON structures they already know, in a way that things such as spec, status, namespace and labels have the meaning they are expected to have, and docs are as close as kubectl explain. Moreover, Authorino does not pile up any other redundant layers of APIs, event-processing, RBAC, transformation and validation webhooks, etc. It is Kubernetes in its best.
-
In terms of scale, Authorino AuthConfigs should grow proportionally to the number of protected services, virtually limited by nothing but the Kubernetes API data storage, while namespace division and label selectors help adjust horizontally and keep distributed.
-
In other words, there are lots of benefits of using Kubernetes custom resources and custom controllers, and unless you are planning on bursting your server with more services than it can keep record, it is totally 👍 to store your AuthN/AuthZ configs as cluster API objects.
-
-
-Can I use Authorino for rate limiting?
-
You can, but you shouldn't. Check out instead Limitador, for simple and efficient global rate limiting. Combine it with Authorino and Authorino's support for Envoy Dynamic Metadata for authenticated rate limiting.
If you are interested in contributing to Authorino, please refer to the Developer's guide for info about the stack and requirements, workflow, policies and Code of Conduct.
-
Join us on the #kuadrant channel in the Kubernetes Slack workspace, for live discussions about the roadmap and more.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
\ No newline at end of file
diff --git a/authorino/install/crd/authorino.kuadrant.io_authconfigs.yaml b/authorino/install/crd/authorino.kuadrant.io_authconfigs.yaml
deleted file mode 100644
index 800f877a..00000000
--- a/authorino/install/crd/authorino.kuadrant.io_authconfigs.yaml
+++ /dev/null
@@ -1,4918 +0,0 @@
----
-apiVersion: apiextensions.k8s.io/v1
-kind: CustomResourceDefinition
-metadata:
- annotations:
- controller-gen.kubebuilder.io/version: v0.9.0
- creationTimestamp: null
- name: authconfigs.authorino.kuadrant.io
-spec:
- group: authorino.kuadrant.io
- names:
- kind: AuthConfig
- listKind: AuthConfigList
- plural: authconfigs
- singular: authconfig
- scope: Namespaced
- versions:
- - additionalPrinterColumns:
- - description: Ready for all hosts
- jsonPath: .status.summary.ready
- name: Ready
- type: string
- - description: Number of hosts ready
- jsonPath: .status.summary.numHostsReady
- name: Hosts
- type: string
- - description: Number of trusted identity sources
- jsonPath: .status.summary.numIdentitySources
- name: Authentication
- priority: 2
- type: integer
- - description: Number of external metadata sources
- jsonPath: .status.summary.numMetadataSources
- name: Metadata
- priority: 2
- type: integer
- - description: Number of authorization policies
- jsonPath: .status.summary.numAuthorizationPolicies
- name: Authorization
- priority: 2
- type: integer
- - description: Number of items added to the authorization response
- jsonPath: .status.summary.numResponseItems
- name: Response
- priority: 2
- type: integer
- - description: Whether issuing Festival Wristbands
- jsonPath: .status.summary.festivalWristbandEnabled
- name: Wristband
- priority: 2
- type: boolean
- name: v1beta1
- schema:
- openAPIV3Schema:
- description: AuthConfig is the schema for Authorino's AuthConfig API
- properties:
- apiVersion:
- description: 'APIVersion defines the versioned schema of this representation
- of an object. Servers should convert recognized schemas to the latest
- internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
- type: string
- kind:
- description: 'Kind is a string value representing the REST resource this
- object represents. Servers may infer this from the endpoint the client
- submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
- type: string
- metadata:
- type: object
- spec:
- description: Specifies the desired state of the AuthConfig resource, i.e.
- the authencation/authorization scheme to be applied to protect the matching
- service hosts.
- properties:
- authorization:
- description: Authorization is the list of authorization policies.
- All policies in this list MUST evaluate to "true" for a request
- be successful in the authorization phase.
- items:
- description: 'Authorization policy to be enforced. Apart from "name",
- one of the following parameters is required and only one of the
- following parameters is allowed: "opa", "json" or "kubernetes".'
- properties:
- authzed:
- description: Authzed authorization
- properties:
- endpoint:
- description: Endpoint of the Authzed service.
- type: string
- insecure:
- description: Insecure HTTP connection (i.e. disables TLS
- verification)
- type: boolean
- permission:
- description: The name of the permission (or relation) on
- which to execute the check.
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from the
- authorization JSON. It can be any path pattern
- to fetch from the authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers are
- available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and @strip.'
- type: string
- type: object
- type: object
- resource:
- description: The resource on which to check the permission
- or relation.
- properties:
- kind:
- description: StaticOrDynamicValue is either a constant
- static string value or a config for fetching a value
- from a dynamic source (e.g. a path pattern of authorization
- JSON)
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from
- the authorization JSON. It can be any path
- pattern to fetch from the authorization JSON
- (e.g. ''context.request.http.host'') or a
- string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers
- are available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and
- @strip.'
- type: string
- type: object
- type: object
- name:
- description: StaticOrDynamicValue is either a constant
- static string value or a config for fetching a value
- from a dynamic source (e.g. a path pattern of authorization
- JSON)
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from
- the authorization JSON. It can be any path
- pattern to fetch from the authorization JSON
- (e.g. ''context.request.http.host'') or a
- string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers
- are available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and
- @strip.'
- type: string
- type: object
- type: object
- type: object
- sharedSecretRef:
- description: Reference to a Secret key whose value will
- be used by Authorino to authenticate with the Authzed
- service.
- properties:
- key:
- description: The key of the secret to select from. Must
- be a valid secret key.
- type: string
- name:
- description: The name of the secret in the Authorino's
- namespace to select from.
- type: string
- required:
- - key
- - name
- type: object
- subject:
- description: The subject that will be checked for the permission
- or relation.
- properties:
- kind:
- description: StaticOrDynamicValue is either a constant
- static string value or a config for fetching a value
- from a dynamic source (e.g. a path pattern of authorization
- JSON)
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from
- the authorization JSON. It can be any path
- pattern to fetch from the authorization JSON
- (e.g. ''context.request.http.host'') or a
- string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers
- are available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and
- @strip.'
- type: string
- type: object
- type: object
- name:
- description: StaticOrDynamicValue is either a constant
- static string value or a config for fetching a value
- from a dynamic source (e.g. a path pattern of authorization
- JSON)
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from
- the authorization JSON. It can be any path
- pattern to fetch from the authorization JSON
- (e.g. ''context.request.http.host'') or a
- string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers
- are available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and
- @strip.'
- type: string
- type: object
- type: object
- type: object
- required:
- - endpoint
- type: object
- cache:
- description: Caching options for the policy evaluation results
- when enforcing this config. Omit it to avoid caching policy
- evaluation results for this config.
- properties:
- key:
- description: Key used to store the entry in the cache. Cache
- entries from different metadata configs are stored and
- managed separately regardless of the key.
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from the
- authorization JSON. It can be any path pattern
- to fetch from the authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers are
- available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and @strip.'
- type: string
- type: object
- type: object
- ttl:
- default: 60
- description: Duration (in seconds) of the external data
- in the cache before pulled again from the source.
- type: integer
- required:
- - key
- type: object
- json:
- description: JSON pattern matching authorization policy.
- properties:
- rules:
- description: The rules that must all evaluate to "true"
- for the request to be authorized.
- items:
- properties:
- all:
- description: A list of pattern expressions to be evaluated
- as a logical AND.
- items:
- type: object
- x-kubernetes-preserve-unknown-fields: true
- type: array
- any:
- description: A list of pattern expressions to be evaluated
- as a logical OR.
- items:
- type: object
- x-kubernetes-preserve-unknown-fields: true
- type: array
- operator:
- description: 'The binary operator to be applied to
- the content fetched from the authorization JSON,
- for comparison with "value". Possible values are:
- "eq" (equal to), "neq" (not equal to), "incl" (includes;
- for arrays), "excl" (excludes; for arrays), "matches"
- (regex)'
- enum:
- - eq
- - neq
- - incl
- - excl
- - matches
- type: string
- patternRef:
- description: Name of a named pattern
- type: string
- selector:
- description: Any pattern supported by https://pkg.go.dev/github.com/tidwall/gjson.
- The value is used to fetch content from the input
- authorization JSON built by Authorino along the
- identity and metadata phases.
- type: string
- value:
- description: The value of reference for the comparison
- with the content fetched from the authorization
- JSON. If used with the "matches" operator, the value
- must compile to a valid Golang regex.
- type: string
- type: object
- type: array
- required:
- - rules
- type: object
- kubernetes:
- description: Kubernetes authorization policy based on `SubjectAccessReview`
- Path and Verb are inferred from the request.
- properties:
- groups:
- description: Groups to test for.
- items:
- type: string
- type: array
- resourceAttributes:
- description: Use ResourceAttributes for checking permissions
- on Kubernetes resources If omitted, it performs a non-resource
- `SubjectAccessReview`, with verb and path inferred from
- the request.
- properties:
- group:
- description: StaticOrDynamicValue is either a constant
- static string value or a config for fetching a value
- from a dynamic source (e.g. a path pattern of authorization
- JSON)
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from
- the authorization JSON. It can be any path
- pattern to fetch from the authorization JSON
- (e.g. ''context.request.http.host'') or a
- string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers
- are available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and
- @strip.'
- type: string
- type: object
- type: object
- name:
- description: StaticOrDynamicValue is either a constant
- static string value or a config for fetching a value
- from a dynamic source (e.g. a path pattern of authorization
- JSON)
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from
- the authorization JSON. It can be any path
- pattern to fetch from the authorization JSON
- (e.g. ''context.request.http.host'') or a
- string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers
- are available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and
- @strip.'
- type: string
- type: object
- type: object
- namespace:
- description: StaticOrDynamicValue is either a constant
- static string value or a config for fetching a value
- from a dynamic source (e.g. a path pattern of authorization
- JSON)
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from
- the authorization JSON. It can be any path
- pattern to fetch from the authorization JSON
- (e.g. ''context.request.http.host'') or a
- string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers
- are available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and
- @strip.'
- type: string
- type: object
- type: object
- resource:
- description: StaticOrDynamicValue is either a constant
- static string value or a config for fetching a value
- from a dynamic source (e.g. a path pattern of authorization
- JSON)
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from
- the authorization JSON. It can be any path
- pattern to fetch from the authorization JSON
- (e.g. ''context.request.http.host'') or a
- string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers
- are available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and
- @strip.'
- type: string
- type: object
- type: object
- subresource:
- description: StaticOrDynamicValue is either a constant
- static string value or a config for fetching a value
- from a dynamic source (e.g. a path pattern of authorization
- JSON)
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from
- the authorization JSON. It can be any path
- pattern to fetch from the authorization JSON
- (e.g. ''context.request.http.host'') or a
- string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers
- are available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and
- @strip.'
- type: string
- type: object
- type: object
- verb:
- description: StaticOrDynamicValue is either a constant
- static string value or a config for fetching a value
- from a dynamic source (e.g. a path pattern of authorization
- JSON)
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from
- the authorization JSON. It can be any path
- pattern to fetch from the authorization JSON
- (e.g. ''context.request.http.host'') or a
- string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers
- are available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and
- @strip.'
- type: string
- type: object
- type: object
- type: object
- user:
- description: User to test for. If without "Groups", then
- is it interpreted as "What if User were not a member of
- any groups"
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from the
- authorization JSON. It can be any path pattern
- to fetch from the authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers are
- available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and @strip.'
- type: string
- type: object
- type: object
- required:
- - user
- type: object
- metrics:
- default: false
- description: Whether this authorization config should generate
- individual observability metrics
- type: boolean
- name:
- description: Name of the authorization policy. It can be used
- to refer to the resolved authorization object in other configs.
- type: string
- opa:
- description: Open Policy Agent (OPA) authorization policy.
- properties:
- allValues:
- default: false
- description: Returns the value of all Rego rules in the
- virtual document. Values can be read in subsequent evaluators/phases
- of the Auth Pipeline. Otherwise, only the default `allow`
- rule will be exposed. Returning all Rego rules can affect
- performance of OPA policies during reconciliation (policy
- precompile) and at runtime.
- type: boolean
- externalRegistry:
- description: External registry of OPA policies.
- properties:
- credentials:
- description: Defines where client credentials will be
- passed in the request to the service. If omitted,
- it defaults to client credentials passed in the HTTP
- Authorization header and the "Bearer" prefix expected
- prepended to the secret value.
- properties:
- in:
- default: authorization_header
- description: The location in the request where client
- credentials shall be passed on requests authenticating
- with this identity source/authentication mode.
- enum:
- - authorization_header
- - custom_header
- - query
- - cookie
- type: string
- keySelector:
- description: Used in conjunction with the `in` parameter.
- When used with `authorization_header`, the value
- is the prefix of the client credentials string,
- separated by a white-space, in the HTTP Authorization
- header (e.g. "Bearer", "Basic"). When used with
- `custom_header`, `query` or `cookie`, the value
- is the name of the HTTP header, query string parameter
- or cookie key, respectively.
- type: string
- required:
- - keySelector
- type: object
- endpoint:
- description: Endpoint of the HTTP external registry.
- The endpoint must respond with either plain/text or
- application/json content-type. In the latter case,
- the JSON returned in the body must include a path
- `result.raw`, where the raw Rego policy will be extracted
- from. This complies with the specification of the
- OPA REST API (https://www.openpolicyagent.org/docs/latest/rest-api/#get-a-policy).
- type: string
- sharedSecretRef:
- description: Reference to a Secret key whose value will
- be passed by Authorino in the request. The HTTP service
- can use the shared secret to authenticate the origin
- of the request.
- properties:
- key:
- description: The key of the secret to select from. Must
- be a valid secret key.
- type: string
- name:
- description: The name of the secret in the Authorino's
- namespace to select from.
- type: string
- required:
- - key
- - name
- type: object
- ttl:
- description: Duration (in seconds) of the external data
- in the cache before pulled again from the source.
- type: integer
- type: object
- inlineRego:
- description: Authorization policy as a Rego language document.
- The Rego document must include the "allow" condition,
- set by Authorino to "false" by default (i.e. requests
- are unauthorized unless changed). The Rego document must
- NOT include the "package" declaration in line 1.
- type: string
- type: object
- priority:
- default: 0
- description: Priority group of the config. All configs in the
- same priority group are evaluated concurrently; consecutive
- priority groups are evaluated sequentially.
- type: integer
- when:
- description: Conditions for Authorino to enforce this authorization
- policy. If omitted, the config will be enforced for all requests.
- If present, all conditions must match for the config to be
- enforced; otherwise, the config will be skipped.
- items:
- properties:
- all:
- description: A list of pattern expressions to be evaluated
- as a logical AND.
- items:
- type: object
- x-kubernetes-preserve-unknown-fields: true
- type: array
- any:
- description: A list of pattern expressions to be evaluated
- as a logical OR.
- items:
- type: object
- x-kubernetes-preserve-unknown-fields: true
- type: array
- operator:
- description: 'The binary operator to be applied to the
- content fetched from the authorization JSON, for comparison
- with "value". Possible values are: "eq" (equal to),
- "neq" (not equal to), "incl" (includes; for arrays),
- "excl" (excludes; for arrays), "matches" (regex)'
- enum:
- - eq
- - neq
- - incl
- - excl
- - matches
- type: string
- patternRef:
- description: Name of a named pattern
- type: string
- selector:
- description: Any pattern supported by https://pkg.go.dev/github.com/tidwall/gjson.
- The value is used to fetch content from the input authorization
- JSON built by Authorino along the identity and metadata
- phases.
- type: string
- value:
- description: The value of reference for the comparison
- with the content fetched from the authorization JSON.
- If used with the "matches" operator, the value must
- compile to a valid Golang regex.
- type: string
- type: object
- type: array
- required:
- - name
- type: object
- type: array
- callbacks:
- description: List of callback configs. Authorino sends callbacks to
- specified endpoints at the end of the auth pipeline.
- items:
- description: Endpoints to callback at the end of each auth pipeline.
- properties:
- http:
- description: Generic HTTP interface to obtain authorization
- metadata from a HTTP service.
- properties:
- body:
- description: Raw body of the HTTP request. Supersedes 'bodyParameters';
- use either one or the other. Use it with method=POST;
- for GET requests, set parameters as query string in the
- 'endpoint' (placeholders can be used).
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from the
- authorization JSON. It can be any path pattern
- to fetch from the authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers are
- available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and @strip.'
- type: string
- type: object
- type: object
- bodyParameters:
- description: Custom parameters to encode in the body of
- the HTTP request. Superseded by 'body'; use either one
- or the other. Use it with method=POST; for GET requests,
- set parameters as query string in the 'endpoint' (placeholders
- can be used).
- items:
- properties:
- name:
- description: The name of the JSON property
- type: string
- value:
- description: Static value of the JSON property
- x-kubernetes-preserve-unknown-fields: true
- valueFrom:
- description: Dynamic value of the JSON property
- properties:
- authJSON:
- description: 'Selector to fetch a value from the
- authorization JSON. It can be any path pattern
- to fetch from the authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers
- are available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and
- @strip.'
- type: string
- type: object
- required:
- - name
- type: object
- type: array
- contentType:
- default: application/x-www-form-urlencoded
- description: Content-Type of the request body. Shapes how
- 'bodyParameters' are encoded. Use it with method=POST;
- for GET requests, Content-Type is automatically set to
- 'text/plain'.
- enum:
- - application/x-www-form-urlencoded
- - application/json
- type: string
- credentials:
- description: Defines where client credentials will be passed
- in the request to the service. If omitted, it defaults
- to client credentials passed in the HTTP Authorization
- header and the "Bearer" prefix expected prepended to the
- secret value.
- properties:
- in:
- default: authorization_header
- description: The location in the request where client
- credentials shall be passed on requests authenticating
- with this identity source/authentication mode.
- enum:
- - authorization_header
- - custom_header
- - query
- - cookie
- type: string
- keySelector:
- description: Used in conjunction with the `in` parameter.
- When used with `authorization_header`, the value is
- the prefix of the client credentials string, separated
- by a white-space, in the HTTP Authorization header
- (e.g. "Bearer", "Basic"). When used with `custom_header`,
- `query` or `cookie`, the value is the name of the
- HTTP header, query string parameter or cookie key,
- respectively.
- type: string
- required:
- - keySelector
- type: object
- endpoint:
- description: Endpoint of the HTTP service. The endpoint
- accepts variable placeholders in the format "{selector}",
- where "selector" is any pattern supported by https://pkg.go.dev/github.com/tidwall/gjson
- and selects value from the authorization JSON. E.g. https://ext-auth-server.io/metadata?p={context.request.http.path}
- type: string
- headers:
- description: Custom headers in the HTTP request.
- items:
- properties:
- name:
- description: The name of the JSON property
- type: string
- value:
- description: Static value of the JSON property
- x-kubernetes-preserve-unknown-fields: true
- valueFrom:
- description: Dynamic value of the JSON property
- properties:
- authJSON:
- description: 'Selector to fetch a value from the
- authorization JSON. It can be any path pattern
- to fetch from the authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers
- are available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and
- @strip.'
- type: string
- type: object
- required:
- - name
- type: object
- type: array
- method:
- default: GET
- description: 'HTTP verb used in the request to the service.
- Accepted values: GET (default), POST. When the request
- method is POST, the authorization JSON is passed in the
- body of the request.'
- enum:
- - GET
- - POST
- type: string
- oauth2:
- description: Authentication with the HTTP service by OAuth2
- Client Credentials grant.
- properties:
- cache:
- default: true
- description: Caches and reuses the token until expired.
- Set it to false to force fetch the token at every
- authorization request regardless of expiration.
- type: boolean
- clientId:
- description: OAuth2 Client ID.
- type: string
- clientSecretRef:
- description: Reference to a Kubernetes Secret key that
- stores that OAuth2 Client Secret.
- properties:
- key:
- description: The key of the secret to select from. Must
- be a valid secret key.
- type: string
- name:
- description: The name of the secret in the Authorino's
- namespace to select from.
- type: string
- required:
- - key
- - name
- type: object
- extraParams:
- additionalProperties:
- type: string
- description: Optional extra parameters for the requests
- to the token URL.
- type: object
- scopes:
- description: Optional scopes for the client credentials
- grant, if supported by he OAuth2 server.
- items:
- type: string
- type: array
- tokenUrl:
- description: Token endpoint URL of the OAuth2 resource
- server.
- type: string
- required:
- - clientId
- - clientSecretRef
- - tokenUrl
- type: object
- sharedSecretRef:
- description: Reference to a Secret key whose value will
- be passed by Authorino in the request. The HTTP service
- can use the shared secret to authenticate the origin of
- the request. Ignored if used together with oauth2.
- properties:
- key:
- description: The key of the secret to select from. Must
- be a valid secret key.
- type: string
- name:
- description: The name of the secret in the Authorino's
- namespace to select from.
- type: string
- required:
- - key
- - name
- type: object
- required:
- - endpoint
- type: object
- metrics:
- default: false
- description: Whether this callback config should generate individual
- observability metrics
- type: boolean
- name:
- description: Name of the callback. It can be used to refer to
- the resolved callback response in other configs.
- type: string
- priority:
- default: 0
- description: Priority group of the config. All configs in the
- same priority group are evaluated concurrently; consecutive
- priority groups are evaluated sequentially.
- type: integer
- when:
- description: Conditions for Authorino to perform this callback.
- If omitted, the callback will be attempted for all requests.
- If present, all conditions must match for the callback to
- be attempted; otherwise, the callback will be skipped.
- items:
- properties:
- all:
- description: A list of pattern expressions to be evaluated
- as a logical AND.
- items:
- type: object
- x-kubernetes-preserve-unknown-fields: true
- type: array
- any:
- description: A list of pattern expressions to be evaluated
- as a logical OR.
- items:
- type: object
- x-kubernetes-preserve-unknown-fields: true
- type: array
- operator:
- description: 'The binary operator to be applied to the
- content fetched from the authorization JSON, for comparison
- with "value". Possible values are: "eq" (equal to),
- "neq" (not equal to), "incl" (includes; for arrays),
- "excl" (excludes; for arrays), "matches" (regex)'
- enum:
- - eq
- - neq
- - incl
- - excl
- - matches
- type: string
- patternRef:
- description: Name of a named pattern
- type: string
- selector:
- description: Any pattern supported by https://pkg.go.dev/github.com/tidwall/gjson.
- The value is used to fetch content from the input authorization
- JSON built by Authorino along the identity and metadata
- phases.
- type: string
- value:
- description: The value of reference for the comparison
- with the content fetched from the authorization JSON.
- If used with the "matches" operator, the value must
- compile to a valid Golang regex.
- type: string
- type: object
- type: array
- required:
- - http
- - name
- type: object
- type: array
- denyWith:
- description: Custom denial response codes, statuses and headers to
- override default 40x's.
- properties:
- unauthenticated:
- description: Denial status customization when the request is unauthenticated.
- properties:
- body:
- description: HTTP response body to override the default denial
- body.
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from the authorization
- JSON. It can be any path pattern to fetch from the
- authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers are
- available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and @strip.'
- type: string
- type: object
- type: object
- code:
- description: HTTP status code to override the default denial
- status code.
- format: int64
- maximum: 599
- minimum: 300
- type: integer
- headers:
- description: HTTP response headers to override the default
- denial headers.
- items:
- properties:
- name:
- description: The name of the JSON property
- type: string
- value:
- description: Static value of the JSON property
- x-kubernetes-preserve-unknown-fields: true
- valueFrom:
- description: Dynamic value of the JSON property
- properties:
- authJSON:
- description: 'Selector to fetch a value from the
- authorization JSON. It can be any path pattern
- to fetch from the authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers are
- available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and @strip.'
- type: string
- type: object
- required:
- - name
- type: object
- type: array
- message:
- description: HTTP message to override the default denial message.
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from the authorization
- JSON. It can be any path pattern to fetch from the
- authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers are
- available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and @strip.'
- type: string
- type: object
- type: object
- type: object
- unauthorized:
- description: Denial status customization when the request is unauthorized.
- properties:
- body:
- description: HTTP response body to override the default denial
- body.
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from the authorization
- JSON. It can be any path pattern to fetch from the
- authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers are
- available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and @strip.'
- type: string
- type: object
- type: object
- code:
- description: HTTP status code to override the default denial
- status code.
- format: int64
- maximum: 599
- minimum: 300
- type: integer
- headers:
- description: HTTP response headers to override the default
- denial headers.
- items:
- properties:
- name:
- description: The name of the JSON property
- type: string
- value:
- description: Static value of the JSON property
- x-kubernetes-preserve-unknown-fields: true
- valueFrom:
- description: Dynamic value of the JSON property
- properties:
- authJSON:
- description: 'Selector to fetch a value from the
- authorization JSON. It can be any path pattern
- to fetch from the authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers are
- available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and @strip.'
- type: string
- type: object
- required:
- - name
- type: object
- type: array
- message:
- description: HTTP message to override the default denial message.
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from the authorization
- JSON. It can be any path pattern to fetch from the
- authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers are
- available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and @strip.'
- type: string
- type: object
- type: object
- type: object
- type: object
- hosts:
- description: The list of public host names of the services protected
- by this authentication/authorization scheme. Authorino uses the
- requested host to lookup for the corresponding authentication/authorization
- configs to enforce.
- items:
- type: string
- type: array
- identity:
- description: List of identity sources/authentication modes. At least
- one config of this list MUST evaluate to a valid identity for a
- request to be successful in the identity verification phase.
- items:
- description: 'The identity source/authentication mode config. Apart
- from "name", one of the following parameters is required and only
- one of the following parameters is allowed: "oicd", "apiKey" or
- "kubernetes".'
- properties:
- anonymous:
- type: object
- apiKey:
- properties:
- allNamespaces:
- default: false
- description: Whether Authorino should look for API key secrets
- in all namespaces or only in the same namespace as the
- AuthConfig. Enabling this option in namespaced Authorino
- instances has no effect.
- type: boolean
- selector:
- description: Label selector used by Authorino to match secrets
- from the cluster storing valid credentials to authenticate
- to this service
- properties:
- matchExpressions:
- description: matchExpressions is a list of label selector
- requirements. The requirements are ANDed.
- items:
- description: A label selector requirement is a selector
- that contains values, a key, and an operator that
- relates the key and values.
- properties:
- key:
- description: key is the label key that the selector
- applies to.
- type: string
- operator:
- description: operator represents a key's relationship
- to a set of values. Valid operators are In,
- NotIn, Exists and DoesNotExist.
- type: string
- values:
- description: values is an array of string values.
- If the operator is In or NotIn, the values array
- must be non-empty. If the operator is Exists
- or DoesNotExist, the values array must be empty.
- This array is replaced during a strategic merge
- patch.
- items:
- type: string
- type: array
- required:
- - key
- - operator
- type: object
- type: array
- matchLabels:
- additionalProperties:
- type: string
- description: matchLabels is a map of {key,value} pairs.
- A single {key,value} in the matchLabels map is equivalent
- to an element of matchExpressions, whose key field
- is "key", the operator is "In", and the values array
- contains only "value". The requirements are ANDed.
- type: object
- type: object
- required:
- - selector
- type: object
- cache:
- description: Caching options for the identity resolved when
- applying this config. Omit it to avoid caching identity objects
- for this config.
- properties:
- key:
- description: Key used to store the entry in the cache. Cache
- entries from different metadata configs are stored and
- managed separately regardless of the key.
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from the
- authorization JSON. It can be any path pattern
- to fetch from the authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers are
- available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and @strip.'
- type: string
- type: object
- type: object
- ttl:
- default: 60
- description: Duration (in seconds) of the external data
- in the cache before pulled again from the source.
- type: integer
- required:
- - key
- type: object
- credentials:
- description: Defines where client credentials are required to
- be passed in the request for this identity source/authentication
- mode. If omitted, it defaults to client credentials passed
- in the HTTP Authorization header and the "Bearer" prefix expected
- prepended to the credentials value (token, API key, etc).
- properties:
- in:
- default: authorization_header
- description: The location in the request where client credentials
- shall be passed on requests authenticating with this identity
- source/authentication mode.
- enum:
- - authorization_header
- - custom_header
- - query
- - cookie
- type: string
- keySelector:
- description: Used in conjunction with the `in` parameter.
- When used with `authorization_header`, the value is the
- prefix of the client credentials string, separated by
- a white-space, in the HTTP Authorization header (e.g.
- "Bearer", "Basic"). When used with `custom_header`, `query`
- or `cookie`, the value is the name of the HTTP header,
- query string parameter or cookie key, respectively.
- type: string
- required:
- - keySelector
- type: object
- extendedProperties:
- description: Extends the resolved identity object with additional
- custom properties before appending to the authorization JSON.
- It requires the resolved identity object to always be of the
- JSON type 'object'. Other JSON types (array, string, etc)
- will break.
- items:
- properties:
- name:
- description: The name of the JSON property
- type: string
- overwrite:
- default: false
- description: Whether the value should overwrite the value
- of an existing property with the same name.
- type: boolean
- value:
- description: Static value of the JSON property
- x-kubernetes-preserve-unknown-fields: true
- valueFrom:
- description: Dynamic value of the JSON property
- properties:
- authJSON:
- description: 'Selector to fetch a value from the authorization
- JSON. It can be any path pattern to fetch from the
- authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers are
- available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and @strip.'
- type: string
- type: object
- required:
- - name
- type: object
- type: array
- kubernetes:
- properties:
- audiences:
- description: The list of audiences (scopes) that must be
- claimed in a Kubernetes authentication token supplied
- in the request, and reviewed by Authorino. If omitted,
- Authorino will review tokens expecting the host name of
- the requested protected service amongst the audiences.
- items:
- type: string
- type: array
- type: object
- metrics:
- default: false
- description: Whether this identity config should generate individual
- observability metrics
- type: boolean
- mtls:
- properties:
- allNamespaces:
- default: false
- description: Whether Authorino should look for TLS secrets
- in all namespaces or only in the same namespace as the
- AuthConfig. Enabling this option in namespaced Authorino
- instances has no effect.
- type: boolean
- selector:
- description: Label selector used by Authorino to match secrets
- from the cluster storing trusted CA certificates to validate
- clients trying to authenticate to this service
- properties:
- matchExpressions:
- description: matchExpressions is a list of label selector
- requirements. The requirements are ANDed.
- items:
- description: A label selector requirement is a selector
- that contains values, a key, and an operator that
- relates the key and values.
- properties:
- key:
- description: key is the label key that the selector
- applies to.
- type: string
- operator:
- description: operator represents a key's relationship
- to a set of values. Valid operators are In,
- NotIn, Exists and DoesNotExist.
- type: string
- values:
- description: values is an array of string values.
- If the operator is In or NotIn, the values array
- must be non-empty. If the operator is Exists
- or DoesNotExist, the values array must be empty.
- This array is replaced during a strategic merge
- patch.
- items:
- type: string
- type: array
- required:
- - key
- - operator
- type: object
- type: array
- matchLabels:
- additionalProperties:
- type: string
- description: matchLabels is a map of {key,value} pairs.
- A single {key,value} in the matchLabels map is equivalent
- to an element of matchExpressions, whose key field
- is "key", the operator is "In", and the values array
- contains only "value". The requirements are ANDed.
- type: object
- type: object
- required:
- - selector
- type: object
- name:
- description: The name of this identity source/authentication
- mode. It usually identifies a source of identities or group
- of users/clients of the protected service. It can be used
- to refer to the resolved identity object in other configs.
- type: string
- oauth2:
- properties:
- credentialsRef:
- description: Reference to a Kubernetes secret in the same
- namespace, that stores client credentials to the OAuth2
- server.
- properties:
- name:
- description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
- TODO: Add other useful fields. apiVersion, kind, uid?'
- type: string
- type: object
- tokenIntrospectionUrl:
- description: The full URL of the token introspection endpoint.
- type: string
- tokenTypeHint:
- description: The token type hint for the token introspection.
- If omitted, it defaults to "access_token".
- type: string
- required:
- - credentialsRef
- - tokenIntrospectionUrl
- type: object
- oidc:
- properties:
- endpoint:
- description: Endpoint of the OIDC issuer. Authorino will
- append to this value the well-known path to the OpenID
- Connect discovery endpoint (i.e. "/.well-known/openid-configuration"),
- used to automatically discover the OpenID Connect configuration,
- whose set of claims is expected to include (among others)
- the "jkws_uri" claim. The value must coincide with the
- value of the "iss" (issuer) claim of the discovered OpenID
- Connect configuration.
- type: string
- ttl:
- description: Decides how long to wait before refreshing
- the OIDC configuration (in seconds).
- type: integer
- required:
- - endpoint
- type: object
- plain:
- properties:
- authJSON:
- description: 'Selector to fetch a value from the authorization
- JSON. It can be any path pattern to fetch from the authorization
- JSON (e.g. ''context.request.http.host'') or a string
- template with variable placeholders that resolve to patterns
- (e.g. "Hello, {auth.identity.name}!"). Any patterns supported
- by https://pkg.go.dev/github.com/tidwall/gjson can be
- used. The following string modifiers are available: @extract:{sep:"
- ",pos:0}, @replace{old:"",new:""}, @case:upper|lower,
- @base64:encode|decode and @strip.'
- type: string
- type: object
- priority:
- default: 0
- description: Priority group of the config. All configs in the
- same priority group are evaluated concurrently; consecutive
- priority groups are evaluated sequentially.
- type: integer
- when:
- description: Conditions for Authorino to enforce this identity
- config. If omitted, the config will be enforced for all requests.
- If present, all conditions must match for the config to be
- enforced; otherwise, the config will be skipped.
- items:
- properties:
- all:
- description: A list of pattern expressions to be evaluated
- as a logical AND.
- items:
- type: object
- x-kubernetes-preserve-unknown-fields: true
- type: array
- any:
- description: A list of pattern expressions to be evaluated
- as a logical OR.
- items:
- type: object
- x-kubernetes-preserve-unknown-fields: true
- type: array
- operator:
- description: 'The binary operator to be applied to the
- content fetched from the authorization JSON, for comparison
- with "value". Possible values are: "eq" (equal to),
- "neq" (not equal to), "incl" (includes; for arrays),
- "excl" (excludes; for arrays), "matches" (regex)'
- enum:
- - eq
- - neq
- - incl
- - excl
- - matches
- type: string
- patternRef:
- description: Name of a named pattern
- type: string
- selector:
- description: Any pattern supported by https://pkg.go.dev/github.com/tidwall/gjson.
- The value is used to fetch content from the input authorization
- JSON built by Authorino along the identity and metadata
- phases.
- type: string
- value:
- description: The value of reference for the comparison
- with the content fetched from the authorization JSON.
- If used with the "matches" operator, the value must
- compile to a valid Golang regex.
- type: string
- type: object
- type: array
- required:
- - name
- type: object
- type: array
- metadata:
- description: List of metadata source configs. Authorino fetches JSON
- content from sources on this list on every request.
- items:
- description: 'The metadata config. Apart from "name", one of the
- following parameters is required and only one of the following
- parameters is allowed: "http", userInfo" or "uma".'
- properties:
- cache:
- description: Caching options for the external metadata fetched
- when applying this config. Omit it to avoid caching metadata
- from this source.
- properties:
- key:
- description: Key used to store the entry in the cache. Cache
- entries from different metadata configs are stored and
- managed separately regardless of the key.
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from the
- authorization JSON. It can be any path pattern
- to fetch from the authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers are
- available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and @strip.'
- type: string
- type: object
- type: object
- ttl:
- default: 60
- description: Duration (in seconds) of the external data
- in the cache before pulled again from the source.
- type: integer
- required:
- - key
- type: object
- http:
- description: Generic HTTP interface to obtain authorization
- metadata from a HTTP service.
- properties:
- body:
- description: Raw body of the HTTP request. Supersedes 'bodyParameters';
- use either one or the other. Use it with method=POST;
- for GET requests, set parameters as query string in the
- 'endpoint' (placeholders can be used).
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from the
- authorization JSON. It can be any path pattern
- to fetch from the authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers are
- available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and @strip.'
- type: string
- type: object
- type: object
- bodyParameters:
- description: Custom parameters to encode in the body of
- the HTTP request. Superseded by 'body'; use either one
- or the other. Use it with method=POST; for GET requests,
- set parameters as query string in the 'endpoint' (placeholders
- can be used).
- items:
- properties:
- name:
- description: The name of the JSON property
- type: string
- value:
- description: Static value of the JSON property
- x-kubernetes-preserve-unknown-fields: true
- valueFrom:
- description: Dynamic value of the JSON property
- properties:
- authJSON:
- description: 'Selector to fetch a value from the
- authorization JSON. It can be any path pattern
- to fetch from the authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers
- are available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and
- @strip.'
- type: string
- type: object
- required:
- - name
- type: object
- type: array
- contentType:
- default: application/x-www-form-urlencoded
- description: Content-Type of the request body. Shapes how
- 'bodyParameters' are encoded. Use it with method=POST;
- for GET requests, Content-Type is automatically set to
- 'text/plain'.
- enum:
- - application/x-www-form-urlencoded
- - application/json
- type: string
- credentials:
- description: Defines where client credentials will be passed
- in the request to the service. If omitted, it defaults
- to client credentials passed in the HTTP Authorization
- header and the "Bearer" prefix expected prepended to the
- secret value.
- properties:
- in:
- default: authorization_header
- description: The location in the request where client
- credentials shall be passed on requests authenticating
- with this identity source/authentication mode.
- enum:
- - authorization_header
- - custom_header
- - query
- - cookie
- type: string
- keySelector:
- description: Used in conjunction with the `in` parameter.
- When used with `authorization_header`, the value is
- the prefix of the client credentials string, separated
- by a white-space, in the HTTP Authorization header
- (e.g. "Bearer", "Basic"). When used with `custom_header`,
- `query` or `cookie`, the value is the name of the
- HTTP header, query string parameter or cookie key,
- respectively.
- type: string
- required:
- - keySelector
- type: object
- endpoint:
- description: Endpoint of the HTTP service. The endpoint
- accepts variable placeholders in the format "{selector}",
- where "selector" is any pattern supported by https://pkg.go.dev/github.com/tidwall/gjson
- and selects value from the authorization JSON. E.g. https://ext-auth-server.io/metadata?p={context.request.http.path}
- type: string
- headers:
- description: Custom headers in the HTTP request.
- items:
- properties:
- name:
- description: The name of the JSON property
- type: string
- value:
- description: Static value of the JSON property
- x-kubernetes-preserve-unknown-fields: true
- valueFrom:
- description: Dynamic value of the JSON property
- properties:
- authJSON:
- description: 'Selector to fetch a value from the
- authorization JSON. It can be any path pattern
- to fetch from the authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers
- are available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and
- @strip.'
- type: string
- type: object
- required:
- - name
- type: object
- type: array
- method:
- default: GET
- description: 'HTTP verb used in the request to the service.
- Accepted values: GET (default), POST. When the request
- method is POST, the authorization JSON is passed in the
- body of the request.'
- enum:
- - GET
- - POST
- type: string
- oauth2:
- description: Authentication with the HTTP service by OAuth2
- Client Credentials grant.
- properties:
- cache:
- default: true
- description: Caches and reuses the token until expired.
- Set it to false to force fetch the token at every
- authorization request regardless of expiration.
- type: boolean
- clientId:
- description: OAuth2 Client ID.
- type: string
- clientSecretRef:
- description: Reference to a Kubernetes Secret key that
- stores that OAuth2 Client Secret.
- properties:
- key:
- description: The key of the secret to select from. Must
- be a valid secret key.
- type: string
- name:
- description: The name of the secret in the Authorino's
- namespace to select from.
- type: string
- required:
- - key
- - name
- type: object
- extraParams:
- additionalProperties:
- type: string
- description: Optional extra parameters for the requests
- to the token URL.
- type: object
- scopes:
- description: Optional scopes for the client credentials
- grant, if supported by he OAuth2 server.
- items:
- type: string
- type: array
- tokenUrl:
- description: Token endpoint URL of the OAuth2 resource
- server.
- type: string
- required:
- - clientId
- - clientSecretRef
- - tokenUrl
- type: object
- sharedSecretRef:
- description: Reference to a Secret key whose value will
- be passed by Authorino in the request. The HTTP service
- can use the shared secret to authenticate the origin of
- the request. Ignored if used together with oauth2.
- properties:
- key:
- description: The key of the secret to select from. Must
- be a valid secret key.
- type: string
- name:
- description: The name of the secret in the Authorino's
- namespace to select from.
- type: string
- required:
- - key
- - name
- type: object
- required:
- - endpoint
- type: object
- metrics:
- default: false
- description: Whether this metadata config should generate individual
- observability metrics
- type: boolean
- name:
- description: The name of the metadata source. It can be used
- to refer to the resolved metadata object in other configs.
- type: string
- priority:
- default: 0
- description: Priority group of the config. All configs in the
- same priority group are evaluated concurrently; consecutive
- priority groups are evaluated sequentially.
- type: integer
- uma:
- description: User-Managed Access (UMA) source of resource data.
- properties:
- credentialsRef:
- description: Reference to a Kubernetes secret in the same
- namespace, that stores client credentials to the resource
- registration API of the UMA server.
- properties:
- name:
- description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
- TODO: Add other useful fields. apiVersion, kind, uid?'
- type: string
- type: object
- endpoint:
- description: The endpoint of the UMA server. The value must
- coincide with the "issuer" claim of the UMA config discovered
- from the well-known uma configuration endpoint.
- type: string
- required:
- - credentialsRef
- - endpoint
- type: object
- userInfo:
- description: OpendID Connect UserInfo linked to an OIDC identity
- config of this same spec.
- properties:
- identitySource:
- description: The name of an OIDC identity source included
- in the "identity" section and whose OpenID Connect configuration
- discovered includes the OIDC "userinfo_endpoint" claim.
- type: string
- required:
- - identitySource
- type: object
- when:
- description: Conditions for Authorino to apply this metadata
- config. If omitted, the config will be applied for all requests.
- If present, all conditions must match for the config to be
- applied; otherwise, the config will be skipped.
- items:
- properties:
- all:
- description: A list of pattern expressions to be evaluated
- as a logical AND.
- items:
- type: object
- x-kubernetes-preserve-unknown-fields: true
- type: array
- any:
- description: A list of pattern expressions to be evaluated
- as a logical OR.
- items:
- type: object
- x-kubernetes-preserve-unknown-fields: true
- type: array
- operator:
- description: 'The binary operator to be applied to the
- content fetched from the authorization JSON, for comparison
- with "value". Possible values are: "eq" (equal to),
- "neq" (not equal to), "incl" (includes; for arrays),
- "excl" (excludes; for arrays), "matches" (regex)'
- enum:
- - eq
- - neq
- - incl
- - excl
- - matches
- type: string
- patternRef:
- description: Name of a named pattern
- type: string
- selector:
- description: Any pattern supported by https://pkg.go.dev/github.com/tidwall/gjson.
- The value is used to fetch content from the input authorization
- JSON built by Authorino along the identity and metadata
- phases.
- type: string
- value:
- description: The value of reference for the comparison
- with the content fetched from the authorization JSON.
- If used with the "matches" operator, the value must
- compile to a valid Golang regex.
- type: string
- type: object
- type: array
- required:
- - name
- type: object
- type: array
- patterns:
- additionalProperties:
- items:
- properties:
- operator:
- description: 'The binary operator to be applied to the content
- fetched from the authorization JSON, for comparison with
- "value". Possible values are: "eq" (equal to), "neq" (not
- equal to), "incl" (includes; for arrays), "excl" (excludes;
- for arrays), "matches" (regex)'
- enum:
- - eq
- - neq
- - incl
- - excl
- - matches
- type: string
- selector:
- description: Any pattern supported by https://pkg.go.dev/github.com/tidwall/gjson.
- The value is used to fetch content from the input authorization
- JSON built by Authorino along the identity and metadata
- phases.
- type: string
- value:
- description: The value of reference for the comparison with
- the content fetched from the authorization JSON. If used
- with the "matches" operator, the value must compile to a
- valid Golang regex.
- type: string
- type: object
- type: array
- description: Named sets of JSON patterns that can be referred in `when`
- conditionals and in JSON-pattern matching policy rules.
- type: object
- response:
- description: List of response configs. Authorino gathers data from
- the auth pipeline to build custom responses for the client.
- items:
- description: 'Dynamic response to return to the client. Apart from
- "name", one of the following parameters is required and only one
- of the following parameters is allowed: "wristband" or "json".'
- properties:
- cache:
- description: Caching options for dynamic responses built when
- applying this config. Omit it to avoid caching dynamic responses
- for this config.
- properties:
- key:
- description: Key used to store the entry in the cache. Cache
- entries from different metadata configs are stored and
- managed separately regardless of the key.
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from the
- authorization JSON. It can be any path pattern
- to fetch from the authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers are
- available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and @strip.'
- type: string
- type: object
- type: object
- ttl:
- default: 60
- description: Duration (in seconds) of the external data
- in the cache before pulled again from the source.
- type: integer
- required:
- - key
- type: object
- json:
- properties:
- properties:
- description: List of JSON property-value pairs to be added
- to the dynamic response.
- items:
- properties:
- name:
- description: The name of the JSON property
- type: string
- value:
- description: Static value of the JSON property
- x-kubernetes-preserve-unknown-fields: true
- valueFrom:
- description: Dynamic value of the JSON property
- properties:
- authJSON:
- description: 'Selector to fetch a value from the
- authorization JSON. It can be any path pattern
- to fetch from the authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers
- are available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and
- @strip.'
- type: string
- type: object
- required:
- - name
- type: object
- type: array
- required:
- - properties
- type: object
- metrics:
- default: false
- description: Whether this response config should generate individual
- observability metrics
- type: boolean
- name:
- description: Name of the custom response. It can be used to
- refer to the resolved response object in other configs.
- type: string
- plain:
- description: StaticOrDynamicValue is either a constant static
- string value or a config for fetching a value from a dynamic
- source (e.g. a path pattern of authorization JSON)
- properties:
- value:
- description: Static value
- type: string
- valueFrom:
- description: Dynamic value
- properties:
- authJSON:
- description: 'Selector to fetch a value from the authorization
- JSON. It can be any path pattern to fetch from the
- authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders that
- resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers are available:
- @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and @strip.'
- type: string
- type: object
- type: object
- priority:
- default: 0
- description: Priority group of the config. All configs in the
- same priority group are evaluated concurrently; consecutive
- priority groups are evaluated sequentially.
- type: integer
- when:
- description: Conditions for Authorino to enforce this custom
- response config. If omitted, the config will be enforced for
- all requests. If present, all conditions must match for the
- config to be enforced; otherwise, the config will be skipped.
- items:
- properties:
- all:
- description: A list of pattern expressions to be evaluated
- as a logical AND.
- items:
- type: object
- x-kubernetes-preserve-unknown-fields: true
- type: array
- any:
- description: A list of pattern expressions to be evaluated
- as a logical OR.
- items:
- type: object
- x-kubernetes-preserve-unknown-fields: true
- type: array
- operator:
- description: 'The binary operator to be applied to the
- content fetched from the authorization JSON, for comparison
- with "value". Possible values are: "eq" (equal to),
- "neq" (not equal to), "incl" (includes; for arrays),
- "excl" (excludes; for arrays), "matches" (regex)'
- enum:
- - eq
- - neq
- - incl
- - excl
- - matches
- type: string
- patternRef:
- description: Name of a named pattern
- type: string
- selector:
- description: Any pattern supported by https://pkg.go.dev/github.com/tidwall/gjson.
- The value is used to fetch content from the input authorization
- JSON built by Authorino along the identity and metadata
- phases.
- type: string
- value:
- description: The value of reference for the comparison
- with the content fetched from the authorization JSON.
- If used with the "matches" operator, the value must
- compile to a valid Golang regex.
- type: string
- type: object
- type: array
- wrapper:
- default: httpHeader
- description: How Authorino wraps the response. Use "httpHeader"
- (default) to wrap the response in an HTTP header; or "envoyDynamicMetadata"
- to wrap the response as Envoy Dynamic Metadata
- enum:
- - httpHeader
- - envoyDynamicMetadata
- type: string
- wrapperKey:
- description: The name of key used in the wrapped response (name
- of the HTTP header or property of the Envoy Dynamic Metadata
- JSON). If omitted, it will be set to the name of the configuration.
- type: string
- wristband:
- properties:
- customClaims:
- description: Any claims to be added to the wristband token
- apart from the standard JWT claims (iss, iat, exp) added
- by default.
- items:
- properties:
- name:
- description: The name of the JSON property
- type: string
- value:
- description: Static value of the JSON property
- x-kubernetes-preserve-unknown-fields: true
- valueFrom:
- description: Dynamic value of the JSON property
- properties:
- authJSON:
- description: 'Selector to fetch a value from the
- authorization JSON. It can be any path pattern
- to fetch from the authorization JSON (e.g. ''context.request.http.host'')
- or a string template with variable placeholders
- that resolve to patterns (e.g. "Hello, {auth.identity.name}!").
- Any patterns supported by https://pkg.go.dev/github.com/tidwall/gjson
- can be used. The following string modifiers
- are available: @extract:{sep:" ",pos:0}, @replace{old:"",new:""},
- @case:upper|lower, @base64:encode|decode and
- @strip.'
- type: string
- type: object
- required:
- - name
- type: object
- type: array
- issuer:
- description: 'The endpoint to the Authorino service that
- issues the wristband (format: ://:/,
- where = /://:/,
- where = /://:/,
- where = /
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- RELEASE - Kuadrant Documentation
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
We now will constantly reconcile DNS records. The reasoning is that other controllers may override/change records in the DNS provider so there is a need to requeue the DNS Record from time to time even when no local changes are introduced.
There are a few new fields on the DNS Record status:
-* QueuedAt is a time when the DNS record was received for the reconciliation
-* QueuedFor is a time when we expect a DNS record to be reconciled again
-* ValidFor indicates the duration since the last reconciliation we consider data in the record to be valid
-* WriteCounter represents a number of consecutive write attempts on the same generation of the record. It is being reset to 0 when the generation changes or there are no changes to write.
-
There is an option to override the ValidFor and DefaultRequeueTime with valid-for and requeue-time flags respectively.
-
The DefaultRequeueTime is the duration between successful validation and the next reconciliation to ensure that the record is still up-to-date.
-
The ValidFor is used to determine if we should do a full reconciliation when we get the record. If the record is still valid we will only update finalizers and validate the record itself. It will not perform anything that involves a DNS provider.
Once we enqueue the DNS record, controller will compile a list of changes to the DNS provider and will apply it. After this, the record is enqueued with the validationRequeueTime and the Ready condition will be marked as false with a message Awaiting Validation. When the record is received again and the controller ensures there are no changes needed (the ones applied are present in the DNS Provider) it sets the Ready condition to true and enqueues it with the defaultRequeueTime.
-
At any time when the record is requeued we also set the record.Status.QueuedFor field with a timestamp for when we expect to receive the record again. And on every reconciliation we set the record.Status.QueuedAt to be the time of the reconciliation.
-
Upon deletion, the process will be similar. The controller will determine the changes needed to the DNS provider and will apply them. The record will be requeued with the validationRequeueTime. Once we receive it back and ensure that there are no changes needed for the DNS provider we remove the finalizer from the record.
-
The validationRequeueTime duration is randomized +/- 50%.
If the record is received prematurely - the ValidFor + QueuedAt is more than the current time - we requeue it again for the ValidFor duration.
-
When we encounter an error during the reconciliation we will not requeue the record and will put in an appropriate error message in the log and on the record. In order for it to reconcile again there must be a change to the DNS Record CR.
-
It is possible for a user to mess with the timestamps field or the ValidFor field. Kubernetes will not let setting an invalid value to the timestamp fields. Once the timestamp fields are set manually it will trigger reconciliation since there is a change in the record CR. The only one that could impact the controller is the QueuedAt field and the controller will believe that to be the last time the record was reconciled. As for the ValidFor: since it is a simple string it is possible to set an incorrect value. If we fail to parse it we treat the ValidFor as 0. This means that the controller will believe that the information in the record is expired and will probe the DNS provider for an update. If a valid value is provided controller will obey it. Eventually, the controller will naturally enqueue the record and those values will be overridden.
-
In case the controller fails to retain changes in the DNS Provider: write are successful, but the validation fails again and the WriteCounter reaches the WriteCounterLimit we give up on the reconciliation. The appropriate message will be put under the Ready - false condition as well as in the logs of the controller. The reconciliation will resume once the generation of the DNS Record is changed.
A ManagedZone is a reference to a DNS zone.
-By creating a ManagedZone we are instructing the MGC about a domain or subdomain that can be used as a host by any gateways in the same namespace.
-These gateways can use a subdomain of the ManagedZone.
-
If a gateway attempts to a use a domain as a host, and there is no matching ManagedZone for that host, then that host on that gateway will fail to function.
-
A gateway's host will be matched to any ManagedZone that the host is a subdomain of, i.e. test.api.hcpapps.net will be matched by any ManagedZone (in the same namespace) of: test.api.hcpapps.net, api.hcpapps.net or hcpapps.net.
-
When MGC wants to create the DNS Records for a host, it will create them in the most exactly matching ManagedZone.
-e.g. given the zones hcpapps.net and api.hcpapps.net the DNS Records for the host test.api.hcpapps.net will be created in the api.hcpapps.net zone.
Delegation allows you to give control of a subdomain of a root domain to MGC while the root domain has it's DNS zone elsewhere.
-
In the scenario where a root domain has a zone outside Route53, e.g. external.com, and a ManagedZone for delegated.external.com is required, the following steps can be taken:
-- Create the ManagedZone for delegated.external.com and wait until the status is updated with an array of nameservers (e.g. ns1.hcpapps.net, ns2.hcpapps.net).
-- Copy these nameservers to your root zone for external.com, you can create a NS record for each nameserver against the delegated.external.com record.
-
For example:
-
delegated.external.com. 3600 IN NS ns1.hcpapps.net.
-delegated.external.com. 3600 IN NS ns2.hcpapps.net.
-
-
Now, when MGC creates a DNS record in it's Route53 zone for delegated.external.com, it will be resolved correctly.
To create a ManagedZone, you will first need to create a DNS provider Secret. To create one, see our DNS Provider setup guide, and make note of your provider's secret name.
We have tested using the available policy AmazonRoute53FullAccess however it should also be possible to restrict the credential down to a particular zone. More info can be found in the AWS docs:
It is recommended that you create the secret in the same namespace as your ManagedZones. In the examples above, we've stored these in a namespace called kuadrant-dns-system.
-
Now that we have the credential created we have a DNS provider ready to go and can start using it.
The DNS Operator is a kubernetes based controller responsible for reconciling DNS Record and Managed Zone custom resources. It interfaces with cloud DNS providers such as AWS and Google to bring the DNS zone into the state declared in these CRDs.
-One of the key use cases the DNS operator solves, is allowing complex DNS routing strategies such as Geo and Weighted to be expressed allowing you to leverage DNS as the first layer of traffic management. In order to make these strategies valuable, it also works across multiple clusters allowing you to use a shared domain name balance traffic based on your requirements.
NOTE: You can optionally skip this step but at least one ManagedZone will need to be configured and have valid credentials linked to use the DNS Operator.
You’ll need a Kubernetes cluster to run against. You can use KIND to get a local cluster for testing, or run against a remote cluster.
-Note: Your controller will automatically use the current context in your kubeconfig file (i.e. whatever cluster kubectl cluster-info shows).
The e2e test suite can be executed against any cluster running the DNS Operator with configuration added for any supported provider.
-
make test-e2e TEST_DNS_MANAGED_ZONE_NAME=<My managed zone name> TEST_DNS_ZONE_DOMAIN_NAME=<My domain name> TEST_DNS_NAMESPACE=<My test namesapace> TEST_DNS_PROVIDER=<aws|gcp>
-
-
-
-
-
Environment Variable
-
Description
-
-
-
-
-
TEST_DNS_MANAGED_ZONE_NAME
-
Name of the managed zone relevant for the test domain (TEST_DNS_ZONE_DOMAIN_NAME). If using local-setup Managed zones, one of [dev-mz-aws; dev-mz-gcp]
-
-
-
TEST_DNS_ZONE_DOMAIN_NAME
-
Domain name being used for the test, must match the domain of the managed zone (TEST_DNS_MANAGED_ZONE_NAME)
-
-
-
TEST_DNS_NAMESPACE
-
The namespace to run the test in, must be the same namespace as the TEST_DNS_MANAGED_ZONE_NAME
-
-
-
TEST_DNS_PROVIDER
-
DNS Provider currently being tested, one of [aws; gcp]
Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
http://www.apache.org/licenses/LICENSE-2.0
-
-
Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
Export environment variables with the keys listed below for your desired provider. Fill in your own values as appropriate. Note that you will need to have created a root domain in AWS Route 53 or in GCP Cloud DNS:
In this quick start, we will cover the setup of Kuadrant in multiple local kind clusters.
-This document is intended as a follow on to the single cluster guide.
-It can be used for adding 1 or more clusters to your local setup.
The script will detect if you already have a cluster from the single cluster setup running, and prompt you for a multi cluster setup.
-This will setup an additional kind cluster, install Istio and install Kuadrant.
-You can re-run the script multiple times to add more clusters.
-Each cluster will have a number suffix in the name. For example: kuadrant-local-1, kuadrant-local-2, kuadrant-local-3.
-The original cluster from the single cluster setup will keep its name of just kuadrant-local.
-This will setup a single kind cluster, install Istio and install Kuadrant. Once this completes you should be able to move on to using the various policy apis offered by Kuadrant. To use project sail to install istio, ensure you set the ISTIO_INSTALL_SAIL=true env var before executing the quick start.
-
Targets Gateway API networking resources such as HTTPRoutes and Gateways, using these resources to obtain additional context, i.e., which traffic workload (HTTP attributes, hostnames, user attributes, etc) to enforce auth.
-
Supports targeting subsets (sections) of a network resource to apply the auth rules to.
-
Abstracts the details of the underlying external authorization protocol and configuration resources, that have a much broader remit and surface area.
-
Enables cluster operators to set defaults that govern behavior at the lower levels of the network, until a more specific policy is applied.
Kuadrant's Auth implementation relies on the Envoy's External Authorization protocol. The workflow per request goes:
-1. On incoming request, the gateway checks the matching rules for enforcing the auth rules, as stated in the AuthPolicy custom resources and targeted Gateway API networking objects
-2. If the request matches, the gateway sends one CheckRequest to the external auth service ("Authorino").
-3. The external auth service responds with a CheckResponse back to the gateway with either an OK or DENIED response code.
-
An AuthPolicy and its targeted Gateway API networking resource contain all the statements to configure both the ingress gateway and the external auth service.
Each auth rule can declare specific routeSelectors and when conditions for the rule to apply.
-
The auth scheme (rules), as well as conditions and named patterns can be declared at the top-level level of the spec (with the semantics of defaults) or alternatively within explicit defaults or overrides blocks.
apiVersion:kuadrant.io/v1beta2
-kind:AuthPolicy
-metadata:
-name:my-auth-policy
-spec:
-# Reference to an existing networking resource to attach the policy to. REQUIRED.
-# It can be a Gateway API HTTPRoute or Gateway resource.
-# It can only refer to objects in the same namespace as the AuthPolicy.
-targetRef:
-group:gateway.networking.k8s.io
-kind:HTTPRoute / Gateway
-name:myroute / mygateway
-
-# Selectors of HTTPRouteRules within the targeted HTTPRoute that activate the AuthPolicy.
-# Each element contains a HTTPRouteMatch object that will be used to select HTTPRouteRules that include at least
-# one identical HTTPRouteMatch.
-# The HTTPRouteMatch part does not have to be fully identical, but the what's stated in the selector must be
-# identically stated in the HTTPRouteRule.
-# Do not use it on AuthPolicies that target a Gateway.
-routeSelectors:
--matches:
--path:
-type:PathPrefix
-value:"/admin"
-
-# Additional dynamic conditions to trigger the AuthPolicy.
-# Use it for filtering attributes not supported by HTTPRouteRule or with AuthPolicies that target a Gateway.
-# Check out https://github.com/Kuadrant/architecture/blob/main/rfcs/0002-well-known-attributes.md to learn more
-# about the Well-known Attributes that can be used in this field.
-# Equivalent to if otherwise declared within `defaults`.
-when:[…]
-
-# Sets of common patterns of selector-operator-value triples, to be referred by name in `when` conditions
-# and pattern-matching rules. Often employed to avoid repetition in the policy.
-# Equivalent to if otherwise declared within `defaults`.
-patterns:{…}
-
-# The auth rules to apply to the network traffic routed through the targeted resource.
-# Equivalent to if otherwise declared within `defaults`.
-rules:
-# Authentication rules to enforce.
-# At least one config must evaluate to a valid identity object for the auth request to be successful.
-# If omitted or empty, anonymous access is assumed.
-authentication:
-"my-authn-rule":
-# The authentication method of this rule.
-# One-of: apiKey, jwt, oauth2Introspection, kubernetesTokenReview, x509, plain, anonymous.
-apiKey:{…}
-
-# Where credentials are required to be passed in the request for authentication based on this rule.
-# One-of: authorizationHeader, customHeader, queryString, cookie.
-credentials:
-authorizationHeader:
-prefix:APIKEY
-
-# Rule-level route selectors.
-routeSelectors:[…]
-
-# Rule-level additional conditions.
-when:[…]
-
-# Configs for caching the resolved object returned out of evaluating this auth rule.
-cache:{…}
-
-# Rules for fetching auth metadata from external sources.
-metadata:
-"my-external-source":
-# The method for fetching metadata from the external source.
-# One-of: http: userInfo, uma.
-http:{…}
-
-# Authorization rules to enforce.
-# All policies must allow access for the auth request be successful.
-authorization:
-"my-authz-rule":
-# The authorization method of this rule.
-# One-of: patternMatching, opa, kubernetesSubjectAccessReview, spicedb.
-opa:{…}
-
-# Customizations to the authorization response.
-response:
-# Custom denial status and other HTTP attributes for unauthenticated requests.
-unauthenticated:{…}
-
-# Custom denial status and other HTTP attributes for unauhtorized requests.
-unauthorized:{…}
-
-# Custom response items when access is granted.
-success:
-# Custom response items wrapped as HTTP headers to be injected in the request
-headers:
-"my-custom-header":
-# One-of: plain, json, wristband.
-plain:{…}
-
-# Custom response items wrapped as envoy dynamic metadata.
-dynamicMetadata:
-# One-of: plain, json, wristband.
-"my-custom-dyn-metadata":
-json:{…}
-
-# Rules for post-authorization callback requests to external services.
-# Triggered regardless of the result of the authorization request.
-callbacks:
-"my-webhook":
-http:{…}
-
-# Explicit defaults. Used in policies that target a Gateway object to express default rules to be enforced on
-# routes that lack a more specific policy attached to.
-# Mutually exclusive with `overrides` and with declaring the `rules`, `when` and `patterns` at the top-level of
-# the spec.
-defaults:
-rules:
-authentication:{…}
-metadata:{…}
-authorization:{…}
-response:{…}
-callbacks:{…}
-when:[…]
-patterns:{…}
-
-# Overrides. Used in policies that target a Gateway object to be enforced on all routes linked to the gateway,
-# thus also overriding any more specific policy occasionally attached to any of those routes.
-# Mutually exclusive with `defaults` and with declaring `rules`, `when` and `patterns` at the top-level of
-# the spec.
-overrides:
-rules:
-authentication:{…}
-metadata:{…}
-authorization:{…}
-response:{…}
-callbacks:{…}
-when:[…]
-patterns:{…}
-
-
Check out the API reference for a full specification of the AuthPolicy CRD.
When an AuthPolicy targets a HTTPRoute, the policy is enforced to all traffic routed according to the rules and hostnames specified in the HTTPRoute, across all Gateways referenced in the spec.parentRefs field of the HTTPRoute.
-
The targeted HTTPRoute's rules and/or hostnames to which the policy must be enforced can be filtered to specific subsets, by specifying the routeSelectors field of the AuthPolicy spec.
-
Target a HTTPRoute by setting the spec.targetRef field of the AuthPolicy as follows:
If an AuthPolicy targets a route defined for *.com and another AuthPolicy targets another route for api.com, the Kuadrant control plane will not merge these two AuthPolicies. Rather, it will mimic the behavior of gateway implementation by which the "most specific hostname wins", thus enforcing only the corresponding applicable policies and auth rules.
-
E.g., a request coming for api.com will be protected according to the rules from the AuthPolicy that targets the route for api.com; while a request for other.com will be protected with the rules from the AuthPolicy targeting the route for *.com.
-
Example with 3 AuthPolicies and 3 HTTPRoutes:
-- AuthPolicy A → HTTPRoute A (a.toystore.com)
-- AuthPolicy B → HTTPRoute B (b.toystore.com)
-- AuthPolicy W → HTTPRoute W (*.toystore.com)
-
Expected behavior:
-- Request to a.toystore.com → AuthPolicy A will be enforced
-- Request to b.toystore.com → AuthPolicy B will be enforced
-- Request to other.toystore.com → AuthPolicy W will be enforced
An AuthPolicy that targets a Gateway can declare a block of defaults (spec.defaults) or a block of overrides (spec.overrides). As a standard, gateway policies that do not specify neither defaults nor overrides, act as defaults.
-
When declaring defaults, an AuthPolicy which targets a Gateway will be enforced to all HTTP traffic hitting the gateway, unless a more specific AuthPolicy targeting a matching HTTPRoute exists. Any new HTTPRoute referrencing the gateway as parent will be automatically covered by the default AuthPolicy, as well as changes in the existing HTTPRoutes.
-
Defaults provide cluster operators with the ability to protect the infrastructure against unplanned and malicious network traffic attempt, such as by setting preemptive "deny-all" policies on hostnames and hostname wildcards.
-
Inversely, a gateway policy that specify overrides declares a set of rules to be enforced on all routes attached to the gateway, thus atomically replacing any more specific policy occasionally attached to any of those routes.
-
Target a Gateway HTTPRoute by setting the spec.targetRef field of the AuthPolicy as follows:
Two possible semantics are to be considered here – gateway policy defaults vs gateway policy overrides.
-
Gateway AuthPolicies that declare defaults (or alternatively neither defaults nor overrides) protect all traffic routed through the gateway except where a more specific HTTPRoute AuthPolicy exists, in which case the HTTPRoute AuthPolicy prevails.
-
Example with 4 AuthPolicies, 3 HTTPRoutes and 1 Gateway default (plus 2 HTTPRoute and 2 Gateways without AuthPolicies attached):
-- AuthPolicy A → HTTPRoute A (a.toystore.com) → Gateway G (*.com)
-- AuthPolicy B → HTTPRoute B (b.toystore.com) → Gateway G (*.com)
-- AuthPolicy W → HTTPRoute W (*.toystore.com) → Gateway G (*.com)
-- AuthPolicy G (defaults) → Gateway G (*.com)
-
Expected behavior:
-- Request to a.toystore.com → AuthPolicy A will be enforced
-- Request to b.toystore.com → AuthPolicy B will be enforced
-- Request to other.toystore.com → AuthPolicy W will be enforced
-- Request to other.com (suppose a route exists) → AuthPolicy G will be enforced
-- Request to yet-another.net (suppose a route and gateway exist) → No AuthPolicy will be enforced
-
Gateway AuthPolicies that declare overrides protect all traffic routed through the gateway, regardless of existence of any more specific HTTPRoute AuthPolicy.
-
Example with 4 AuthPolicies, 3 HTTPRoutes and 1 Gateway override (plus 2 HTTPRoute and 2 Gateways without AuthPolicies attached):
-- AuthPolicy A → HTTPRoute A (a.toystore.com) → Gateway G (*.com)
-- AuthPolicy B → HTTPRoute B (b.toystore.com) → Gateway G (*.com)
-- AuthPolicy W → HTTPRoute W (*.toystore.com) → Gateway G (*.com)
-- AuthPolicy G (overrides) → Gateway G (*.com)
-
Expected behavior:
-- Request to a.toystore.com → AuthPolicy G will be enforced
-- Request to b.toystore.com → AuthPolicy G will be enforced
-- Request to other.toystore.com → AuthPolicy G will be enforced
-- Request to other.com (suppose a route exists) → AuthPolicy G will be enforced
-- Request to yet-another.net (suppose a route and gateway exist) → No AuthPolicy will be enforced
Route selectors allow targeting sections of a HTTPRoute, by specifying sets of HTTPRouteMatches and/or hostnames that make the policy controller look up within the HTTPRoute spec for compatible declarations, and select the corresponding HTTPRouteRules and hostnames, to then build conditions that activate the policy or policy rule.
-
Check out Route selectors for a full description, semantics and API reference.
when conditions can be used to scope an AuthPolicy or auth rule within an AuthPolicy (i.e. to filter the traffic to which a policy or policy rule applies) without any coupling to the underlying network topology, i.e. without making direct references to HTTPRouteRules via routeSelectors.
-
Use when conditions to conditionally activate policies and policy rules based on attributes that cannot be expressed in the HTTPRoutes' spec.hostnames and spec.rules.matches fields, or in general in AuthPolicies that target a Gateway.
-
when conditions in an AuthPolicy are compatible with Authorino conditions, thus supporting complex boolean expressions with AND and OR operators, as well as grouping.
-
The selectors within the when conditions of an AuthPolicy are a subset of Kuadrant's Well-known Attributes (RFC 0002). Check out the reference for the full list of supported selectors.
-
Authorino JSON path string modifiers can also be applied to the selectors within the when conditions of an AuthPolicy.
One HTTPRoute can only be targeted by one AuthPolicy.
-
One Gateway can only be targeted by one AuthPolicy.
-
AuthPolicies can only target HTTPRoutes/Gateways defined within the same namespace of the AuthPolicy.
-
2+ AuthPolicies cannot target network resources that define/inherit the same exact hostname.
-
-
Limitation: Multiple network resources with identical hostnames¶
-
Kuadrant currently does not support multiple AuthPolicies simultaneously targeting network resources that declare identical hostnames. This includes multiple HTTPRoutes that specify the same hostnames in the spec.hostnames field, as well as HTTPRoutes that specify a hostname that is identical to a hostname specified in a listener of one of the route's parent gateways or HTTPRoutes that don't specify any hostname at all thus inheriting the hostnames from the parent gateways. In any of these cases, a maximum of one AuthPolicy targeting any of those resources that specify identical hostnames is allowed.
-
Moreover, having multiple resources that declare identical hostnames may lead to unexpected behavior and therefore should be avoided.
-
This limitation is rooted at the underlying components configured by Kuadrant for the implementation of its policies and the lack of information in the data plane regarding the exact route that is honored by the API gateway at each specific request, in cases of conflicting hostnames.
-
To exemplify one way this limitation can impact deployments, consider the following topology:
In the example above, with the policy-1 resource created before policy-2, policy-1 will be enforced on all requests to app.io/foo while policy-2 will be rejected. I.e. app.io/bar will not be secured. In fact, the status conditions of policy-2 shall reflect Enforced=false with message "AuthPolicy has encountered some issues: AuthScheme is not ready yet".
A slightly different way the limitation applies is when two or more routes of a gateway declare the exact same hostname and a gateway policy is defined with expectation to set default rules for the cases not covered by more specific policies. E.g.:
Once again, requests to app.io/foo will be protected under AuthPolicy policy-1, while requests to app.io/bar will not be protected under any policy at all, unlike expected gateway policy policy-2 enforced as default. Both policies will report status condition as Enforced nonetheless.
-
To avoid these problems, use different hostnames in each route.
Under the hood, for each AuthPolicy, Kuadrant creates an Istio AuthorizationPolicy and an Authorino AuthConfig custom resources.
-
Only requests that matches the rules in the Istio AuthorizationPolicy cause an authorization request to be sent to the external authorization service ("Authorino"), i.e., only requests directed to the HTTPRouteRules targeted by the AuthPolicy (directly or indirectly), according to the declared top-level route selectors (if present), or all requests for which a matching HTTPRouteRule exists (otherwise).
-
Authorino looks up for the auth scheme (AuthConfig custom resource) to enforce using the provided hostname of the original request as key. It then checks again if the request matches at least one of the selected HTTPRouteRules, in which case it enforces the auth scheme.
-
- Exception to the rule
-
- Due to limitations imposed by the Istio `AuthorizationPolicy`, there are a few patterns of HTTPRouteRules that cannot be translated to filters for the external authorization request. Therefore, the following patterns used in HTTPRouteMatches of top-level route selectors of an AuthPolicy will not be included in the Istio AuthorizationPolicy rules that trigger the check request with Authorino: `PathMatchRegularExpression`, `HeaderMatchRegularExpression`, and `HTTPQueryParamMatch`.
-
- As a consequence to the above, requests that do not match these rules and otherwise would not be checked with Authorino will result in a request to the external authorization service. Authorino nonetheless will still verify those patterns and ensure the auth scheme is enforced only when it matches a selected HTTPRouteRule. Users of Kuadrant may observe an unnecessary call to the authorization service in those cases where the request is out of the scope of the AuthPolicy and therefore always authorized.
-
-
-
While the Istio AuthorizationPolicy needs to be created in the same namespace as the gateway workload, the Authorino AuthConfig is created in the namespace of the AuthPolicy itself. This allows to simplify references such as to Kubernetes Secrets referred in the AuthPolicy, as well as the RBAC to support the architecture.
You can deploy kuadrant using OLM just running few commands.
-No need to build any image. Kuadrant engineering team provides latest and
-release version tagged images. They are available in
-the Quay.io/Kuadrant image repository.
-
Create kind cluster
-
makekind-create-cluster
-
-
Deploy OLM system
-
makeinstall-olm
-
-
Deploy kuadrant using OLM. The make deploy-catalog target accepts the following variables:
If you want to deploy (using OLM) a custom kuadrant operator, you need to build your own catalog.
-Furthermore, if you want to deploy a custom limitador or authorino operator, you also need
-to build your own catalog. The kuadrant operator bundle includes the authorino or limtador operator
-dependency version, hence using other than latest version requires a custom kuadrant operator
-bundle and a custom catalog including the custom bundle.
Frequently, you may need to build custom kuadrant bundle with the default (latest) Limitador and
-Authorino bundles. These are the example commands to build the manifests, build the bundle image
-and push to the registry.
-
In the example, a new kuadrant operator bundle version 0.8.0 will be created that references
-the kuadrant operator image quay.io/kuadrant/kuadrant-operator:v0.5.0 and latest Limitador and
-Authorino bundles.
Targets Gateway API networking resources Gateways to provide dns management by managing the lifecycle of dns records in external dns providers such as AWS Route53 and Google DNS.
A DNSPolicy and its targeted Gateway API networking resource contain all the statements to configure both the ingress gateway and the external DNS service.
-The needed dns names are gathered from the listener definitions and the IPAdresses | CNAME hosts are gathered from the status block of the gateway resource.
apiVersion:kuadrant.io/v1alpha1
-kind:DNSPolicy
-metadata:
-name:my-dns-policy
-spec:
-# reference to an existing networking resource to attach the policy to
-# it can only be a Gateway API Gateway resource
-# it can only refer to objects in the same namespace as the DNSPolicy
-targetRef:
-group:gateway.networking.k8s.io
-kind:Gateway
-name:mygateway
-
-# (optional) routing strategy to use when creating DNS records, defaults to `loadbalanced`
-# determines what DNS records are created in the DNS provider
-# check out Kuadrant RFC 0005 https://github.com/Kuadrant/architecture/blob/main/rfcs/0005-single-cluster-dnspolicy.md to learn more about the Routing Strategy field
-# One-of: simple, loadbalanced.
-routingStrategy:loadbalanced
-
-# (optional) loadbalancing specification
-# use it for providing the specification of how dns will be configured in order to provide balancing of load across multiple clusters when using the `loadbalanced` routing strategy
-# Primary use of this is for multi cluster deployments
-# check out Kuadrant RFC 0003 https://github.com/Kuadrant/architecture/blob/main/rfcs/0003-dns-policy.md to learn more about the options that can be used in this field
-loadBalancing:
-# (optional) weighted specification
-# use it to control the weight value applied to records
-weighted:
-# use it to change the weight of a record based on labels applied to the target meta resource i.e. Gateway in a single cluster context or ManagedCluster in multi cluster with OCM
-custom:
--weight:200
-selector:
-matchLabels:
-kuadrant.io/lb-attribute-custom-weight:AWS
-# (optional) weight value that will be applied to weighted dns records by default. Integer greater than 0 and no larger than the maximum value accepted by the target dns provider, defaults to `120`
-defaultWeight:100
-# (optional) geo specification
-# use it to control the geo value applied to records
-geo:
-# (optional) default geo to be applied to records
-defaultGeo:IE
-
-# (optional) health check specification
-# health check probes with the following specification will be created for each DNS target
-healthCheck:
-allowInsecureCertificates:true
-endpoint:/
-expectedResponses:
--200
--201
--301
-failureThreshold:5
-port:443
-protocol:https
-
-
Check out the API reference for a full specification of the DNSPolicy CRD.
A DNSPolicy acts against a target Gateway by processing its listeners for hostnames that it can create dns records for.
-In order for it to do this, it must know about dns providers, and what domains these dns providers are currently hosting.
-This is done through the creation of ManagedZones and dns provider secrets containing the credentials for the dns provider account.
-
If for example a Gateway is created with a listener with a hostname of echo.apps.hcpapps.net:
-
The DNSPolicy will create a DNSRecord resource for each listener hostname with a suitable ManagedZone configured. The DNSPolicy resource uses the status of the Gateway to determine what dns records need to be created based on the clusters it has been placed onto.
-
Given the following multi cluster gateway status:
-
DNS Health Checks are a tool provided by some DNS Providers for ensuring the availability and reliability of your DNS Records and only publishing DNS Records that resolve to healthy workloads. Kuadrant offers a powerful feature known as DNSPolicy, which allows you to configure these health checks for all the managed DNS endpoints created as a result of that policy. This guide provides a comprehensive overview of how to set up, utilize, and understand these DNS health checks.
The DNS Record CR will show whether the health check has been created or not in the DNS Provider, and will also show any errors encountered when trying to create or update the health check configuration.
-
To see the status of the executing health check requires logging in to the Route53 console to view the current probe results.
To reconfigure the health checks, update the HealthCheck section of the DNS Policy, this will be reflected into all the health checks created as a result of this policy.
To remove the health checks created in AWS, delete the healthcheck section of the DNS Policy. All health checks will be deleted automatically, if the DNS Policy is deleted.
As Route53 will only perform health checks on an IP address, currently do not create health checks on DNS Policies that target gateways with hostname addresses.
Although we intend to support integrating with the DNS Health checks provided by other DNS Providers in the future, we currently only support AWS Route53.
exportAWS_ACCESS_KEY_ID=xxxxxxx# Key ID from AWS with Route 53 access
-exportAWS_SECRET_ACCESS_KEY=xxxxxxx# Access Key from AWS with Route 53 access
-exportREDIS_URL=redis://user:xxxxxx@some-redis.com:10340# A Redis cluster URL
-
Kuadrant integrates with Istio as a Gateway API provider. Before you can use Kuadrant, you must set up an Istio-based Gateway API provider. For this step, you will use the Sail Operator.
Kuadrant provides a set of sample dashboards that use known metrics exported by Kuadrant and Gateway components to provide insight into different areas of your APIs and Gateways. While not essential, it is best to set up an observability stack. This section provides links to OpenShift and Thanos documentation on configuring monitoring and metrics storage.
-
OpenShift supports a user facing monitoring stack. This can be cofigured and setup this documentation:
There are a set of example dashboards and alerts for observing Kuadrant functionality.
-These dashboards and alerts make use of low level cpu, metrics and network metrics available from the user monitoring stack in Openshift. They also make use of resource state metrics from Gateway API and Kuadrant resources.
-To scrape these additional metrics, you can install a kube-state-metrics instance, with a custom resource config:
The dashboards can be imported into Grafana, if you have it installed in your cluster.
-You'll find an example of how to install Grafana on Openshift here. Once installed, you will need to add your Thanos instance as a data source to Grafana. Alternatively, if you are just using the user workload monitoring stack in your Openshift cluster (and not writing metrics to an external thanos instance), you can set up a data source to the thanos-querier route in the Openshift cluster.
The kuadrant operator outputs 3 levels of log messages: (from lowest to highest level)
-
-
debug
-
info (default)
-
error
-
-
info logging is restricted to high-level information. Actions like creating, deleteing or updating kubernetes resources will be logged with reduced details about the corresponding objects, and without any further detailed logs of the steps in between, except for errors.
-
Only debug logging will include processing details.
-
To configure the desired log level, set the environment variable LOG_LEVEL to one of the supported values listed above. Default log level is info.
-
Apart from log level, the operator can output messages to the logs in 2 different formats:
-
-
production (default): each line is a parseable JSON object with properties {"level":string, "ts":int, "msg":string, "logger":string, extra values...}
-
development: more human-readable outputs, extra stack traces and logging info, plus extra values output as JSON, in the format: <timestamp-iso-8601>\t<log-level>\t<logger>\t<message>\t{extra-values-as-json}
-
-
To configure the desired log mode, set the environment variable LOG_MODE to one of the supported values listed above. Default log level is production.
Explore a variety of starting points for monitoring your Kuadrant installation with our examples folder. These dashboards and alerts are ready-to-use and easily customizable to fit your environment.
UI Method: Use the 'Import' feature in the Grafana UI to upload dashboard JSON files directly.
-
ConfigMap Method: Automate dashboard provisioning by adding files to a ConfigMap, which should be mounted at /etc/grafana/provisioning/dashboards.
-
-
Datasources are configured as template variables, automatically integrating with your existing data sources. Metrics for these dashboards are sourced from Prometheus. For more details on the metrics used, visit the metrics documentation page.
This is a reference page for some of the different metrics used in example
-dashboards and alerts. It is not an exhaustive list. The documentation for each
-component may provide more details on a per-component basis. Some of the metrics
-are sourced from components outside the Kuadrant project, for example, Envoy.
-The value of this reference is showing some of the more widely desired metrics,
-and how to join the metrics from different sources together in a meaningful way.
Resource metrics, like CPU, memory and disk usage, primarily come from the Kubernetes
-metrics components. These include container_cpu_usage_seconds_total, container_memory_working_set_bytes
-and kubelet_volume_stats_used_bytes. A stable list of metrics is maintained in
-the Kubernetes repository. These low-level metrics typically have a set of
-recording rules that
-aggregate values by labels and time ranges.
-For example, node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate or namespace_workload_pod:kube_pod_owner:relabel.
-If you have deployed the kube-prometheus project, you should have the majority of
-these metrics being scraped.
Low-level networking metrics like container_network_receive_bytes_total are also
-available from the Kubernetes metrics components.
-HTTP & GRPC traffic metrics with higher level labels are available from Istio.
-One of the main metrics would be istio_requests_total, which is a counter incremented for every request handled by an Istio proxy.
-Latency metrics are available via the istio_request_duration_milliseconds metric, with buckets for varying response times.
-
Some example dashboards have panels that make use of the request URL path.
-The path is not added as a label to Istio metrics by default, as it has the potential
-to increase metric cardinality, and thus storage requirements.
-If you want to make use of the path in your queries or visualisations, you can enable
-the request path metric via the Telemetry resource in istio:
The kube-state-metrics
-project exposes the state of various kuberenetes resources
-as metrics and labels. For example, the ready status of a Pod is available as
-kube_pod_status_ready, with labels for the pod name and namespace. This can
-be useful for linking lower level container metrics back to a meaningful resource
-in the Kubernetes world.
Metric queries can be as simple as just the name of the metric, or can be complex
-with joining & grouping. A lot of the time it can be useful to tie back low level
-metrics to more meaningful Kubernetes resources. For example, if the memory usage
-is maxed out on a container and that container is constantly being OOMKilled, it
-can be useful to get the Deployment and Namespace of that container for debugging.
-Prometheus query language (or promql) allows vector matching
-or results (sometimes called joining).
-
When using Gateway API and Kuadrant resources like HTTPRoute and RateLimitPolicy,
-the state metrics can be joined to Istio metrics to give a meaningful result set.
-Here's an example that queries the number of requests per second, and includes
-the name of the HTTPRoute that the traffic is for.
The result set here will include a label for the destination service name (i.e.
-the Service in Kubernetes). This label is key to looking up the HTTPRoute this
-traffic belongs to.
-
The 2nd part of the query uses the gatewayapi_httproute_labels metric and the
-label_replace function. The gatewayapi_httproute_labels metric gives a list
-of all httproutes, including any labels on them. The HTTPRoute in this example
-has a label called 'service', set to be the same as the Istio service name.
-This allows us to join the 2 results set.
-However, because the label doesn't match exactly (destination_service_name and service),
-we can replace the label so that it does match. That's what the label_replace
-does.
This guide outlines the steps to enable tracing in Istio and Kuadrant components (Authorino and Limitador), directing traces to a central collector for improved observability and troubleshooting. We'll also explore a typical troubleshooting flow using traces and logs.
Enable tracing in Istio by using the Telemetry API.
-Depending on your method for installing Istio, you will need to configure a tracing extensionProvider in your MeshConfig, Istio or IstioOperator resource as well.
-Here is an example Telemetry and Istio config to sample 100% of requests, if using the Istio Sail Operator.
The Authorino and Limitador components have request tracing capabilities.
-Here is an example configuration to enable and send traces to a central collector.
-Ensure the collector is the same one that Istio is sending traces so that they can be correlated later.
Once the changes are applied, the authorino and limitador components will be redeployed tracing enabled.
-
Note:
-
There are plans to consolidate the tracing configuration to a single location i.e. the Kuadrant CR.
-This will eventually eliminate the need to configure tracing in both the Authorino and Limitador CRs.
-
Important:
-
Currently, trace IDs do not propagate to wasm modules in Istio/Envoy, affecting trace continuity in Limitador.
-This means that requests passed to limitador will not have the relavant 'parent' trace ID in its trace information.
-If however the trace initiation point is outside of Envoy/Istio, the 'parent' trace ID will be available to limitador and included in traces passed to the collector.
-This has an impact on correlating traces from limitador with traces from authorino, the gateway and any other components in the path of requests.
Using a tracing interface like the Jaeger UI or Grafana, you can search for trace information by the trace ID.
-You may get the trace ID from logs, or from a header in a sample request you want to troubleshoot.
-You can also search for recent traces, filtering by the service you want to focus on.
-
Here is an example trace in the Grafana UI showing the total request time from the gateway (Istio), the time to check the curent rate limit count (and update it) in limitador and the time to check auth in Authorino:
-
-
In limitador, it is possible to enable request logging with trace IDs to get more information on requests.
-This requires the log level to be increased to at least debug, so the verbosity must be set to 3 or higher in the Limitador CR. For example:
If you centrally aggregate logs using something like promtail and loki, you can jump between trace information and the relevant logs for that service:
-
-
Using a combination of tracing and logs, you can visualise and troubleshoot reuqest timing issues and drill down to specific services.
-This method becomes even more powerful when combined with metrics and dashboards to get a more complete picture of your users traffic.
Having in place this HTTPRoute - Gateway hierarchy, we are also considering to apply Policy Attachment's defaults/overrides approach to the RateLimitPolicy CRD. But for now, it will only be about targeting the Gateway resource.
-
-
On designing Kuadrant's rate limiting and considering Istio/Envoy's rate limiting offering, we hit two limitations (described here). Therefore, not giving up entirely in existing Envoy's RateLimit Filter, we decided to move on and leverage the Envoy's Wasm Network Filter and implement rate limiting wasm-shim module compliant with the Envoy's Rate Limit Service (RLS). This wasm-shim module accepts a PluginConfig struct object as input configuration object.
A key use case is being able to provide governance over what service providers can and cannot do when exposing a service via a shared ingress gateway. As well as providing certainty that no service is exposed without my ability as a cluster administrator to protect my infrastructure from unplanned load from badly behaving clients etc.
The goal of this document is to define:
-* The schema of this PluginConfig struct.
-* The kuadrant-operator behavior filling the PluginConfig struct having as input the RateLimitPolicy k8s objects
-* The behavior of the wasm-shim having the PluginConfig struct as input.
Kuadrant's rate limit relies on the Rate Limit Service (RLS)
-protocol, hence the gateway generates, based on a set of
-actions,
-a set of descriptors
-(one descriptor is a set of descriptor entries). Those descriptors are send to the external rate limit service provider.
-When multiple descriptors are provided, the external service provider will limit on ALL of them and
-return an OVER_LIMIT response if any of them are over limit.
.spec.rateLimits holds a list of rate limit configurations represented by the object RateLimit.
-Each RateLimit object represents a complete rate limit configuration. It contains three fields:
-
-
-
rules (optional): Rules allow matching hosts and/or methods and/or paths.
-Matching occurs when at least one rule applies against the incoming request.
-If rules are not set, it is equivalent to matching all the requests.
-
-
-
configurations (required): Specifies a set of rate limit configurations that could be applied.
-The rate limit configuration object is the equivalent of the
-config.route.v3.RateLimit envoy object.
-One configuration is, in turn, a list of rate limit actions.
-Each action populates a descriptor entry. A vector of descriptor entries compose a descriptor.
-Each configuration produces, at most, one descriptor.
-Depending on the incoming request, one configuration may or may not produce a rate limit descriptor.
-These rate limiting configuration rules provide flexibility to produce multiple descriptors.
-For example, you may want to define one generic rate limit descriptor and another descriptor
-depending on some header.
-If the header does not exist, the second descriptor is not generated, but traffic keeps being rate
-limited based on the generic descriptor.
limits (optional): configuration of the rate limiting service (Limitador).
-Check out limitador documentation for more information about the fields of each Limit object.
-
-
Note: No namespace/domain defined. Kuadrant operator will figure out.
-
Note: There is no PREAUTH, POSTAUTH stage defined. Ratelimiting filter should be placed after authorization filter to enable authenticated rate limiting. In the future, stage can be implemented.
One HTTPRoute can only be targeted by one rate limit policy.
-
Similarly, one Gateway can only be targeted by one rate limit policy.
-
However, indirectly, one gateway will be affected by multiple rate limit policies.
-It is by design of the Gateway API, one gateway can be referenced by multiple HTTPRoute objects.
-Furthermore, one HTTPRoute can reference multiple gateways.
-
The kuadrant operator will aggregate all the rate
-limit policies that apply for each gateway, including RLP targeting HTTPRoutes and Gateways.
When there are multiple HTTPRoutes with the same hostname, HTTPRoutes are all admitted and
-envoy merge the routing configuration in the same virtualhost. In these cases, the control plane
-has to "merge" the rate limit configuration into a single entry for the wasm filter.
If some RLP targets a route for *.com and other RLP targets another route for api.com,
-the control plane does not do any merging.
-A request coming for api.com will be rate limited with the rules from the RLP targeting
-the route api.com.
-Also, a request coming for other.com will be rate limited with the rules from the RLP targeting
-the route *.com.
The kuadrant operator will add domain attribute of the Envoy's Rate Limit Service (RLS). It will also add the namespace attribute of the Limitador's rate limit config. The operator will ensure that the associated actions and rate limits have a common domain/namespace.
-
The value of this domain/namespace seems to be related to the virtualhost for which rate limit applies.
-
Schema of the WASM filter configuration object: the PluginConfig¶
-
Currently the PluginConfig looks like this:
-
# The filter’s behaviour in case the rate limiting service does not respond back. When it is set to true, Envoy will not allow traffic in case of communication failure between rate limiting service and the proxy.
-failure_mode_deny:true
-ratelimitpolicies:
-default/toystore:# rate limit policy {NAMESPACE/NAME}
-hosts:# HTTPRoute hostnames
--'*.toystore.com'
-rules:# route level actions
--operations:
--paths:
--/admin/toy
-methods:
--POST
--DELETE
-actions:
--generic_key:
-descriptor_value:yes
-descriptor_key:admin
-global_actions:# virtualHost level actions
--generic_key:
-descriptor_value:yes
-descriptor_key:vhaction
-upstream_cluster:rate-limit-cluster# Limitador address reference
-domain:toystore-app# RLS protocol domain value
-
-
Proposed new design for the WASM filter configuration object (PluginConfig struct):
-
# The filter’s behaviour in case the rate limiting service does not respond back. When it is set to true, Envoy will not allow traffic in case of communication failure between rate limiting service and the proxy.
-failure_mode_deny:true
-rate_limit_policies:
--name:toystore
-rate_limit_domain:toystore-app
-upstream_cluster:rate-limit-cluster
-hostnames:["*.toystore.com"]
-gateway_actions:
--rules:
--paths:["/admin/toy"]
-methods:["GET"]
-hosts:["pets.toystore.com"]
-configurations:
--actions:
--generic_key:
-descriptor_key:admin
-descriptor_value:"1"
-
-
Update highlights:
-* [minor] rate_limit_policies is a list instead of a map indexed by the name/namespace.
-* [major] no distinction between "rules" and global actions
-* [major] more aligned with RLS: multiple descriptors structured by "rate limit configurations" with matching rules
WASM filter rate limit policies are not exactly the same as user managed RateLimitPolicy
-custom resources. The WASM filter rate limit policies is part of the internal configuration
-and therefore not exposed to the end user.
-
At the WASM filter level, there are no route level or gateway level rate limit policies.
-The rate limit policies in the wasm plugin configuration may not map 1:1 to
-user managed RateLimitPolicy custom resources. WASM rate limit policies have an internal logical
-name and a set of hostnames to activate it based on the incoming request’s host header.
-
The WASM filter builds a tree based data structure holding the rate limit policies.
-The longest (sub)domain match is used to select the policy to be applied.
-Only one policy is being applied per invocation.
The WASM filter configuration object contains a list of rate limit configurations
-to build a list of Envoy's RLS descriptors. These configurations are defined at
Each configuration produces, at most, one descriptor. Depending on the incoming request, one configuration may or may not produce a rate limit descriptor.
-
-
-
Each policy configuration has associated, optionally, a set of rules to match. Rules allow matching hosts and/or methods and/or paths. Matching occurs when at least one rule applies against the incoming request. If rules are not set, it is equivalent to matching all the requests.
-
-
-
Each configuration object defines a list of actions. Each action may (or may not) produce a descriptor entry (descriptor list item). If an action cannot append a descriptor entry, no descriptor is generated for the configuration.
-
-
-
Note: The external rate limit service will be called when the gateway_actions object produces at least one not empty descriptor.
WASM filter rate limit policy for *.toystore.com. I want some rate limit descriptors configuration
-only for api.toystore.com and another set of descriptors for admin.toystore.com.
-The wasm filter config would look like this:
A Kuadrant RateLimitPolicy custom resource, often abbreviated "RateLimitPolicy":
-
-
Targets Gateway API networking resources such as HTTPRoutes and Gateways, using these resources to obtain additional context, i.e., which traffic workload (HTTP attributes, hostnames, user attributes, etc) to rate limit.
-
Supports targeting subsets (sections) of a network resource to apply the limits to.
-
Abstracts the details of the underlying Rate Limit protocol and configuration resources, that have a much broader remit and surface area.
-
Enables cluster operators to set defaults that govern behavior at the lower levels of the network, until a more specific policy is applied.
Kuadrant's Rate Limit implementation relies on the Envoy's Rate Limit Service (RLS) protocol. The workflow per request goes:
-1. On incoming request, the gateway checks the matching rules for enforcing rate limits, as stated in the RateLimitPolicy custom resources and targeted Gateway API networking objects
-2. If the request matches, the gateway sends one RateLimitRequest to the external rate limiting service ("Limitador").
-1. The external rate limiting service responds with a RateLimitResponse back to the gateway with either an OK or OVER_LIMIT response code.
-
A RateLimitPolicy and its targeted Gateway API networking resource contain all the statements to configure both the ingress gateway and the external rate limiting service.
The RateLimitPolicy spec includes, basically, two parts:
-
-
A reference to an existing Gateway API resource (spec.targetRef)
-
Limit definitions (spec.limits)
-
-
Each limit definition includes:
-* A set of rate limits (spec.limits.<limit-name>.rates[])
-* (Optional) A set of dynamic counter qualifiers (spec.limits.<limit-name>.counters[])
-* (Optional) A set of route selectors, to further qualify the specific routing rules when to activate the limit (spec.limits.<limit-name>.routeSelectors[])
-* (Optional) A set of additional dynamic conditions to activate the limit (spec.limits.<limit-name>.when[])
-
The limit definitions (limits) can be declared at the top-level level of the spec (with the semantics of defaults) or alternatively within explicit defaults or overrides blocks.
-
-
-
-
Check out Kuadrant RFC 0002 to learn more about the Well-known Attributes that can be used to define counter qualifiers (counters) and conditions (when).
apiVersion:kuadrant.io/v1beta2
-kind:RateLimitPolicy
-metadata:
-name:my-rate-limit-policy
-spec:
-# Reference to an existing networking resource to attach the policy to. REQUIRED.
-# It can be a Gateway API HTTPRoute or Gateway resource.
-# It can only refer to objects in the same namespace as the RateLimitPolicy.
-targetRef:
-group:gateway.networking.k8s.io
-kind:HTTPRoute / Gateway
-name:myroute / mygateway
-
-# The limits definitions to apply to the network traffic routed through the targeted resource.
-# Equivalent to if otherwise declared within `defaults`.
-limits:
-"my_limit":
-# The rate limits associated with this limit definition. REQUIRED.
-# E.g., to specify a 50rps rate limit, add `{ limit: 50, duration: 1, unit: secod }`
-rates:[…]
-
-# Counter qualifiers.
-# Each dynamic value in the data plane starts a separate counter, combined with each rate limit.
-# E.g., to define a separate rate limit for each user name detected by the auth layer, add `metadata.filter_metadata.envoy\.filters\.http\.ext_authz.username`.
-# Check out Kuadrant RFC 0002 (https://github.com/Kuadrant/architecture/blob/main/rfcs/0002-well-known-attributes.md) to learn more about the Well-known Attributes that can be used in this field.
-counters:[…]
-
-# Further qualification of the scpecific HTTPRouteRules within the targeted HTTPRoute that should trigger the limit.
-# Each element contains a HTTPRouteMatch object that will be used to select HTTPRouteRules that include at least one identical HTTPRouteMatch.
-# The HTTPRouteMatch part does not have to be fully identical, but the what's stated in the selector must be identically stated in the HTTPRouteRule.
-# Do not use it on RateLimitPolicies that target a Gateway.
-routeSelectors:[…]
-
-# Additional dynamic conditions to trigger the limit.
-# Use it for filtering attributes not supported by HTTPRouteRule or with RateLimitPolicies that target a Gateway.
-# Check out Kuadrant RFC 0002 (https://github.com/Kuadrant/architecture/blob/main/rfcs/0002-well-known-attributes.md) to learn more about the Well-known Attributes that can be used in this field.
-when:[…]
-
-# Explicit defaults. Used in policies that target a Gateway object to express default rules to be enforced on
-# routes that lack a more specific policy attached to.
-# Mutually exclusive with `overrides` and with declaring `limits` at the top-level of the spec.
-defaults:
-limits:{…}
-
-# Overrides. Used in policies that target a Gateway object to be enforced on all routes linked to the gateway,
-# thus also overriding any more specific policy occasionally attached to any of those routes.
-# Mutually exclusive with `defaults` and with declaring `limits` at the top-level of the spec.
-overrides:
-limits:{…}
-
When a RateLimitPolicy targets a HTTPRoute, the policy is enforced to all traffic routed according to the rules and hostnames specified in the HTTPRoute, across all Gateways referenced in the spec.parentRefs field of the HTTPRoute.
-
The targeted HTTPRoute's rules and/or hostnames to which the policy must be enforced can be filtered to specific subsets, by specifying the routeSelectors field of the limit definition.
-
Target a HTTPRoute by setting the spec.targetRef field of the RateLimitPolicy as follows:
If a RateLimitPolicy targets a route defined for *.com and another RateLimitPolicy targets another route for api.com, the Kuadrant control plane will not merge these two RateLimitPolicies. Unless one of the policies declare an overrides set of limites, the control plane will configure to mimic the behavior of gateway implementation by which the "most specific hostname wins", thus enforcing only the corresponding applicable policies and limit definitions.
-
E.g., by default, a request coming for api.com will be rate limited according to the rules from the RateLimitPolicy that targets the route for api.com; while a request for other.com will be rate limited with the rules from the RateLimitPolicy targeting the route for *.com.
A RateLimitPolicy that targets a Gateway can declare a block of defaults (spec.defaults) or a block of overrides (spec.overrides). As a standard, gateway policies that do not specify neither defaults nor overrides, act as defaults.
-
When declaring defaults, a RateLimitPolicy which targets a Gateway will be enforced to all HTTP traffic hitting the gateway, unless a more specific RateLimitPolicy targeting a matching HTTPRoute exists. Any new HTTPRoute referrencing the gateway as parent will be automatically covered by the default RateLimitPolicy, as well as changes in the existing HTTPRoutes.
-
Defaults provide cluster operators with the ability to protect the infrastructure against unplanned and malicious network traffic attempt, such as by setting safe default limits on hostnames and hostname wildcards.
-
Inversely, a gateway policy that specify overrides declares a set of rules to be enforced on all routes attached to the gateway, thus atomically replacing any more specific policy occasionally attached to any of those routes.
-
Target a Gateway HTTPRoute by setting the spec.targetRef field of the RateLimitPolicy as follows:
Overlapping Gateway and HTTPRoute RateLimitPolicies¶
-
Two possible semantics are to be considered here – gateway policy defaults vs gateway policy overrides.
-
Gateway RateLimitPolicies that declare defaults (or alternatively neither defaults nor overrides) protect all traffic routed through the gateway except where a more specific HTTPRoute RateLimitPolicy exists, in which case the HTTPRoute RateLimitPolicy prevails.
-
Example with 4 RateLimitPolicies, 3 HTTPRoutes and 1 Gateway default (plus 2 HTTPRoute and 2 Gateways without RateLimitPolicies attached):
-- RateLimitPolicy A → HTTPRoute A (a.toystore.com) → Gateway G (*.com)
-- RateLimitPolicy B → HTTPRoute B (b.toystore.com) → Gateway G (*.com)
-- RateLimitPolicy W → HTTPRoute W (*.toystore.com) → Gateway G (*.com)
-- RateLimitPolicy G (defaults) → Gateway G (*.com)
-
Expected behavior:
-- Request to a.toystore.com → RateLimitPolicy A will be enforced
-- Request to b.toystore.com → RateLimitPolicy B will be enforced
-- Request to other.toystore.com → RateLimitPolicy W will be enforced
-- Request to other.com (suppose a route exists) → RateLimitPolicy G will be enforced
-- Request to yet-another.net (suppose a route and gateway exist) → No RateLimitPolicy will be enforced
-
Gateway RateLimitPolicies that declare overrides protect all traffic routed through the gateway, regardless of existence of any more specific HTTPRoute RateLimitPolicy.
-
Example with 4 RateLimitPolicies, 3 HTTPRoutes and 1 Gateway override (plus 2 HTTPRoute and 2 Gateways without RateLimitPolicies attached):
-- RateLimitPolicy A → HTTPRoute A (a.toystore.com) → Gateway G (*.com)
-- RateLimitPolicy B → HTTPRoute B (b.toystore.com) → Gateway G (*.com)
-- RateLimitPolicy W → HTTPRoute W (*.toystore.com) → Gateway G (*.com)
-- RateLimitPolicy G (overrides) → Gateway G (*.com)
-
Expected behavior:
-- Request to a.toystore.com → RateLimitPolicy G will be enforced
-- Request to b.toystore.com → RateLimitPolicy G will be enforced
-- Request to other.toystore.com → RateLimitPolicy G will be enforced
-- Request to other.com (suppose a route exists) → RateLimitPolicy G will be enforced
-- Request to yet-another.net (suppose a route and gateway exist) → No RateLimitPolicy will be enforced
A limit will be activated whenever a request comes in and the request matches:
-- any of the route rules selected by the limit (via routeSelectors or implicit "catch-all" selector), and
-- all of the when conditions specified in the limit.
-
A limit can define:
-- counters that are qualified based on dynamic values fetched from the request, or
-- global counters (implicitly, when no qualified counter is specified)
Route selectors allow targeting sections of a HTTPRoute, by specifying sets of HTTPRouteMatches and/or hostnames that make the policy controller look up within the HTTPRoute spec for compatible declarations, and select the corresponding HTTPRouteRules and hostnames, to then build conditions that activate the policy or policy rule.
-
Check out Route selectors for a full description, semantics and API reference.
when conditions can be used to scope a limit (i.e. to filter the traffic to which a limit definition applies) without any coupling to the underlying network topology, i.e. without making direct references to HTTPRouteRules via routeSelectors.
-
Use when conditions to conditionally activate limits based on attributes that cannot be expressed in the HTTPRoutes' spec.hostnames and spec.rules.matches fields, or in general in RateLimitPolicies that target a Gateway.
-
The selectors within the when conditions of a RateLimitPolicy are a subset of Kuadrant's Well-known Attributes (RFC 0002). Check out the reference for the full list of supported selectors.
One HTTPRoute can only be targeted by one RateLimitPolicy.
-
One Gateway can only be targeted by one RateLimitPolicy.
-
RateLimitPolicies can only target HTTPRoutes/Gateways defined within the same namespace of the RateLimitPolicy.
-
2+ RateLimitPolicies cannot target network resources that define/inherit the same exact hostname.
-
-
Limitation: Multiple network resources with identical hostnames¶
-
Kuadrant currently does not support multiple RateLimitPolicies simultaneously targeting network resources that declare identical hostnames. This includes multiple HTTPRoutes that specify the same hostnames in the spec.hostnames field, as well as HTTPRoutes that specify a hostname that is identical to a hostname specified in a listener of one of the route's parent gateways or HTTPRoutes that don't specify any hostname at all thus inheriting the hostnames from the parent gateways. In any of these cases, a maximum of one RateLimitPolicy targeting any of those resources that specify identical hostnames is allowed.
-
Moreover, having multiple resources that declare identical hostnames may lead to unexpected behavior and therefore should be avoided.
-
This limitation is rooted at the underlying components configured by Kuadrant for the implementation of its policies and the lack of information in the data plane regarding the exact route that honored by the API gateway in cases of conflicting hostnames.
-
To exemplify one way this limitation can impact deployments, consider the following topology:
In the example above, with the policy-1 resource created before policy-2, policy-2 will be enforced on all requests to app.io/bar while policy-1 will not be enforced at all. I.e. app.io/foo will not be rate-limited. Nevertheless, both policies will report status condition as Enforced.
A different way the limitation applies is when two or more routes of a gateway declare the exact same hostname and a gateway policy is defined with expectation to set default rules for the cases not covered by more specific policies. E.g.:
Once again, both policies will report status condition as Enforced. However, in this case, only policy-1 will be enforced on requests to app.io/foo, while policy-2 will not be enforced at all. I.e. app.io/bar will not be not rate-limited. This is same behavior as the analogous problem with the Kuadrant AuthPolicy.
-
To avoid these problems, use different hostnames in each route.
Driven by limitations related to how Istio injects configuration in the filter chains of the ingress gateways, Kuadrant relies on Envoy's Wasm Network filter in the data plane, to manage the integration with rate limiting service ("Limitador"), instead of the Rate Limit filter.
-
Motivation:Multiple rate limit domains
-The first limitation comes from having only one filter chain per listener. This often leads to one single global rate limiting filter configuration per gateway, and therefore to a shared rate limit domain across applications and policies. Even though, in a rate limit filter, the triggering of rate limit calls, via actions to build so-called "descriptors", can be defined at the level of the virtual host and/or specific route rule, the overall rate limit configuration is only one, i.e., always the same rate limit domain for all calls to Limitador.
-
On the other hand, the possibility to configure and invoke the rate limit service for multiple domains depending on the context allows to isolate groups of policy rules, as well as to optimize performance in the rate limit service, which can rely on the domain for indexation.
-
Motivation:Fine-grained matching rules
-A second limitation of configuring the rate limit filter via Istio, particularly from Gateway API resources, is that rate limit descriptors at the level of a specific HTTP route rule require "named routes" – defined only in an Istio VirtualService resource and referred in an EnvoyFilter one. Because Gateway API HTTPRoute rules lack a "name" property1, as well as the Istio VirtualService resources are only ephemeral data structures handled by Istio in-memory in its implementation of gateway configuration for Gateway API, where the names of individual route rules are auto-generated and not referable by users in a policy23, rate limiting by attributes of the HTTP request (e.g., path, method, headers, etc) would be very limited while depending only on Envoy's Rate Limit filter.
-
Motivated by the desire to support multiple rate limit domains per ingress gateway, as well as fine-grained HTTP route matching rules for rate limiting, Kuadrant implements a wasm-shim that handles the rules to invoke the rate limiting service, complying with Envoy's Rate Limit Service (RLS) protocol.
-
The wasm module integrates with the gateway in the data plane via Wasm Network filter, and parses a configuration composed out of user-defined RateLimitPolicy resources by the Kuadrant control plane. Whereas the rate limiting service ("Limitador") remains an implementation of Envoy's RLS protocol, capable of being integrated directly via Rate Limit extension or by Kuadrant, via wasm module for the Istio Gateway API implementation.
-
As a consequence of this design:
-- Users can define fine-grained rate limit rules that match their Gateway and HTTPRoute definitions including for subsections of these.
-- Rate limit definitions are insulated, not leaking across unrelated policies or applications.
-- Conditions to activate limits are evaluated in the context of the gateway process, reducing the gRPC calls to the external rate limiting service only to the cases where rate limit counters are known in advance to have to be checked/incremented.
-- The rate limiting service can rely on the indexation to look up for groups of limit definitions and counters.
-- Components remain compliant with industry protocols and flexible for different integration options.
-
A Kuadrant wasm-shim configuration for 2 RateLimitPolicy custom resources (a Gateway default RateLimitPolicy and a HTTPRoute RateLimitPolicy) looks like the following and it is generated automatically by the Kuadrant control plane:
-
apiVersion:extensions.istio.io/v1alpha1
-kind:WasmPlugin
-metadata:
-name:kuadrant-istio-ingressgateway
-namespace:istio-system
-…
-spec:
-phase:STATS
-pluginConfig:
-failureMode:deny
-rateLimitPolicies:
--domain:istio-system/gw-rlp# allows isolating policy rules and improve performance of the rate limit service
-hostnames:
--'*.website'
--'*.io'
-name:istio-system/gw-rlp
-rules:# match rules from the gateway and according to conditions specified in the policy
--conditions:
--allOf:
--operator:startswith
-selector:request.url_path
-value:/
-data:
--static:# tells which rate limit definitions and counters to activate
-key:limit.internet_traffic_all__593de456
-value:"1"
--conditions:
--allOf:
--operator:startswith
-selector:request.url_path
-value:/
--operator:endswith
-selector:request.host
-value:.io
-data:
--static:
-key:limit.internet_traffic_apis_per_host__a2b149d2
-value:"1"
--selector:
-selector:request.host
-service:kuadrant-rate-limiting-service
--domain:default/app-rlp
-hostnames:
--'*.toystore.website'
--'*.toystore.io'
-name:default/app-rlp
-rules:# matches rules from a httproute and additional specified in the policy
--conditions:
--allOf:
--operator:startswith
-selector:request.url_path
-value:/assets/
-data:
--static:
-key:limit.toystore_assets_all_domains__8cfb7371
-value:"1"
--conditions:
--allOf:
--operator:startswith
-selector:request.url_path
-value:/v1/
--operator:eq
-selector:request.method
-value:GET
--operator:endswith
-selector:request.host
-value:.toystore.website
--operator:eq
-selector:auth.identity.username
-value:""
--allOf:
--operator:startswith
-selector:request.url_path
-value:/v1/
--operator:eq
-selector:request.method
-value:POST
--operator:endswith
-selector:request.host
-value:.toystore.website
--operator:eq
-selector:auth.identity.username
-value:""
-data:
--static:
-key:limit.toystore_v1_website_unauthenticated__3f9c40c6
-value:"1"
-service:kuadrant-rate-limiting-service
-selector:
-matchLabels:
-istio.io/gateway-name:istio-ingressgateway
-url:oci://quay.io/kuadrant/wasm-shim:v0.3.0
-
List of implicit default selectors of HTTPRouteRules whose matching rules activate the policy. At least one HTTPRouteRule must be selected to activate the policy. If omitted, all HTTPRouteRules of the targeted HTTPRoute activate the policy. Do not use it in policies targeting a Gateway.
-
-
-
patterns
-
MapNamedPattern>
-
No
-
Implicit default named patterns of lists of selector, operator and value tuples, to be reused in when conditions and pattern-matching authorization rules.
List of implicit default additional dynamic conditions (expressions) to activate the policy. Use it for filtering attributes that cannot be expressed in the targeted HTTPRoute's spec.hostnames and spec.rules.matches fields, or when targeting a Gateway.
Explicit default definitions. This field is mutually exclusive with any of the implicit default definitions: spec.rules, spec.routeSelectors, spec.patterns, spec.when
Atomic overrides definitions. This field is mutually exclusive with any of the implicit or explicit default definitions: spec.rules, spec.routeSelectors, spec.patterns, spec.when, spec.default
List of selectors of HTTPRouteRules whose matching rules activate the policy. At least one HTTPRouteRule must be selected to activate the policy. If omitted, all HTTPRouteRules of the targeted HTTPRoute activate the policy. Do not use it in policies targeting a Gateway.
-
-
-
patterns
-
MapNamedPattern>
-
No
-
Named patterns of lists of selector, operator and value tuples, to be reused in when conditions and pattern-matching authorization rules.
List of additional dynamic conditions (expressions) to activate the policy. Use it for filtering attributes that cannot be expressed in the targeted HTTPRoute's spec.hostnames and spec.rules.matches fields, or when targeting a Gateway.
Authentication rules. At least one config MUST evaluate to a valid identity object for the auth request to be successful. If omitted or empty, anonymous access is assumed.
-
-
-
metadata
-
MapMetadataRule>
-
No
-
Rules for fetching auth metadata from external sources.
-
-
-
authorization
-
MapAuthorizationRule>
-
No
-
Authorization rules. All policies MUST allow access for the auth request be successful.
Customizations to the response to the authorization request. Use it to set custom values for unauthenticated, unauthorized, and/or success access request.
-
-
-
callbacks
-
MapCallbackRule>
-
No
-
Rules for post-authorization callback requests to external services. Triggered regardless of the result of the authorization request.
List of selectors of HTTPRouteRules whose matching rules activate the auth rule. At least one HTTPRouteRule must be selected to activate the auth rule. If omitted, the auth rule is activated at all requests where the policy is enforced. Do not use it in policies targeting a Gateway.
List of additional dynamic conditions (expressions) to activate the auth rule. Use it for filtering attributes that cannot be expressed in the targeted HTTPRoute's spec.hostnames and spec.rules.matches fields, or when targeting a Gateway.
Caching options for the resolved object returned when applying this auth rule. (Default: disabled)
-
-
-
priority
-
Integer
-
No
-
Priority group of the auth rule. All rules in the same priority group are evaluated concurrently; consecutive priority groups are evaluated sequentially. (Default: 0)
-
-
-
metrics
-
Boolean
-
No
-
Whether the auth rule emits individual observability metrics. (Default: false)
Authentication based on API keys stored in Kubernetes secrets. Use one of: apiKey, jwt, oauth2Introspection, kubernetesTokenReview, x509, plain, anonymous.
Authentication based on client X.509 certificates. The certificates presented by the clients must be signed by a trusted CA whose certificates are stored in Kubernetes secrets. Use one of: apiKey, jwt, oauth2Introspection, kubernetesTokenReview, x509, plain, anonymous.
Identity object extracted from the context. Use this method when authentication is performed beforehand by a proxy and the resulting object passed to Authorino as JSON in the auth request. Use one of: apiKey, jwt, oauth2Introspection, kubernetesTokenReview, x509, plain, anonymous.
Customizations to where credentials are required to be passed in the request for authentication based on this auth rule. Defaults to HTTP Authorization header with prefix "Bearer".
Specification of a JSON object. Use one of: plain, json, wristband.
-
-
-
key
-
String
-
No
-
The key used to add the custom response item (name of the HTTP header or root property of the Dynamic Metadata object). Defaults to the name of the response item if omitted.
A valid Well-known attribute whose resolved value in the data plane will be compared to value, using the operator.
-
-
-
operator
-
String
-
Yes
-
The binary operator to be applied to the resolved value specified by the selector. One of: eq (equal to), neq (not equal to), incl (includes; for arrays), excl (excludes; for arrays), matches (regex).
-
-
-
value
-
String
-
Yes
-
The static value to be compared to the one resolved from the selector.
-
- Note on Limitador
-The Kuadrant operator creates a Limitador CR named `limitador` in the same namespace as the Kuadrant CR. If there is a pre-existing Limitador CR of the same name the kuadrant operator will take ownership of that Limitador CR.
-
-
-
An eviction is allowed if at most "maxUnavailable" limitador pods are unavailable after the eviction, i.e. even in absence of the evicted pod. For example, one can prevent all voluntary evictions by specifying 0. This is a mutually exclusive setting with "minAvailable".
-
-
-
minAvailable
-
Number
-
No
-
An eviction is allowed if at least "minAvailable" limitador pods will still be available after the eviction, i.e. even in the absence of the evicted pod. So for example you can prevent all voluntary evictions by specifying "100%".
Overrides limit definitions. This field is mutually exclusive with the limits field and defaults field. This field is only allowed for policies targeting Gateway in targetRef.kind
-
-
-
limits
-
MapLimit>
-
No
-
Limit definitions. This field is mutually exclusive with the defaults field
List of rate limits associated with the limit definition
-
-
-
counters
-
[]String
-
No
-
List of rate limit counter qualifiers. Items must be a valid Well-known attribute. Each distinct value resolved in the data plane starts a separate counter for each rate limit.
List of selectors of HTTPRouteRules whose matching rules activate the limit. At least one HTTPRouteRule must be selected to activate the limit. If omitted, all HTTPRouteRules of the targeted HTTPRoute activate the limit. Do not use it in policies targeting a Gateway.
List of additional dynamic conditions (expressions) to activate the limit. All expression must evaluate to true for the limit to be applied. Use it for filtering attributes that cannot be expressed in the targeted HTTPRoute's spec.hostnames and spec.rules.matches fields, or when targeting a Gateway.
The route selectors of a policy spec or policy rule (limit definition or auth rule) allow to specify selectors of routes or parts of a route, that transitively induce a set of conditions for a policy or policy rule to be enforced. It is defined as a set of HTTP route matching rules, where these matching rules must exist, partially or identically stated within the HTTPRouteRules of the HTTPRoute that is targeted by the policy.
Route selectors matches and the HTTPRoute's HTTPRouteMatches are pairwise compared to select or not select HTTPRouteRules that should activate a policy rule. To decide whether the route selector selects a HTTPRouteRule or not, for each pair of route selector HTTPRouteMatch and HTTPRoute HTTPRouteMatch:
-1. The route selector selects the HTTPRoute's HTTPRouteRule if the HTTPRouteRule contains at least one HTTPRouteMatch that specifies fields that are literally identical to all the fields specified by at least one HTTPRouteMatch of the route selector.
-2. A HTTPRouteMatch within a HTTPRouteRule may include other fields that are not specified in a route selector match, and yet the route selector match selects the HTTPRouteRule if all fields of the route selector match are identically included in the HTTPRouteRule's HTTPRouteMatch; the opposite is NOT true.
-3. Each field path of a HTTPRouteMatch, as well as each field method of a HTTPRouteMatch, as well as each element of the fields headers and queryParams of a HTTPRouteMatch, is atomic – this is true for the HTTPRouteMatches within a HTTPRouteRule, as well as for HTTPRouteMatches of a route selector.
-
Additionally, at least one hostname specified in a route selector must identically match one of the hostnames specified (or inherited, when omitted) by the targeted HTTPRoute.
-
The semantics of the route selectors allows to assertively relate policy rule definitions to routing rules, with benefits for identifying the subsets of the network that are covered by a policy rule, while preventing unreachable definitions, as well as the overhead associated with the maintenance of such rules across multiple resources throughout time, according to network topology beneath. Moreover, the requirement of not having to be a full copy of the targeted HTTPRouteRule matches, but only partially identical, helps prevent repetition to some degree, as well as it enables to more easily define policy rules that scope across multiple HTTPRouteRules (by specifying less rules in the selector).
A few rules and corner cases to keep in mind while using the RLP's routeSelectors:
-1. The golden rule – The route selectors in a policy or policy rule are not to be interpreted as the route matching rules that activate the policy or policy rule, but as selectors of the route rules that activate the policy or policy rule.
-2. Due to (1) above, this can lead to cases, e.g., where a route selector that states matches: [{ method: POST }] selects a HTTPRouteRule that defines matches: [{ method: POST }, { method: GET }], effectively causing the policy or policy rule to be activated on requests to the HTTP method POST, but also to the HTTP method GET.
-3. The requirement for the route selector match to state patterns that are identical to the patterns stated by the HTTPRouteRule (partially or entirely) makes, e.g., a route selector such as matches: { path: { type: PathPrefix, value: /foo } } to select a HTTPRouteRule that defines matches: { path: { type: PathPrefix, value: /foo }, method: GET }, but not to select a HTTPRouteRule that only defines matches: { method: GET }, even though the latter includes technically all HTTP paths; nor it selects a HTTPRouteRule that only defines matches: { path: { type: Exact, value: /foo } }, even though all requests to the exact path /foo are also technically requests to /foo*.
-4. The atomicity property of fields of the route selectors makes, e.g., a route selector such as matches: { path: { value: /foo } } to select a HTTPRouteRule that defines matches: { path: { value: /foo } }, but not to select a HTTPRouteRule that only defines matches: { path: { type: PathPrefix, value: /foo } }. (This case may actually never happen because PathPrefix is the default value for path.type and will be set automatically by the Kubernetes API server.)
-
Due to the nature of route selectors of defining pointers to HTTPRouteRules, the routeSelectors field is not supported in a RLP that targets a Gateway resource.
Targets Gateway API networking resources Gateways to provide tls for gateway listeners by managing the lifecycle of tls certificates using CertManager.
apiVersion:kuadrant.io/v1alpha1
-kind:TLSPolicy
-metadata:
-name:my-tls-policy
-spec:
-# reference to an existing networking resource to attach the policy to
-# it can only be a Gateway API Gateway resource
-# it can only refer to objects in the same namespace as the TLSPolicy
-targetRef:
-group:gateway.networking.k8s.io
-kind:Gateway
-name:mygateway
-
-
Check out the API reference for a full specification of the TLSPolicy CRD.
Enforcing authentication & authorization with Kuadrant AuthPolicy¶
-
This guide walks you through the process of setting up a local Kubernetes cluster with Kuadrant where you will protect Gateway API endpoints by declaring Kuadrant AuthPolicy custom resources.
-
Two AuthPolicies will be declared:
-
-
-
-
Use case
-
AuthPolicy
-
-
-
-
-
App developer
-
1 AuthPolicy targeting a HTTPRoute that routes traffic to a sample Toy Store application, and enforces API key authentication to all requests in this route, as well as requires API key owners to be mapped to groups:admins metadata to access a specific HTTPRouteRule of the route.
-
-
-
Platform engineer use-case
-
1 AuthPolicy targeting the istio-ingressgateway Gateway that enforces a trivial "deny-all" policy that locks down any other HTTPRoute attached to the Gateway.
curl-H'Host: api.toystore.com'http://$GATEWAY_URL/cars-i
-# HTTP/1.1 200 OK
-
-
curl-H'Host: api.toystore.com'http://$GATEWAY_URL/dolls-i
-# HTTP/1.1 200 OK
-
-
curl-H'Host: api.toystore.com'http://$GATEWAY_URL/admin-i
-# HTTP/1.1 200 OK
-
-
③ Protect the Toy Store application (Persona: App developer)¶
-
Create the AuthPolicy to enforce the following auth rules:
-- Authentication:
- - All users must present a valid API key
-- Authorization:
- - /admin* routes require user mapped to the admins group (kuadrant.io/groups=admins annotation added to the Kubernetes API key Secret)
Authenticated Rate Limiting for Application Developers¶
-
This user guide walks you through an example of how to configure authenticated rate limiting for an application using Kuadrant.
-
Authenticated rate limiting rate limits the traffic directed to an application based on attributes of the client user, who is authenticated by some authentication method. A few examples of authenticated rate limiting use cases are:
-
-
User A can send up to 50rps ("requests per second"), while User B can send up to 100rps.
-
Each user can send up to 20rpm ("request per minute").
-
Admin users (members of the 'admin' group) can send up to 100rps, while regular users (non-admins) can send up to 20rpm and no more than 5rps.
-
-
In this guide, we will rate limit a sample REST API called Toy Store. In reality, this API is just an echo service that echoes back to the user whatever attributes it gets in the request. The API exposes an endpoint at GET http://api.toystore.com/toy, to mimic an operation of reading toy records.
-
We will define 2 users of the API, which can send requests to the API at different rates, based on their user IDs. The authentication method used is API key.
This step uses tooling from the Kuadrant Operator component to create a containerized Kubernetes server locally using Kind,
-where it installs Istio, Kubernetes Gateway API and Kuadrant itself.
-
-
Note: In production environment, these steps are usually performed by a cluster operator with administrator privileges over the Kubernetes cluster.
Authenticated Rate Limiting with JWTs and Kubernetes RBAC¶
-
This user guide walks you through an example of how to use Kuadrant to protect an application with policies to enforce:
-
-
authentication based OpenId Connect (OIDC) ID tokens (signed JWTs), issued by a Keycloak server;
-
alternative authentication method by Kubernetes Service Account tokens;
-
authorization delegated to Kubernetes RBAC system;
-
rate limiting by user ID.
-
-
In this example, we will protect a sample REST API called Toy Store. In reality, this API is just an echo service that echoes back to the user whatever attributes it gets in the request.
-
The API listens to requests at the hostnames *.toystore.com, where it exposes the endpoints GET /toy*, POST /admin/toy and DELETE /amind/toy, respectively, to mimic operations of reading, creating, and deleting toy records.
-
Any authenticated user/service account can send requests to the Toy Store API, by providing either a valid Keycloak-issued access token or Kubernetes token.
-
Privileges to execute the requested operation (read, create or delete) will be granted according to the following RBAC rules, stored in the Kubernetes authorization system:
-
-
-
-
Operation
-
Endpoint
-
Required role
-
-
-
-
-
Read
-
GET /toy*
-
toystore-reader
-
-
-
Create
-
POST /admin/toy
-
toystore-write
-
-
-
Delete
-
DELETE /admin/toy
-
toystore-write
-
-
-
-
Each user will be entitled to a maximum of 5rp10s (5 requests every 10 seconds).
This step uses tooling from the Kuadrant Operator component to create a containerized Kubernetes server locally using Kind,
-where it installs Istio, Kubernetes Gateway API and Kuadrant itself.
-
-
Note: In production environment, these steps are usually performed by a cluster operator with administrator privileges over the Kubernetes cluster.
-
-Q: Can I use Roles and RoleBindings instead of ClusterRoles and ClusterRoleBindings?
-
Yes, you can.
-
The example above is for non-resource URL Kubernetes roles. For using Roles and RoleBindings instead of
- ClusterRoles and ClusterRoleBindings, thus more flexible resource-based permissions to protect the API,
- see the spec for Kubernetes SubjectAccessReview authorization
- in the Authorino docs.
This step uses tooling from the Kuadrant Operator component to create a containerized Kubernetes server locally using Kind,
-where it installs Istio, Kubernetes Gateway API and Kuadrant itself.
This step uses tooling from the Kuadrant Operator component to create a containerized Kubernetes server locally using Kind,
-where it installs Istio, Kubernetes Gateway API and Kuadrant itself.
-
-
Note: In production environment, these steps are usually performed by a cluster operator with administrator privileges over the Kubernetes cluster.
This step uses tooling from the Kuadrant Operator component to create a containerized Kubernetes server locally using Kind,
-where it installs Istio, Kubernetes Gateway API, CertManager and Kuadrant itself.
This guide walks you through using Kuadrant to secure, protect, and connect an API exposed by a Gateway using Kubernetes Gateway API. You can use this walkthrough for a Gateway on a single cluster or a Gateway distributed across multiple clusters with a shared listener hostname. This guide shows how specific user personas can each use Kuadrant to achieve their goals.
This guide expects that you have successfully installed Kuadrant on at least one cluster:
-- You have completed the steps in Install Kuadrant on an OpenShift cluster for one or more clusters.
-- For multicluster scenarios, you have installed Kuadrant on at least two different clusters, and have a shared accessible Redis store.
-- kubectl command line tool is installed.
-- (Optional) User workload monitoring is configured to remote write to a central storage system such as Thanos (also covered in the installation steps).
-
What Kuadrant can do for you in a multicluster environment¶
-
You can leverage Kuadrant's capabilities in single or multiple clusters. The following features are designed to work across multiple clusters as well as in a single-cluster environment.
-
-
Multicluster ingress: Kuadrant provides multicluster ingress connectivity using DNS to bring traffic to your Gateways using a strategy defined in a DNSPolicy.
-
Global rate limiting: Kuadrant can enable global rate limiting use cases when configured to use a shared Redis store for counters based on limits defined by a RateLimitPolicy.
-
Global auth: You can configure Kuadrant's AuthPolicy to leverage external auth providers to ensure different clusters exposing the same API are authenticating and authorizing in the same way.
-
Integration with federated metrics stores: Kuadrant has example dashboards and metrics for visualizing your Gateways and observing traffic hitting those Gateways across multiple clusters.
-
-
Platform engineer user
-
This guide walks you through deploying a Gateway that provides secure communication and is protected and ready for use by development teams to deploy an API. It then walks through using this Gateway in clusters in different geographic regions, leveraging Kuadrant to bring specific traffic to your geo-located Gateways to reduce latency and distribute load, while still being protected and secured with global rate limiting and auth.
-
As an optional extra this guide highlights how, with the user workload monitoring observability stack deployed, these Gateways can then be observed and monitored.
-
Application developer user
-
This guide walks through how you can use the Kuadrant OAS extensions and CLI to generate an HTTPRoute for your API and add specific auth and rate limiting requirements to your API.
While this document uses kubectl, working with multiple clusters is complex, and it is best to use a tool such as Argo CD to manage the deployment of resources to multiple clusters.
To secure communication to the Gateways, you will define a TLS issuer for TLS certificates. This example uses Let's Encrypt, but you can use any issuer supported by cert-manager.
-
The following example uses Let's Encrypt staging, which should also be applied to all clusters.
For Kuadrant to balance traffic using DNS across two or more clusters, you must define a Gateway with a shared host. You will define this by using an HTTPS listener with a wildcard hostname based on the root domain. As mentioned earlier, these resources must be applied to all clusters.
-
NOTE: For now, the Gateway is set to accept an HTTPRoute from the same namespace only. This allows you to restrict who can use the Gateway until it is ready for general use.
Secure and protect the Gateway with TLS rate limiting and auth policies.¶
-
While your Gateway is now deployed, it has no exposed endpoints and your listener is not programmed. Next, you can set up a TLSPolicy that leverages your CertificateIssuer to set up your listener certificates.
-
You will also define an AuthPolicy that will set up a default 403 response for any unprotected endpoints, as well as a RateLimitPolicy that will set up a default artificially low global limit to further protect any endpoints exposed by this Gateway.
-
Set up a default, deny-all AuthPolicy for your Gateway as follows:
Extending this Gateway to multiple clusters and configuring geo-based routing¶
-
To distribute this Gateway across multiple clusters, repeat this setup process for each cluster. By default, this will implement a round-robin DNS strategy to distribute traffic evenly across the different clusters. Setting up your Gateways to serve clients based on their geographic location is straightforward with your current configuration.
-
Assuming you have deployed Gateway instances across multiple clusters as per this guide, the next step involves updating the DNS controller with the geographic regions of the visible Gateways.
-
For instance, if you have one cluster in North America and another in the EU, you can direct traffic to these Gateways based on their location by applying the appropriate labels:
This section of the walkthrough focuses on using an OpenAPI Specification (OAS) to define an API. You will use Kuadrant OAS extensions to specify the routing, authentication, and rate-limiting requirements. Next, you will use the kuadrantctl tool to generate an AuthPolicy, an HTTPRoute, and a RateLimitPolicy, which you will then apply to your cluster to enforce the settings defined in your OAS.
-
NOTE: While this section uses the kuadrantctl tool, this is not essential. You can also create and apply an AuthPolicy, RateLimitPolicy, and HTTPRoute by using the oc or kubectl commands.
-
To begin, you will deploy a new version of the toystore app to a developer namespace:
exportoasPath=examples/oas-apikey.yaml
-# Ensure you still have these environment variables setup from the start of this guide:
-exportrootDomain=example.com
-exportgatewayNS=api-gateway
-
NOTE: The sample OAS has some placeholders for namespaces and domains. You will inject valid values into these placeholders based on your previous environment variables.
-
Generate the resource from your OAS, (envsubst will replace the placeholders):
From this, you can see an AuthPolicy generated based on your OAS that will look for API keys in secrets labeled api_key and look for that key in the header api_key. You can now apply this to the Gateway:
OpenID Connect auth flow (skip if interested in API key only)¶
-
This section of the walkthrough uses the kuadrantctl tool to create an AuthPolicy that integrates with an OpenID provider and a RateLimitPolicy that leverages JWT values for per-user rate limiting. It is important to note that OpenID requires an external provider. Therefore, you should adapt the following example to suit your specific needs and provider.
-
The platform engineer workflow established some default policies for authentication and rate limiting at your Gateway. The new developer-defined policies, which you will create, are intended to target your HTTPRoute and will supersede the existing policies for requests to your API endpoints, similar to your previous API key example.
-
The example OAS uses Kuadrant-based extensions. These extensions enable you to define routing and service protection requirements. For more details, see OpenAPI Kuadrant extensions.
Note: the sample OAS has some placeholders for namespaces and domains - we will inject valid values into these placeholders based on our previous env vars
-
-
Let's use our OAS and kuadrantctl to generate an AuthPolicy to replace the default on the Gateway.
Lastly, you can generate your RateLimitPolicy to add your rate limits, based on your OAS file. Rate limiting is simplified for this walkthrough and is based on either the bearer token or the API key value. There are more advanced examples in the How-to guides on the Kuadrant documentation site, for example:
-Authenticated rate limiting with JWTs and Kubernetes RBAC
-
You can continue to use this sample OAS document, which includes both authentication and a rate limit:
In this guide, we will cover the different policies from Kuadrant and how you can use them to secure, protect and connect an Istio-controlled gateway in a single cluster, and how you can set more refined protection on the HTTPRoutes exposed by that gateway.
You will need to set the KUBECTL_CONTEXT environment variable for the kubectl context of the cluster you are targeting.
-If you have followed the single cluster setup, it should be something like below.
-Adjust the name of the cluster accordingly if you have followed the multi cluster setup.
-
# Typical single cluster context
-exportKUBECTL_CONTEXT=kind-kuadrant-local
-
-# Example context for additional 'multi cluster' clusters
-# export KUBECTL_CONTEXT=kind-kuadrant-local-1
-
-
To help with this walk through, you should also set a KUADRANT_ZONE_ROOT_DOMAIN environment variable to a domain you want to use. If you want to try DNSPolicy, this should also be a domain you have access to the DNS for in AWS Route53 or GCP. E.g.:
-
exportKUADRANT_ZONE_ROOT_DOMAIN=my.domain.iown
-
-
❶ Deploy the example app we will serve via our gateway¶
Now, if you look at the status of the gateway, you will see the error is gone, and the status of the policy will report the listener as now secured with a TLS certificate and the gateway as affected by the TLS policy.
-
Our communication with our gateway is now secured via TLS. Note that any new listeners will also be handled by the TLSPolicy.
-
Let's define a HTTPRoute and test our policy. We will re-use this later on with some of the other policies as well.
We have a secure communication in place. However, there is nothing limiting users from overloading our infrastructure and service components that will sit behind this gateway. Let's add a rate limiting layer to protect our services and infrastructure.
Communication is secured and we have some protection for our infrastructure, but we do not trust any client to access our endpoints. By default, we want to allow only authenticated access. To protect our gateway, we will add a deny-all AuthPolicy. Later, we will override this with a more specific AuthPolicy for the API.
-
kubectl--context$KUBECTL_CONTEXTapply-f-<<EOF
-apiVersion: kuadrant.io/v1beta2
-kind: AuthPolicy
-metadata:
- name: deny-all
- namespace: kuadrant-system
-spec:
- targetRef:
- group: gateway.networking.k8s.io
- kind: Gateway
- name: api-gateway
- rules:
- authorization:
- deny-all:
- opa:
- rego: "allow = false"
- response:
- unauthorized:
- headers:
- "content-type":
- value: application/json
- body:
- value: |
- {
- "error": "Forbidden",
- "message": "Access denied by default by the gateway operator. If you are the administrator of the service, create a specific auth policy for the route."
- }
-EOF
-
-
Let's test it again. This time we expect a 403 Forbidden.
(Skip this step if you did not configure a DNS provider during the setup.)
-
Now, we have our gateway protected and communications secured. We are ready to configure DNS, so it is easy for clients to connect and access the APIs we intend to expose via this gateway. Note that during the setup of this walk through, we created a DNS Provider secret and a ManagedZone resource.
So now we have a wildcard DNS record to bring traffic to our gateway.
-
Let's test it again. This time we expect a 403 still as the deny-all policy is still in effect. Notice we no longer need to set the Host header directly.
-
-
Note: If you have followed through this guide on more than 1 cluster, the DNS record for the HTTPRoute hostname will have multiple IP addresses. This means that requests will be made in a round robin pattern across clusters as your DNS provider sends different responses to lookups. You may need to send multiple requests before one hits the cluster you are currently configuring.
Next, we are going to allow authenticated access to our Toystore API. To do this, we will define an AuthPolicy that targets the HTTPRoute. Note that any new HTTPRoutes will still be affected by the gateway-level policy, but as we want users to now access this API, we need to override that policy. For simplicity, we will use API keys to authenticate the requests, though many other options are available.
The gateway limits are a good set of limits for the general case, but as the developers of this API we know that we only want to allow a certain number of requests to specific users, and a general limit for all other users.
Note: If you configured a DNS provider during the setup and defined the DNSPolicy as described in one of the previous chapters you can omit the --resolve api.${KUADRANT_ZONE_ROOT_DOMAIN}:443:${INGRESS_HOST} flag.
-
Note: If you have followed through this guide on more than 1 cluster, the DNS record for the HTTPRoute hostname will have multiple IP addresses. This means that requests will be made in a round robin pattern across clusters as your DNS provider sends different responses to lookups.
This user guide walks you through an example of how to configure rate limiting for an endpoint of an application using Kuadrant.
-
-
In this guide, we will rate limit a sample REST API called Toy Store. In reality, this API is just an echo service that echoes back to the user whatever attributes it gets in the request. The API listens to requests at the hostname api.toystore.com, where it exposes the endpoints GET /toys* and POST /toys, respectively, to mimic a operations of reading and writing toy records.
-
We will rate limit the POST /toys endpoint to a maximum of 5rp10s ("5 requests every 10 seconds").
This step uses tooling from the Kuadrant Operator component to create a containerized Kubernetes server locally using Kind,
-where it installs Istio, Kubernetes Gateway API and Kuadrant itself.
-
-
Note: In production environment, these steps are usually performed by a cluster operator with administrator privileges over the Kubernetes cluster.
Kuadrant is a re-architecture of API Management using Cloud Native concepts and separating the components to be less coupled,
-more reusable and leverage the underlying kubernetes platform. It aims to deliver a smooth experience to providers and consumers
-of applications & services when it comes to rate limiting, authentication, authorization, discoverability, change management, usage contracts, insights, etc.
-
Kuadrant aims to produce a set of loosely coupled functionalities built directly on top of Kubernetes.
-Furthermore, it only strives to provide what Kubernetes doesn’t offer out of the box, i.e. Kuadrant won’t be designing a new Gateway/proxy,
-instead it will opt to connect with what’s there and what’s being developed (think Envoy, Istio, GatewayAPI).
-
Kuadrant is a system of cloud-native k8s components that grows as users’ needs grow.
-
-
From simple protection of a Service (via AuthN) that is used by teammates working on the same cluster, or “sibling” services, up to AuthZ of users using OIDC plus custom policies.
-
From no rate-limiting to rate-limiting for global service protection on to rate-limiting by users/plans
Kuadrant relies on Istio and the Gateway API
-to operate the cluster (Istio's) ingress gateway to provide API management with authentication (authN),
-authorization (authZ) and rate limiting capabilities.
The control plane takes the customer desired configuration (declaratively as kubernetes custom resources) as input and ensures all components are configured to obey customer's desired behavior. This repository contains the source code of the kuadrant control plane
Installing Kuadrant is a two-step procedure. Firstly, install the Kuadrant Operator and secondly,
-request a Kuadrant instance by creating a Kuadrant custom resource.
The operator is available from OperatorHub.io.
-Just go to the linked page and follow installation steps (or just run these two commands):
-
# Install Operator Lifecycle Manager (OLM), a tool to help manage the operators running on your cluster.
-
-curl-sLhttps://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.23.1/install.sh|bash-sv0.23.1
-
-# Install the operator by running the following command:
-
-kubectlcreate-fhttps://operatorhub.io/install/kuadrant-operator.yaml
-
-
Openshift
-
The operator is available from the Openshift Console OperatorHub.
-Just follow installation steps choosing the "Kuadrant Operator" from the catalog:
(Optionally) deploy istio ingress gateway using the
- Gateway resource.
-
Write and apply the Kuadrant's RateLimitPolicy and/or
- AuthPolicy custom resources targeting the Gateway resource
- to have your gateway traffic protected.
Under the example folder there are examples of OAS 3 that can be used to generate the resources
-
-
As an AuthPolicy and RateLimitPolicy both require a HTTPRoute to target, the user guides for generating those policies include examples of running the kuadrantctl generate gatewayapi httproute command.
In this particular example, the endpoint GET /dog will be protected.
-The token needs to be in the query string of the request included in a parameter named dog_token.
-Kuadrant will validate received tokens against tokens found in kubernetes secrets with label kuadrant.io/apikeys-by: ${sec scheme name}.
-In this particular example the label selector will be: kuadrant.io/apikeys-by: securedDog.
Note: Kuadrant validates tokens against api keys found in secrets. The label selector format kuadrant.io/apikeys-by: ${sec scheme name} is arbitrary and designed for this CLI command.
-
-
For more information about Kuadrant auth based on api key: https://docs.kuadrant.io/authorino/docs/user-guides/api-key-authentication/
The verification steps will lead you to the process of deploying and testing the following api with
-endpoints protected using different security schemes:
-
-
-
-
Operation
-
Security Scheme
-
-
-
-
-
GET /api/v1/cat
-
public (not auth)
-
-
-
POST /api/v1/cat
-
ApiKey in header
-
-
-
GET /api/v1/dog
-
OpenIdConnect
-
-
-
GET /api/v1/snake
-
OpenIdConnect OR ApiKey in query string
-
-
-
-
-
[Optional] Setup SSO service supporting OIDC. For this example, we will be using keycloak.
-
Create a new realm petstore
-
Create a client petstore. In the Client Protocol field, select openid-connect.
-
Configure client settings. Access Type to public. Direct Access Grants Enabled to ON (for this example password will be used directly to generate the token).
-
Add a user to the realm
-
Click the Users menu on the left side of the window. Click Add user.
-
Type the username bob, set the Email Verified switch to ON, and click Save.
-
On the Credentials tab, set the password p. Enter the password in both the fields, set the Temporary switch to OFF to avoid the password reset at the next login, and click Set Password.
-
-
-
-
Now, let's run local cluster to test the kuadrantctl new command to generate authpolicy.
Setup a cluster, Istio and Gateway API CRDs and Kuadrant
-
-
Use our single-cluster quick start script - this will install Kuadrant in a local kind cluster: https://docs.kuadrant.io/getting-started-single-cluster/
The reason headers tell that credential not found.
-Credentials satisfying postCat_cat_api_key authentication is needed.
-
According to the OpenAPI spec, it should be a header named api_key.
-What if we try a wrong token? one token assigned to other endpoint,
-i.e. I_LIKE_SNAKES instead of the valid one I_LIKE_CATS. It should return 401 Unauthorized.
The reason headers tell that the API Key provided is invalid.
-Using valid token (from the secret cat-api-key-1 assigned to POST /api/v1/cats)
-in the api_key header should return 200 Ok
To get the authentication token, this example is using Direct Access Grants oauth2 grant type
-(also known as Client Credentials grant type). When configuring the Keycloak (OIDC provider) client
-settings, we enabled Direct Access Grants to enable this procedure.
-We will be authenticating as bob user with p password.
-We previously created bob user in Keycloak in the petstore realm.
With apiKey it should also work. According to the OpenAPI spec security scheme,
-it should be a query string named snake_token and the token needs to be valid token
-(from the secret snake-api-key-1 assigned to GET /api/v1/snake)
Setup a cluster, Istio and Gateway API CRDs and Kuadrant
-
-
Use our single-cluster quick start script - this will install Kuadrant in a local kind cluster: https://docs.kuadrant.io/getting-started-single-cluster/
This guide demonstrates setting up a CI/CD pipeline using Tekton to deploy Kubernetes Gateway API and Kuadrant resources generated by kuadrantctl, from an OpenAPI specification. In this example, these resources are applied directly to the cluster where Tekton is running.
Define the task that outlines steps to clone a repository, generate Kuadrant and Kubernetes resources using kuadrantctl, and apply them directly to the cluster:
-
kubectlapply-f-<<'EOF'
-apiVersion: tekton.dev/v1beta1
-kind: Task
-metadata:
- name: run-kuadrantctl
- namespace: petstore
-spec:
- params:
- - name: gitRepoUrl
- description: URL of the git repository to clone
- - name: gitRevision
- description: Git revision to checkout (branch, tag, sha)
- workspaces:
- - name: source
- description: Workspace to checkout the git repo
- - name: kubeconfig
- description: Workspace containing kubeconfig for Kubernetes cluster access
- steps:
- - name: clean-workspace
- image: alpine:latest
- script: |
- sh -c 'rm -rf $(workspaces.source.path)/* $(workspaces.source.path)/.[!.]* $(workspaces.source.path)/..?*'
- - name: clone
- image: alpine/git:latest
- script: |
- git clone $(params.gitRepoUrl) $(workspaces.source.path)
- cd $(workspaces.source.path)
- git checkout $(params.gitRevision)
- - name: download-kuadrantctl
- image: curlimages/curl:latest
- script: |
- ARCH=$(uname -m)
- case $ARCH in
- x86_64) BIN_ARCH="amd64";;
- arm64) BIN_ARCH="arm64";;
- aarch64) BIN_ARCH="arm64";;
- *) echo "Unsupported architecture: $ARCH" && exit 1 ;;
- esac
- cd $(workspaces.source.path)
- curl -LO "https://github.com/Kuadrant/kuadrantctl/releases/download/v0.2.0/kuadrantctl-v0.2.0-linux-$BIN_ARCH.tar.gz"
- tar -xzf kuadrantctl-v0.2.0-linux-$BIN_ARCH.tar.gz
- - name: run-kuadrantctl
- image: alpine:latest
- script: |
- cd $(workspaces.source.path)
- mkdir -p generated-resources
- ./kuadrantctl generate kuadrant authpolicy --oas openapi.yaml | tee generated-resources/authpolicy.yaml
- ./kuadrantctl generate kuadrant ratelimitpolicy --oas openapi.yaml | tee generated-resources/ratelimitpolicy.yaml
- ./kuadrantctl generate gatewayapi httproute --oas openapi.yaml | tee generated-resources/httproute.yaml
- - name: apply-resources
- image: lachlanevenson/k8s-kubectl
- script: |
- apk add --no-cache gettext > /dev/null
- cd $(workspaces.source.path)
- export KUADRANT_ZONE_ROOT_DOMAIN=example.com # domain name used in the HTTPRoute for the petstore sample app
- for file in ./generated-resources/*.yaml; do
- envsubst < "$file" | kubectl apply -n petstore -f -
- done
-EOF
-
-
We're using Tekton here with kubectl to apply resources to a cluster. We recommend looking at a tool such as (ArgoCD)[https://argo-cd.readthedocs.io/en/stable/] to implement continuous delivery via a GitOps approach. In this scenario, you would:
-
-
Use kuadrantctl to generate Kubernetes/Kuadrant resources as part a Tekton pipeline
-
Commit these new resources in to a git respository
-
Use ArgoCD to sync these changes via a Git respository to a Kubernetes or OpenShift cluster
OpenAPI Extensions can
-be used to describe extra functionality beyond what is covered by the standard OpenAPI
-specification. They typically start with x-. Kuadrant OpenAPI extensions start with
-x-kuadrant, and allow you to configure Kuadrant policy information along side
-your API.
-
Apicurio Studio is a UI tool for visualising and editing OpenAPI specifications.
-It has support for visualising security and extensions defined in your spec.
-
This guide assumes you have Apicurio Studio already running.
-See https://www.apicur.io/studio/ for info on how to install Apicurio Studio.
Open or import your OpenAPI spec in the Apicurio Studio UI.
-You can modify the source of the spec from the UI.
-There are a few different configuration and extension points supported by Apicurio Studio, and also supported by the kuadrantctl cli.
-
To generate a HTTPRoute for the API, add the following x-kuadrant block to your spec in the UI, replacing values to match your APIs details and the location of your Gateway.
See this guide for more info on generating a HTTPRoute.
-
To generate an AuthPolicy, add a securityScheme to the components block.
-This securityScheme requires that an API key header is set.
-Although securityScheme is not an OpenAPI extension, it is used by kuadrantctl like the other extensions mentioned here.
Kuadrant extension that can be added at the path level of the OpenAPI spec.
-This configuration at the path level
-is the default when there is no operation level configuration.
-
paths:
-/cat:
-x-kuadrant:## Path level Kuadrant Extension
-disable:true## Remove from the HTTPRoute. Optional. Default: false
-pathMatchType:Exact## Specifies how to match against the path Value. Valid values: [Exact;PathPrefix]. Optional. Default: Exact
-backendRefs:## Backend references to be included in the HTTPRoute. []gateway.networking.k8s.io/v1beta1.HTTPBackendRef. Optional.
--name:petstore
-port:80
-namespace:petstore
-rate_limit:## Rate limit config. Optional.
-rates:## Kuadrant API []github.com/kuadrant/kuadrant-operator/api/v1beta2.Rate
--limit:1
-duration:10
-unit:second
-counters:## Kuadrant API []github.com/kuadrant/kuadrant-operator/api/v1beta2.CountextSelector
--auth.identity.username
-when:## Kuadrant API []github.com/kuadrant/kuadrant-operator/api/v1beta2.WhenCondition
--selector:metadata.filter_metadata.envoy\.filters\.http\.ext_authz.identity.userid
-operator:eq
-value:alice
-
Kuadrant extension that can be added at the operation level of the OpenAPI spec.
-Same schema as path level kuadrant extension.
-
paths:
-/cat:
-get:
-x-kuadrant:## Path level Kuadrant Extension
-disable:true## Remove from the HTTPRoute. Optional. Default: path level "disable" value
-pathMatchType:Exact## Specifies how to match against the path Value. Valid values: [Exact;PathPrefix]. Optional. Default: Exact
-backendRefs:## Backend references to be included in the HTTPRoute. Optional.
--name:petstore
-port:80
-namespace:petstore
-rate_limit:## Rate limit config. Optional.
-rates:## Kuadrant API github.com/kuadrant/kuadrant-operator/api/v1beta2.Rate
--limit:1
-duration:10
-unit:second
-counters:## Kuadrant API github.com/kuadrant/kuadrant-operator/api/v1beta2.CountextSelector
--auth.identity.username
-when:## Kuadrant API github.com/kuadrant/kuadrant-operator/api/v1beta2.WhenCondition
--selector:metadata.filter_metadata.envoy\.filters\.http\.ext_authz.identity.userid
-operator:eq
-value:alice
-
Integrating Kuadrant OAS Extensions with Red Hat OpenShift Dev Spaces¶
-
OpenAPI Extensions enhance the standard OpenAPI specification by adding custom functionality. Kuadrant OpenAPI extensions, identified by the prefix x-kuadrant, allow the integration of Kuadrant policies directly within your API specs.
-
Red Hat OpenShift Dev Spaces offers a browser-based, cloud-native IDE that supports rapid and decentralized development within container-based environments. This tutorial demonstrates how to use OpenShift Dev Spaces to modify an OpenAPI specification by incorporating Kuadrant policies and then employ kuadrantctl to create Kubernetes resources for both Gateway API and Kuadrant.
-
To follow along, you'll need access to a Dev Spaces instance. This can be either:
In Dev Spaces, select Create Workspace, enter the URL of your forked repository (e.g., https://github.com/<your-username>/blank-petstore.git), and then click Create & Open.
Upon creating your workspace, Dev Spaces will launch VSCode loaded with your forked repository. Navigate to the openapi.yaml file within the sample app to begin modifications.
Note: The x-kuadrant extension at the path level applies to all HTTP methods defined within. For method-specific policies, move the extension inside the relevant HTTP method block (e.g., get, post).
Note: by default, oc and kubectl in Dev Spaces will target the cluster running Dev Spaces. If you want to apply resources to another cluster, you will need to login with oc or kubectl to another cluster, and pass a different --context to these to apply resources to another cluster.
-
-
You can now apply these policies to a running app via kubectl or oc. If Dev Spaces is running on a cluster where Kuadrant is also installed, you can apply these resources:
kuadrantctl can be installed either by downloading pre-compiled binaries or by compiling from source. For most users, downloading the binary is the easiest and recommended method.
If you prefer to compile from source or are contributing to the project, you can install kuadrantctl using go install. This method requires Golang 1.21 or newer.
-
goinstallgithub.com/kuadrant/kuadrantctl@latest
-
-
This command will compile kuadrantctl and install the binary executable in $GOBIN (defaulting to $GOPATH/bin).
Generate Gateway API resources from an OpenAPI 3.x specification
-
-
-
-
Subcommand
-
Description
-
Flags
-
-
-
-
-
httproute
-
Generate Gateway API HTTPRoute from OpenAPI 3.0.X
-
--oas string Path to OpenAPI spec file (in JSON or YAML format), URL, or '-' to read from standard input (required). -o Output format: 'yaml' or 'json'. Default: yaml
--oas string Path to OpenAPI spec file (in JSON or YAML format), URL, or '-' to read from standard input (required). -o Output format: 'yaml' or 'json'. Default: yaml
--oas string Path to OpenAPI spec file (in JSON or YAML format), URL, or '-' to read from standard input (required). -o Output format: 'yaml' or 'json'. Default: yaml
You can deploy the operator using OLM just running a few commands.
-No need to build any image. Kuadrant engineering team provides latest and
-released version tagged images. They are available in
-the Quay.io/Kuadrant image repository.
-
Create kind cluster
-
makekind-create-cluster
-
-
Deploy OLM system
-
makeinstall-olm
-
-
Deploy the operator using OLM. The make deploy-catalog target accepts the following variables:
The limitador operator outputs 3 levels of log messages: (from lowest to highest level)
-
-
debug
-
info (default)
-
error
-
-
info logging is restricted to high-level information. Actions like creating, deleting or updating kubernetes resources will be logged with reduced details about the corresponding objects, and without any further detailed logs of the steps in between, except for errors.
-
Only debug logging will include processing details.
-
To configure the desired log level, set the environment variable LOG_LEVEL to one of the supported values listed above. Default log level is info.
-
Apart from log level, the controller can output messages to the logs in 2 different formats:
-- production (default): each line is a parseable JSON object with properties {"level":string, "ts":int, "msg":string, "logger":string, extra values...}
-- development: more human-readable outputs, extra stack traces and logging info, plus extra values output as JSON, in the format: <timestamp-iso-8601>\t<log-level>\t<logger>\t<message>\t{extra-values-as-json}
-
To configure the desired log mode, set the environment variable LOG_MODE to one of the supported values listed above. Default log mode is production.
The default resource requirement for Limitador deployments is specified in Limitador v1alpha1 API reference
-and will be applied if the resource requirement is not set in the spec.
Limitador limits counters are stored in a backend storage. This is In contrast to the storage of
-the limits themselves, which are always stored in ephemeral memory. Limitador's operator
-supports several storage configurations:
-
-
In-Memory: ephemeral and cannot be shared
-
Redis: Persistent (depending on the redis storage configuration) and can be shared
-
Redis Cached: Persistent (depending on the redis storage configuration) and can be shared
-
Disk: Persistent (depending on the underlying disk persistence capabilities) and cannot be shared
TTL for cached counters in milliseconds [default: 5000]
-
-
-
ratio
-
Ratio to apply to the TTL from Redis on cached counters [default: 10]
-
-
-
flush-period
-
Flushing period for counters in milliseconds [default: 1000]
-
-
-
max-cached
-
Maximum amount of counters cached [default: 10000]
-
-
-
-
For example:
-
apiVersion:limitador.kuadrant.io/v1alpha1
-kind:Limitador
-metadata:
-name:limitador-sample
-spec:
-storage:
-redis-cached:
-configSecretRef:# The secret reference storing the URL for Redis
-name:redisconfig
-options:# Every option is optional
-ttl:1000
-max-cached:5000
-
spec.storage.disk.persistentVolumeClaim field is an object with the following fields.
-
-
-
-
Field
-
Description
-
-
-
-
-
storageClassName
-
StorageClass of the storage offered by cluster administrators [default: default storage class of the cluster]
-
-
-
resources
-
The minimum resources the volume should have. Resources will not take any effect when VolumeName is provided. This parameter is not updateable when the underlying PV is not resizable. [default: 1Gi]
-
-
-
volumeName
-
The binding reference to the existing PersistentVolume backing this claim [default: null]
Currently limitador only supports collectors using the OpenTelemetry Protocol with TLS disabled. The endpoint configuration option should contain the scheme, host and port of the service. The quantity and level of the information provided by the spans is configured via the verbosity argument.
See the LICENSE and NOTICE files that should have been provided along with this software for details.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
\ No newline at end of file
diff --git a/limitador/LICENSE b/limitador/LICENSE
deleted file mode 100644
index 261eeb9e..00000000
--- a/limitador/LICENSE
+++ /dev/null
@@ -1,201 +0,0 @@
- Apache License
- Version 2.0, January 2004
- http://www.apache.org/licenses/
-
- TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
-
- 1. Definitions.
-
- "License" shall mean the terms and conditions for use, reproduction,
- and distribution as defined by Sections 1 through 9 of this document.
-
- "Licensor" shall mean the copyright owner or entity authorized by
- the copyright owner that is granting the License.
-
- "Legal Entity" shall mean the union of the acting entity and all
- other entities that control, are controlled by, or are under common
- control with that entity. For the purposes of this definition,
- "control" means (i) the power, direct or indirect, to cause the
- direction or management of such entity, whether by contract or
- otherwise, or (ii) ownership of fifty percent (50%) or more of the
- outstanding shares, or (iii) beneficial ownership of such entity.
-
- "You" (or "Your") shall mean an individual or Legal Entity
- exercising permissions granted by this License.
-
- "Source" form shall mean the preferred form for making modifications,
- including but not limited to software source code, documentation
- source, and configuration files.
-
- "Object" form shall mean any form resulting from mechanical
- transformation or translation of a Source form, including but
- not limited to compiled object code, generated documentation,
- and conversions to other media types.
-
- "Work" shall mean the work of authorship, whether in Source or
- Object form, made available under the License, as indicated by a
- copyright notice that is included in or attached to the work
- (an example is provided in the Appendix below).
-
- "Derivative Works" shall mean any work, whether in Source or Object
- form, that is based on (or derived from) the Work and for which the
- editorial revisions, annotations, elaborations, or other modifications
- represent, as a whole, an original work of authorship. For the purposes
- of this License, Derivative Works shall not include works that remain
- separable from, or merely link (or bind by name) to the interfaces of,
- the Work and Derivative Works thereof.
-
- "Contribution" shall mean any work of authorship, including
- the original version of the Work and any modifications or additions
- to that Work or Derivative Works thereof, that is intentionally
- submitted to Licensor for inclusion in the Work by the copyright owner
- or by an individual or Legal Entity authorized to submit on behalf of
- the copyright owner. For the purposes of this definition, "submitted"
- means any form of electronic, verbal, or written communication sent
- to the Licensor or its representatives, including but not limited to
- communication on electronic mailing lists, source code control systems,
- and issue tracking systems that are managed by, or on behalf of, the
- Licensor for the purpose of discussing and improving the Work, but
- excluding communication that is conspicuously marked or otherwise
- designated in writing by the copyright owner as "Not a Contribution."
-
- "Contributor" shall mean Licensor and any individual or Legal Entity
- on behalf of whom a Contribution has been received by Licensor and
- subsequently incorporated within the Work.
-
- 2. Grant of Copyright License. Subject to the terms and conditions of
- this License, each Contributor hereby grants to You a perpetual,
- worldwide, non-exclusive, no-charge, royalty-free, irrevocable
- copyright license to reproduce, prepare Derivative Works of,
- publicly display, publicly perform, sublicense, and distribute the
- Work and such Derivative Works in Source or Object form.
-
- 3. Grant of Patent License. Subject to the terms and conditions of
- this License, each Contributor hereby grants to You a perpetual,
- worldwide, non-exclusive, no-charge, royalty-free, irrevocable
- (except as stated in this section) patent license to make, have made,
- use, offer to sell, sell, import, and otherwise transfer the Work,
- where such license applies only to those patent claims licensable
- by such Contributor that are necessarily infringed by their
- Contribution(s) alone or by combination of their Contribution(s)
- with the Work to which such Contribution(s) was submitted. If You
- institute patent litigation against any entity (including a
- cross-claim or counterclaim in a lawsuit) alleging that the Work
- or a Contribution incorporated within the Work constitutes direct
- or contributory patent infringement, then any patent licenses
- granted to You under this License for that Work shall terminate
- as of the date such litigation is filed.
-
- 4. Redistribution. You may reproduce and distribute copies of the
- Work or Derivative Works thereof in any medium, with or without
- modifications, and in Source or Object form, provided that You
- meet the following conditions:
-
- (a) You must give any other recipients of the Work or
- Derivative Works a copy of this License; and
-
- (b) You must cause any modified files to carry prominent notices
- stating that You changed the files; and
-
- (c) You must retain, in the Source form of any Derivative Works
- that You distribute, all copyright, patent, trademark, and
- attribution notices from the Source form of the Work,
- excluding those notices that do not pertain to any part of
- the Derivative Works; and
-
- (d) If the Work includes a "NOTICE" text file as part of its
- distribution, then any Derivative Works that You distribute must
- include a readable copy of the attribution notices contained
- within such NOTICE file, excluding those notices that do not
- pertain to any part of the Derivative Works, in at least one
- of the following places: within a NOTICE text file distributed
- as part of the Derivative Works; within the Source form or
- documentation, if provided along with the Derivative Works; or,
- within a display generated by the Derivative Works, if and
- wherever such third-party notices normally appear. The contents
- of the NOTICE file are for informational purposes only and
- do not modify the License. You may add Your own attribution
- notices within Derivative Works that You distribute, alongside
- or as an addendum to the NOTICE text from the Work, provided
- that such additional attribution notices cannot be construed
- as modifying the License.
-
- You may add Your own copyright statement to Your modifications and
- may provide additional or different license terms and conditions
- for use, reproduction, or distribution of Your modifications, or
- for any such Derivative Works as a whole, provided Your use,
- reproduction, and distribution of the Work otherwise complies with
- the conditions stated in this License.
-
- 5. Submission of Contributions. Unless You explicitly state otherwise,
- any Contribution intentionally submitted for inclusion in the Work
- by You to the Licensor shall be under the terms and conditions of
- this License, without any additional terms or conditions.
- Notwithstanding the above, nothing herein shall supersede or modify
- the terms of any separate license agreement you may have executed
- with Licensor regarding such Contributions.
-
- 6. Trademarks. This License does not grant permission to use the trade
- names, trademarks, service marks, or product names of the Licensor,
- except as required for reasonable and customary use in describing the
- origin of the Work and reproducing the content of the NOTICE file.
-
- 7. Disclaimer of Warranty. Unless required by applicable law or
- agreed to in writing, Licensor provides the Work (and each
- Contributor provides its Contributions) on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
- implied, including, without limitation, any warranties or conditions
- of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
- PARTICULAR PURPOSE. You are solely responsible for determining the
- appropriateness of using or redistributing the Work and assume any
- risks associated with Your exercise of permissions under this License.
-
- 8. Limitation of Liability. In no event and under no legal theory,
- whether in tort (including negligence), contract, or otherwise,
- unless required by applicable law (such as deliberate and grossly
- negligent acts) or agreed to in writing, shall any Contributor be
- liable to You for damages, including any direct, indirect, special,
- incidental, or consequential damages of any character arising as a
- result of this License or out of the use or inability to use the
- Work (including but not limited to damages for loss of goodwill,
- work stoppage, computer failure or malfunction, or any and all
- other commercial damages or losses), even if such Contributor
- has been advised of the possibility of such damages.
-
- 9. Accepting Warranty or Additional Liability. While redistributing
- the Work or Derivative Works thereof, You may choose to offer,
- and charge a fee for, acceptance of support, warranty, indemnity,
- or other liability obligations and/or rights consistent with this
- License. However, in accepting such obligations, You may act only
- on Your own behalf and on Your sole responsibility, not on behalf
- of any other Contributor, and only if You agree to indemnify,
- defend, and hold each Contributor harmless for any liability
- incurred by, or claims asserted against, such Contributor by reason
- of your accepting any such warranty or additional liability.
-
- END OF TERMS AND CONDITIONS
-
- APPENDIX: How to apply the Apache License to your work.
-
- To apply the Apache License to your work, attach the following
- boilerplate notice, with the fields enclosed by brackets "[]"
- replaced with your own identifying information. (Don't include
- the brackets!) The text should be enclosed in the appropriate
- comment syntax for the file format. We also recommend that a
- file or class name and description of purpose be included on the
- same "printed page" as the copyright notice for easier
- identification within third-party archives.
-
- Copyright [yyyy] [name of copyright owner]
-
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
diff --git a/limitador/doc/how-it-works/index.html b/limitador/doc/how-it-works/index.html
deleted file mode 100644
index dabb5421..00000000
--- a/limitador/doc/how-it-works/index.html
+++ /dev/null
@@ -1,3967 +0,0 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- How it works - Kuadrant Documentation
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Limitador ensures that the most restrictive limit configuration will apply.
-
Limitador will try to match each incoming descriptor with the same namespaced
-counter's conditions and variables.
-The namespace for the descriptors is defined by the domain field
-whereas for the rate limit configuration the namespace field is being used.
-For each matching counter, the counter is increased and the limits checked.
-
One example to illustrate:
-
Let's say we have 1 rate limit configuration (one counter per config):
The counter's condition will match. Then, the counter will be increased and the limit checked.
-If the limit is exceeded, the request will be rejected with 429 Too Many Requests,
-otherwise accepted.
-
Note that the counter is being activated even though it does not match all the entries of the
-descriptor. The same rule applies for the variables field.
-
Currently, the implementation of condition only allow for equal (==) and not equal (!=) operators.
-More operators will be implemented based off the use cases for them.
-
The variables field is a list of keys.
-The matching rule is defined just as the existence of the list of descriptor entries with the
-same key values. If variables is variables: [A, B, C],
-one descriptor matches if it has at least three entries with the same A, B, C keys.
The new syntax formalizes what part of an expression is the identifier and which is the value to test against.
-Identifiers are simple string value, while string literals are to be demarcated by single quotes (') or double quotes
-(") so that foo == " bar" now makes it explicit that the value is to be prefixed with a space character.
-
A few remarks:
- - Only string values are supported, as that's what they really are
- - There is no escape character sequence supported in string literals
- - A new operator has been added, !=
The previous syntax wouldn't differentiate between values and the identifier, so that foo == bar was valid. In this
-case foo was the identifier of the variable, while bar was the value to evaluate it against. Whitespaces before and
-after the operator == would be equally important. SO that foo == bar would test for a foo variable being equal
-to bar where the trailing whitespace after the identifier, and the one prefixing the value, would have been
-evaluated.
Which should output Deprecated syntax for conditions corrected! to stderr while stdout would be the limits using
-the new syntax. It is recommended you manually verify the resulting LIMITS_FILE.
A feature lenient_conditions has been added, which lets you use the syntax used in previous version of the crate.
-The function limitador::limit::check_deprecated_syntax_usages_and_reset() lets you verify if the deprecated syntax
-has been used as limit::Limits are created with their condition strings using the deprecated syntax.
The preferred way of starting and configuring the Limitador server is using the command line:
-
Rate Limiting Server
-
-Usage: limitador-server [OPTIONS] <LIMITS_FILE> [STORAGE]
-
-STORAGES:
- memory Counters are held in Limitador (ephemeral)
- disk Counters are held on disk (persistent)
- redis Uses Redis to store counters
- redis_cached Uses Redis to store counters, with an in-memory cache
-
-Arguments:
- <LIMITS_FILE> The limit file to use
-
-Options:
- -b, --rls-ip <ip>
- The IP to listen on for RLS [default: 0.0.0.0]
- -p, --rls-port <port>
- The port to listen on for RLS [default: 8081]
- -B, --http-ip <http_ip>
- The IP to listen on for HTTP [default: 0.0.0.0]
- -P, --http-port <http_port>
- The port to listen on for HTTP [default: 8080]
- -l, --limit-name-in-labels
- Include the Limit Name in prometheus label
- -v...
- Sets the level of verbosity
- --tracing-endpoint <tracing_endpoint>
- The endpoint for the tracing service
- --validate
- Validates the LIMITS_FILE and exits
- -H, --rate-limit-headers <rate_limit_headers>
- Enables rate limit response headers [default: NONE] [possible values: NONE, DRAFT_VERSION_03]
- --grpc-reflection-service
- Enables gRPC server reflection service
- -h, --help
- Print help
- -V, --version
- Print version
-
The LIMITS_FILE provided is the source of truth for all the limits that will be enforced. The file location will be
-monitored by the server for any changes and be hot reloaded. If the changes are invalid, they will be ignored on hot
-reload, or the server will fail to start.
When starting the server, you point it to a LIMITS_FILE, which is expected to be a yaml file with an array of
-limit definitions, with the following format:
Each condition is an expression producing a boolean value (true or false). All conditionsmust evaluate to
-true for the limit to be applied on a request.
-
Expressions follow the following syntax: $IDENTIFIER $OP $STRING_LITERAL, where:
-
-
$IDENTIFIER will be used to resolve the value at evaluation time, e.g. role
-
$OP is an operator, either == or !=
-
$STRING_LITERAL is a literal string value, " or ' demarcated, e.g. "admin"
-
-
So that role != "admin" would apply the limit on request from all users, but admin's.
Limitador will load all the limit definitions from the LIMITS_FILE and keep these in memory. To enforce these
-limits, Limitador needs to track requests in the form of counters. There would be at least one counter per limit, but
-that number grows when variables are used to qualify counters per some arbitrary values.
As the name implies, Limitador will keep all counters in memory. This yields the best results in terms of latency as
-well as accuracy. By default, only up to 1000 "concurrent" counters will be kept around, evicting the oldest entries.
-"Concurrent" in this context means counters that need to exist at the "same time", based of the period of the limit,
-as "expired" counters are discarded.
-
This storage is ephemeral, as if the process is restarted, all the counters are lost and effectively "reset" all the
-limits as if no traffic had been rate limited, which can be fine for short-lived limits, less for longer-lived ones.
When you want persistence of your counters, such as for disaster recovery or across restarts, using redis will store
-the counters in a redis instance using the provided URL. Increments to individual counters is made within redis
-itself, providing accuracy over these, races tho can occur when multiple Limitador servers are used against a single
-redis and using "stacked" limits (i.e. over different periods). Latency is also impacted, as it results in one
-additional hop to talk to redis and maintain the counters.
-
TLS Support
-
Connect to a redis instance using the rediss:// URL scheme.
-
To enable insecure mode, append #insecure at the end of the URL. For example:
In order to avoid some communication overhead to redis, redis_cached adds an in memory caching layer within the
-Limitador servers. This lowers the latency, but sacrifices some accuracy as it will not only cache counters, but also
-coalesce counters updates to redis over time. See this configuration option for more
-information.
-
TLS Support
-
Connect to a redis instance using the rediss:// URL scheme.
-
To enable insecure mode, append #insecure at the end of the URL. For example:
Uses Redis to store counters, with an in-memory cache
-
-Usage: limitador-server <LIMITS_FILE> redis_cached [OPTIONS] <URL>
-
-Arguments:
- <URL> Redis URL to use
-
-Options:
- --ttl <TTL> TTL for cached counters in milliseconds [default: 5000]
- --ratio <ratio> Ratio to apply to the TTL from Redis on cached counters [default: 10000]
- --flush-period <flush> Flushing period for counters in milliseconds [default: 1000]
- --max-cached <max> Maximum amount of counters cached [default: 10000]
- -h, --help Print help
-
Disk storage using RocksDB. Counters are held on disk (persistent).
-
Counters are held on disk (persistent)
-
-Usage: limitador-server <LIMITS_FILE> disk [OPTIONS] <PATH>
-
-Arguments:
- <PATH> Path to counter DB
-
-Options:
- --optimize <OPTIMIZE> Optimizes either to save disk space or higher throughput [default: throughput] [possible values: throughput, disk]
- -h, --help Print help
-
The default binary will not support Infinispan as a storage backend for counters. If you
-want to give it a try, you would need to build your own binary of the server using:
-
cargo build --release --features=infinispan
-
-
Which will add the infinispan to the supported STORAGES.
-
USAGE:
- limitador-server <LIMITS_FILE> infinispan [OPTIONS] <URL>
-
-ARGS:
- <URL> Infinispan URL to use
-
-OPTIONS:
- -n, --cache-name <cache name> Name of the cache to store counters in [default: limitador]
- -c, --consistency <consistency> The consistency to use to read from the cache [default:
- Strong] [possible values: Strong, Weak]
- -h, --help Print help information
-
-
For an in-depth coverage of the different topologies supported and how they affect the behavior, see the
-topologies' document.
The Limitador server has some options that can be configured with environment variables. These will override the
-default values the server uses. Any argument used when starting the server will prevail over the
-environment variables.
YAML file that contains the limits to create when Limitador boots. If the
-limits specified already have counters associated, Limitador will not delete them.
-Changes to the file will be picked up by the running server.
Enables using limit names as labels in Prometheus metrics. This is disabled by
-default because for a few limits it should be fine, but it could become a
-problem when defining lots of limits. See the caution note in the Prometheus
-docs
Enables a storage implementation that uses Redis, but also caches some data in
-memory. The idea is to improve throughput and latencies by caching the counters
-in memory to reduce the number of accesses to Redis. To achieve that, this mode
-sacrifices some rate-limit accuracy. This mode does two things:
-
Batches counter updates. Instead of updating the counters on every
-request, it updates them in memory and commits them to Redis in batches. The
-flushing interval can be configured with the
-REDIS_LOCAL_CACHE_FLUSHING_PERIOD_MS
-env. The trade-off is that when running several instances of Limitador,
-other instances will not become aware of the counter updates until they're
-committed to Redis.
-
Caches counters. Instead of fetching the value of a counter every time
-it's needed, the value is cached for a configurable period. The trade-off is
-that when running several instances of Limitador, an instance will not
-become aware of the counter updates other instances do while the value is
-cached. When a counter is already at 0 (limit exceeded), it's cached until
-it expires in Redis. In this case, no matter what other instances do, we
-know that the quota will not be reestablished until the key expires in
-Redis, so in this case, rate-limit accuracy is not affected. When a counter
-has still some quota remaining the situation is different, that's why we can
-tune for how long it will be cached. The formula is as follows:
-MIN(ttl_in_redis/REDIS_LOCAL_CACHE_TTL_RATIO_CACHED_COUNTERS,
-REDIS_LOCAL_CACHE_MAX_TTL_CACHED_COUNTERS_MS).
-For example, let's image that the current TTL (time remaining until the
-limit resets) in Redis for a counter is 10 seconds, and we set the ratio to
-2, and the max time for 30s. In this case, the counter will be cached for 5s
-(min(10/2, 30)). During those 5s, Limitador will not fetch the value of that
-counter from Redis, so it will answer faster, but it will also miss the
-updates done by other instances, so it can go over the limits in that 5s
-interval.
The RedisLabs version of Redis supports active-active
-replication.
-Limitador is compatible with that deployment mode, but there are a few things to
-take into account regarding limit accuracy.
With an active-active deployment, the data needs to be replicated between
-instances. An update in an instance takes a short time to be reflected in the
-other. That time lag depends mainly on the network speed between the Redis
-instances, and it affects the accuracy of the rate-limiting performed by
-Limitador because it can go over limits while the updates of the counters are
-being replicated.
-
The impact of that greatly depends on the use case. With limits of a few
-seconds, and a low number of hits, we could easily go over limits. On the other
-hand, if we have defined limits with a high number of hits and a long period,
-the effect will be basically negligible. For example, if we define a limit of
-one hour, and we know that the data takes around one second to be replicated,
-the accuracy loss is going to be negligible.
Limitador is a generic rate-limiter written in Rust. It can be used as a
-library, or as a service. The service exposes HTTP endpoints to apply and observe
-limits. Limitador can be used with Envoy because it also exposes a grpc service, on a different
-port, that implements the Envoy Rate Limit protocol (v3).
By default, Limitador starts the HTTP server in localhost:8080, and the grpc
-service that implements the Envoy Rate Limit protocol in localhost:8081. That
-can be configured with these ENVs: ENVOY_RLS_HOST, ENVOY_RLS_PORT,
-HTTP_API_HOST, and HTTP_API_PORT.
-
Or using the command line arguments:
-
Rate Limiting Server
-
-Usage: limitador-server [OPTIONS] <LIMITS_FILE> [STORAGE]
-
-STORAGES:
- memory Counters are held in Limitador (ephemeral)
- disk Counters are held on disk (persistent)
- redis Uses Redis to store counters
- redis_cached Uses Redis to store counters, with an in-memory cache
-
-Arguments:
- <LIMITS_FILE> The limit file to use
-
-Options:
- -b, --rls-ip <ip>
- The IP to listen on for RLS [default: 0.0.0.0]
- -p, --rls-port <port>
- The port to listen on for RLS [default: 8081]
- -B, --http-ip <http_ip>
- The IP to listen on for HTTP [default: 0.0.0.0]
- -P, --http-port <http_port>
- The port to listen on for HTTP [default: 8080]
- -l, --limit-name-in-labels
- Include the Limit Name in prometheus label
- -v...
- Sets the level of verbosity
- --tracing-endpoint <tracing_endpoint>
- The endpoint for the tracing service
- --validate
- Validates the LIMITS_FILE and exits
- -H, --rate-limit-headers <rate_limit_headers>
- Enables rate limit response headers [default: NONE] [possible values: NONE, DRAFT_VERSION_03]
- -h, --help
- Print help
- -V, --version
- Print version
-
-
When using environment variables, these will override the defaults. While environment variable are themselves
-overridden by the command line arguments provided. See the individual STORAGES help for more options relative to
-each of the storages.
Limitador has to be started with a YAML file that has some limits defined.
-There's an example file that allows 10 requests per minute and per user_id when the HTTP method is "GET" and 5 when it is a "POST".
-You can run it with Docker (replace latest with the version you want):
-
If you want to use Limitador with Envoy, there's a minimal Envoy config for
-testing purposes here. The config
-forwards the "userid" header and the request method to Limitador. It assumes
-that there's an upstream API deployed on port 1323. You can use
-echo, for example.
-
Limitador has several options that can be configured via ENV. This
-doc specifies them.
Limitador can store its limits and counters in-memory, disk or in Redis. In-memory is
-faster, but the limits are applied per instance. When using Redis, multiple
-instances of Limitador can share the same limits, but it's slower.
The purpose of this documentation is to deploy a sample application published via AWS ELB, that will be ratelimited at infrastructure level, thanks to the use the envoyproxy sidecar container, that will be in charge of contacting to a ratelimit service (limitador), that will allow the request (or not) if it is within the permitted limits.
-
There are mainly two recommended way of using limitador in kubernetes:
-
-
There is an ingress based on envoyproxy that contacts with limitador ratelimit service before forwarding (or not) the request to the application
-
There is an envoyproxy sidecar container living in the application pod that contacts with limitador ratelimit service before forwarding (or not) the request to the main application container in the same pod
-
-
In this example it will be described the second scenario (where there is an application with an envoyproxy sidecar container contacting to limitador service).
-
-
NOTE
- If you don't want to manually manage the sidecar container definitions on your deployments (harcoding the container spec, loading the envoy configuration from a configmap that requires a pod restart to reload possibles configuration changes...), you can use marin3r, a light weight envoy control plane that allows you to inject envoyproxy sidecar containers and dynamically consume configs from Kubernetes custom resources.
-
-
This is the network diagram of the deployed example:
Application (a sample application deployment called kuard):
-
App has an envoyproxy sidecar container with its configuration file in a configmap, composed by:
-
Cluster kuard points to main application container (127.0.0.1:8080)
-
Cluster kuard_ratelimit points to limitador headless service (limitador:8081)
-
Listener HTTP points to envoyproxy sidecar (0.0.0.0:38080)
-
When envoy contacts with the ratelimit service, you can define a timeout, and if there is no response within that timeout (because ratelimit is overloaded taking more time to process the request, or because rateliit service is down), you can choose from envoy to deny the request or pass it to the application. In this case, there is set a 1s timeout, and if there is no answer in this 1 second, request is passed to the application (failure_mode_deny: false), so we guarantee that the maximum overhead added by a non working ratelimit service is 1 extra second to the final response time.
-
-
-
-
App service published with type: LoadBalancer, which creates a AWS ELB. This service has an annotation to enable proxy protocol on the AWS Load balancer in order to be able to keep the real client IP at envoy level (instead of the k8s node private IP), so it can be used to ratelimit per each real client IP if desired.
-
-
-
Ratelimit application (a deployment called limitador):
-
-
Limitador Configmap with limits definition (1000 rps per hostname).
-
-
Limitador headless service published on limitador:8081. It is important to use a headless service in order to balance correctly the traffic between limitador pods, otherwise GRPC connections are not well balanced.
-
-
-
Redis database to persist ratelimit configuration:
Used to executed hey tool benchmarks from the cluster, so we ensure network latency does not affect the results. Actually, to achieve better results, this pod should be on another cluster (to not share the network between client and network) and be placed on the same Region (to reduce latency). The client could be a bottle neck for the performance test.
-
This centos is going to public AWS ELB to access the app, so simulating it is a normal client from the same Region
-
Prometheus monitoring and grafana dashboard resources
Deploy limitador application. It is important to create the configmap with limitador limits before the deployment, in order to load it from limitador pods. At the moment, if you update the limits configmap you need to restart the pods. Additionally, limitador has an API in order to load limits dynamically, but for simplicity for this test a configmap has been used:
-
Deploy sample kuard application with the envoyproxy sidecar container (if you do any change on the envoy configmap, remember you need to restart app pods in order to reload the config):
-
At this point you shoud see all pods running, and kuard pods should have 2 containers (the main kuard container, and the envoyproxy sidecar container):
-
Now you should be able to access to kuard application using the load balancer DNS name:
-
▶ kubectl get service kuard
-NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
-kuard LoadBalancer 172.30.117.198 a96d5449fbc3f4cd892e15e5b36cde48-457963259.us-east-1.elb.amazonaws.com 80:31450/TCP 4m
-
-
-
-
If you go to the browser and paste the EXTERNAL-IP, your request will follow the next workflow:
-
-
The requests will go from your local machine through internet to the public AWS ELB where the app is published
-
Then it will go to the NodePort of your k8s cluster nodes
-
Once on a k8s node, it will go to kuard Service Virtual IP, and will arrive to an envoyproxy sidecar container inside kuard pod
-
Envoyproxy sidecar container will contact with limitador headless Service, to authorize the requests or not:
-
If the request is authorized (within the configured limits), it will send the request to the app container (0.0.0.0:8080) in the same pod, and request will end up with a HTTP 200 response
-
If the request is limited (beyond the limits), request will end up with HTTP 429 response
In order to scrape that metrics within a prometheus-operator deployed in the cluster, you need to create a PodMonitor resource for every application:
-
Then, if you have grafana deployed in the cluster, you can import a Kuadrant Limitador grafana dashboard that we have prepared, which includes:
-- Kuard envoyproxy sidecar metrics (globally and per pod)
-- Limitador metrics (globally and per pod)
-- And for every deployed component (limitador, kuard, redis):
- - Number of pods (total, available, unavaible, pod restarts...)
- - CPU usage per pod
- - Memory usage per pod
- - Network usage per pod
11932 requests (about 12000) were limited (HTTP 429)
-
Average latency (since the request goes out from the client to AWS ELB, k8s node, envoyproxy container, limitador+redis, kuar app container) is 10ms
-
-
-
-
In addition, if we do a longer test with 5 minutes traffic for example, you can check with the grafana dashboard how these requests are processed by envoyproxy sidecar container of kuard pods and limitador pods:
-
-
Kuard Envoyproxy Sidecar Metrics:
-
Globally it handles around 1200rps: it permits around 1krps and limits around 200rps
-
Each envoyproxy sidecar of each kuard pod handles around half of the traffic: it permits around 500rps and limits around 100rps. The balance between pods is not 100% perfect, caused by random iptables forwarding when using a k8s service
-
-
-
-
Limitador Metrics:
-
Globally it handles around 1200rps: it permits around 1krps and limits around 200rps
-
Each limitador pod handles around half of the traffic: it permits around 500rps and limits around 100rps. The balance between pods is perfect thanks to using a headless service with GRPC connections
-
In this guide, we will go through defining a Gateway in the OCM hub cluster that can then be distributed to and instantiated on a set of managed spoke clusters.
If you are looking to change provider from the default Istio:
-* Please have the Gateway provider of your choice installed and configured (in this example we use Envoy gateway. See the following docs)
export MGC_SUB_DOMAIN in each terminal if you haven't already added it to your .zshrc or .bash_profile.
-
Going through the quick start above, will ensure that a supported GatewayClass is registered in the hub cluster that the Kuadrant multi-cluster gateway controller will handle.
-
NOTE The quick start script will create a placement resource as part of the setup. You can use this as further inspiration for other placement resources you would like to create.
Once you have the Kuadrant multi-cluster gateway controller installed into the OCM hub cluster, you can begin defining and placing Gateways across your OCM managed infrastructure.
-
To define a Gateway and have it managed by the multi-cluster gateway controller, we need to do the following things
-
-
Create a Gateway API Gateway resource in the Hub cluster, ensuring the gateway resource specifies the correct gateway class allowing it to be picked up and managed by the multi-cluster gateway controller
While we recommend using Istio as the gateway provider as that is how you will get access to the full suite of policy APIs, it is possible to use another provider if you choose to however this will result in a reduced set of applicable policy objects.
-
If you are only using the DNSPolicy and TLSPolicy resources, you can use these APIs with any Gateway provider. To change the underlying provider, you need to set the gatewayclass param downstreamClass.
-
-
-
Create the following configmap. Note: In this example, 'eg' stands for the Envoy gateway, which is mentioned in the prerequisites above:
Once this has been created, any gateways created from that gateway class will result in a downstream gateway being provisioned with the configured downstreamClass.
-Run the following in both your hub and spoke cluster to see the gateways:
Gateways and DNS records have a 1:1 relationship only, when a gateway gets deleted the corresponding DNS record also gets marked for deletion. This then triggers the DNS record to be removed from the managed zone in the DNS provider (currently only route 53 is accepted).
-3. Certs and secrets deletion :
-
When a gateway is created a cert is also created for the host in the gateway, this is also removed when the gateway is deleted.
This document will detail the setup of a reference architecture to support a number of API management use-cases connecting Kuadrant with other projects the wider API management on Kubernetes ecosystem.
This document will guide you in installing metrics for your application and provide directions on where to access them. Additionally, it will include dashboards set up to display these metrics.
Prior to commencing the metrics installation process, it is imperative that you have successfully completed the initial getting started guide. For reference, please consult the guide available at the following link: Getting Started Guide.
In the graph view you should see something that looks like the graph below.
-This shows the rate of requests (per second) for each Isito workload.
-In this case, there is 1 workload, balanced across 2 clusters.
-
-
To see the rate of requests per cluster (actually per pod across all clusters), the below query can be used.
-Over long periods of time, this graph can show traffic load balancing between application instances.
To view the operational metrics and status, proceed with the following steps:
-
-
Access the Grafana dashboard by clicking or entering the provided URL for the Grafana UI in your web browser.
-
-
https://grafana.172.31.0.2.nip.io
-
-
-
Note: The default login credentials for Grafana are admin/admin. You may need to accept the non-CA signed certificate to proceed.
-
-
-
Navigate to the included Grafana Dashboard
-
-
Using the left sidebar in the Grafana UI, navigate to Dashboards > Browse and select the Istio Workload Dashboard, MGC SRE Dashboard or any of the following Gateway Api State dashboards.
-
-
In Istio Workload Dashboard you should be able to see the following layout, which will include data from the curl command you ran in the previous section.
-
-
The MGC SRE Dashboard displays real-time insights and visualizations of resources managed by the multicluster-gateway-controller e.g. DNSPolicy, TLSPolicy, DNSRecord etc..
-
-
The Gateway API State / Gateways provides real-time insights and visualizations for Gateways. It offers information about gateway listeners, listener status, gateway status, addresses, and attached routes
-
-
The Gateway API State / GatewayClasses provides insights into Gateways organized by their respective Gateway Classes. It offers information about GatewayClasses and the supported features for each class.
-
-
The Gateway API State / HTTPRoutes or any of the remaining routes focuses on their Routes and provides insights into their configuration. It displays their targeted parent references, and attached parent references, offering a detailed view of how these routes are structured and associated with their respective resources.
-
-
The Grafana dashboard will provide you with real-time insights and visualizations of your gateway's performance and metrics.
-
By utilizing the Grafana dashboard, you can effectively monitor the health and behavior of your system, making informed decisions based on the displayed data. This monitoring capability enables you to proactively identify and address any potential issues to ensure the smooth operation of your environment.
This document will walk you through using Open Cluster Management (OCM) and Kuadrant to configure and deploy a multi-cluster gateway.
-
You will also deploy a simple application that uses that gateway for ingress and protects that applications endpoints with a rate limit policy.
-
We will start with a hub cluster and 2 workload clusters and highlight the automatic TLS integration and also the automatic DNS load balancing between gateway instances.
In this walkthrough, we'll deploy test echo services across multiple clusters. If you followed the Getting Started - Multi Cluster Guide, you would have already set up a KUADRANT_ZONE_ROOT_DOMAIN environment variable. For this tutorial, we'll derive a host from this domain for these echo services.
NAME DOMAIN NAME ID RECORD COUNT NAMESERVERS READY
-mgc-dev-mz test.hcpapps.net /hostedzone/Z08224701SVEG4XHW89W0 7 ["ns-1414.awsdns-48.org","ns-1623.awsdns-10.co.uk","ns-684.awsdns-21.net","ns-80.awsdns-10.com"] True
-
-
-
-
You are now ready to begin creating a gateway!
-
-
We will now create a multi-cluster gateway definition in the hub cluster:
In the hub cluster there will be a single gateway definition but no actual gateway for handling traffic yet. This is because we haven't placed the gateway yet onto any of our ingress clusters.
-
-
-
To place the gateway, we need to add a placement label to gateway resource to instruct the gateway controller where we want this gateway instantiated:
Once this is done, the Kuadrant multi-cluster gateway controller will pick up when a HTTPRoute has been attached to the Gateway it is managing from the hub and it will setup a DNS record to start bringing traffic to that gateway for the host defined in that listener.
So now we have a working gateway with DNS and TLS configured. Let's place this gateway on the workload clusters and bring traffic to those gateways also.
-
-
-
We need to modify our placement to update our numberOfClusters to 2. To patch, run:
NAME TYPE DATA AGE
-apps-hcpapps-tls kubernetes.io/tls 3 13m
-
-
-
-
The listener is configured to use this TLS secret also. So now our gateway has been placed and is running in the right locations with the right configuration and TLS has been setup for the HTTPS listeners.
-
So now we have workload ingress clusters configured with the same Gateway.
-
-
-
Let's create the HTTPRoute in the first workload cluster. Again, remembering to replace the hostname accordingly if you haven't already set a value for the KUADRANT_ZONE_ROOT_DOMAIN variable as described in the Getting Started - Multi Cluster Guide:
-
A DNSPolicy acts against a target Gateway by processing its listeners for hostnames that it can create dns records for. In order for it to do this, it must know about dns providers, and what domains these dns providers are currently hosting.
-This is done through the creation of ManagedZones and dns provider secrets containing the credentials for the dns provider account.
-
If for example a Gateway is created with a listener with a hostname of echo.apps.hcpapps.net:
-
targetRef field is taken from policy attachment's target reference API. It can only target one resource at a time. Fields included inside:
-- Group is the group of the target resource. Only valid option is gateway.networking.k8s.io.
-- Kind is kind of the target resource. Only valid options are Gateway.
-- Name is the name of the target resource.
-- Namespace is the namespace of the referent. Currently only local objects can be referred so value is ignored.
The health check section is optional, the following fields are available:
-
-
allowInsecureCertificates: Added for development environments, allows health probes to not fail when finding an invalid (e.g. self-signed) certificate.
-
additionalHeadersRef: A reference to a secret which contains additional headers such as an authentication token
-
endpoint: The path to specify for these health checks, e.g. /healthz
-
expectedResponses: Defaults to 200 or 201, this allows other responses to be considered valid
-
failureThreshold: How many consecutive fails are required to consider this endpoint unhealthy
-
port: The port to connect to
-
protocol: The protocol to use for this connection
-
-
For more information about DNS Health Checks, see this guide.
The DNSPolicy will create a DNSRecord resource for each listener hostname with a suitable ManagedZone configured. The DNSPolicy resource uses the status of the Gateway to determine what dns records need to be created based on the clusters it has been placed onto.
Configuration of DNS Load Balancing features is done through the loadBalancing field in the DNSPolicy spec.
-
loadBalancing field contains the specification of how dns will be configured in order to provide balancing of load across multiple clusters. Fields included inside:
-- weighted field describes how weighting will be applied to weighted dns records. Fields included inside:
- - defaultWeight arbitrary weight value that will be applied to weighted dns records by default. Integer greater than 0 and no larger than the maximum value accepted by the target dns provider.
- - custom array of custom weights to apply when custom attribute values match.
-- geo field enables the geo routing strategy. Fields included inside:
- - defaultGeo geo code to apply to geo dns records by default. The values accepted are determined by the target dns provider.
A DNSPolicy with an empty loadBalancing spec, or with a loadBalancing.weighted.defaultWeight set and nothing else produces a set of records grouped and weighted to produce a Round Robin routing strategy where all target clusters will have an equal chance of being returned in DNS queries.
-
If we apply the following update to the DNSPolicy:
-
apiVersion:kuadrant.io/v1alpha1
-kind:DNSPolicy
-metadata:
-name:prod-web
-namespace:multi-cluster-gateways
-spec:
-targetRef:
-name:prod-web
-group:gateway.networking.k8s.io
-kind:Gateway
-loadBalancing:
-weighted:
-defaultWeight:100# <--- New Default Weight being added
-
-
The weight of all records is adjusted to reflect the new defaultWeight value of 100. This will still produce the same Round Robin routing strategy as before since all records still have equal weight values.
In the above scenario the managed cluster kind-mgc-workload-2 (GCP) IP address will be returned far less frequently in DNS queries than kind-mgc-workload-1 (AWS)
To enable Geo Load balancing the loadBalancing.geo.defaultGeo field should be added. This informs the DNSPolicy that we now want to start making use of Geo Location features in our target provider.
-This will change the single record set group created from default (What is created for weighted only load balancing) to a geo specific one based on the value of defaultGeo.
-
If we apply the following update to the DNSPolicy:
-
The DNSRecord for our listener host gets updated, and the default geo is replaced with the one we specified:
-
kubectlgetdnsrecordecho.apps.hcpapps.net-nmulti-cluster-gateways-oyaml|yq.spec.endpoints
--dnsName:24osuu.lb-2903yb.echo.apps.hcpapps.net
-recordTTL:60
-recordType:A
-targets:
--172.31.202.1
--dnsName:echo.apps.hcpapps.net
-recordTTL:300
-recordType:CNAME
-targets:
--lb-2903yb.echo.apps.hcpapps.net
--dnsName:lb-2903yb.echo.apps.hcpapps.net# <--- New `us` geo location CNAME is created
-providerSpecific:
--name:geo-country-code
-value:US
-recordTTL:300
-recordType:CNAME
-setIdentifier:US
-targets:
--us.lb-2903yb.echo.apps.hcpapps.net
--dnsName:lb-2903yb.echo.apps.hcpapps.net
-providerSpecific:
--name:geo-country-code
-value:'*'
-recordTTL:300
-recordType:CNAME
-setIdentifier:default
-targets:
--us.lb-2903yb.echo.apps.hcpapps.net# <--- Default catch all CNAME is updated to point to `us` target
--dnsName:lrnse3.lb-2903yb.echo.apps.hcpapps.net
-recordTTL:60
-recordType:A
-targets:
--172.31.201.1
--dnsName:us.lb-2903yb.echo.apps.hcpapps.net# <--- Gateway default group is now `us`
-providerSpecific:
--name:weight
-value:"10"
-recordTTL:60
-recordType:CNAME
-setIdentifier:24osuu.lb-2903yb.echo.apps.hcpapps.net
-targets:
--24osuu.lb-2903yb.echo.apps.hcpapps.net
--dnsName:us.lb-2903yb.echo.apps.hcpapps.net# <--- Gateway default group is now `us`
-providerSpecific:
--name:weight
-value:"255"
-recordTTL:60
-recordType:CNAME
-setIdentifier:lrnse3.lb-2903yb.echo.apps.hcpapps.net
-targets:
--lrnse3.lb-2903yb.echo.apps.hcpapps.net
-
-
-
The listener hostname is still resolvable, but now routed through the us record set:
-
digecho.apps.hcpapps.net+short
-lb-2903yb.echo.apps.hcpapps.net.
-us.lb-2903yb.echo.apps.hcpapps.net.# <--- `us` CNAME now in the chain
-lrnse3.lb-2903yb.echo.apps.hcpapps.net.
-172.31.201.1
-
The defaultGeo as described above puts all clusters into the same geo group, but for geo to be useful we need to mark our clusters as being in different locations.
-We can do this though by adding geo-code attributes on the ManagedCluster to show which county each cluster is in. The values that can be used are determined by the dns provider (See Below).
-
Apply geo-code labels to each of our managed cluster resources:
-
The above indicates that kind-mgc-workload-1 is located in the US (United States), which is the same as our current default geo, and kind-mgc-workload-2 is in ES (Spain).
-
The DNSRecord for our listener host gets updated, and records are now divided into two groups, us and es:
-
In the above scenario any requests made in Spain will be returned the IP address of kind-mgc-workload-2 and requests made from anywhere else in the world will be returned the IP address of kind-mgc-workload-1.
-Weighting of records is still enforced between clusters in the same geo group, in the case above however they are having no effect since there is only one cluster in each group.
-
-If an unsupported value is given to a provider, DNS records will not be created. Please choose carefully. For more information on what location is right for your needs please, read that provider's documentation (see links below).
Continents and country codes supported by AWS Route 53¶
-
:Note: For more information please the official AWS documentation
-
To see all regions supported by AWS Route 53, please see the official (documentation)[https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resource-record-sets-values-geo.html]
In this guide, we will rate limit a sample REST API called Toy Store. In reality, this API is just an echo service that echoes back to the user whatever attributes it gets in the request. The API listens to requests at the hostname api.$KUADRANT_ZONE_ROOT_DOMAIN, where it exposes the endpoints GET /toys* and POST /toys, respectively, to mimic operations of reading and writing toy records.
-
We will rate limit the POST /toys endpoint to a maximum of 5rp10s ("5 requests every 10 seconds").
Create a HTTPRoute to route traffic to the services via the Gateways:
-
-
forcontextinkind-mgc-workload-1kind-mgc-workload-2;dokubectl--context$contextapply-f-<<EOF
-apiVersion: gateway.networking.k8s.io/v1
-kind: HTTPRoute
-metadata:
- name: toystore
-spec:
- parentRefs:
- - kind: Gateway
- name: prod-web
- namespace: kuadrant-multi-cluster-gateways
- hostnames:
- - toystore.$KUADRANT_ZONE_ROOT_DOMAIN
- rules:
- - matches:
- - method: GET
- path:
- type: PathPrefix
- value: "/toys"
- backendRefs:
- - name: toystore
- port: 80
- - matches: # it has to be a separate HTTPRouteRule so we do not rate limit other endpoints
- - method: POST
- path:
- type: Exact
- value: "/toys"
- backendRefs:
- - name: toystore
- port: 80
-EOF
-done
-
-
Verify the routes work:
-
curl-ikhttps://toystore.$KUADRANT_ZONE_ROOT_DOMAIN/toys
-# HTTP/1.1 200 OK
-
-
Given the two clusters, and our previously created DNSPolicy, traffic will load balance between these clusters round-robin style. Load balancing here will be determined in part by DNS TTLs, so it can take a minute or two for requests to flow to both services.
-
② Enforce rate limiting on requests to the Toy Store API¶
-
Create a Kuadrant RateLimitPolicy to configure rate limiting:
## Installation and Setup
-1. Clone this repo locally
-1. Setup a ./controller-config.env file in the root of the repo with the following key values
-
```bash
-# this sets up your default managed zone
-AWS_DNS_PUBLIC_ZONE_ID=<AWS ZONE ID>
-# this is the domain at the root of your zone (foo.example.com)
-ZONE_ROOT_DOMAIN=<replace.this>
-LOG_LEVEL=1
-```
-
-
-
-
Setup a ./aws-credentials.env with credentials to access route 53
This guide will show you how to install and configure the Multi-Cluster Gateway Controller in pre-existing Open Cluster Management configured clusters.
Cert-manager first needs to be installed on your hub cluster. If this has not previously been installed on the cluster, see the documentation for installation instructions here.
customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io condition met
-customresourcedefinition.apiextensions.k8s.io/gateways.gateway.networking.k8s.io condition met
-customresourcedefinition.apiextensions.k8s.io/httproutes.gateway.networking.k8s.io condition met
-
-
Then run the following command to install the MGC:
Note: To manage the creation of DNS records, MGC uses ManagedZone resources. A ManagedZone can be configured to use DNS Zones on both AWS (Route53), and GCP (Cloud DNS). Commands to create each are provided below.
NAME DOMAIN NAME ID RECORD COUNT NAMESERVERS READY
-mgc-dev-mz ef.hcpapps.net /hostedzone/Z06419551EM30QQYMZN7F 2 ["ns-1547.awsdns-01.co.uk","ns-533.awsdns-02.net","ns-200.awsdns-25.com","ns-1369.awsdns-43.org"] True
-
Create a ClusterIssuer to be used with cert-manager. For simplicity, we will create a self-signed cert issuer here, but other issuers can also be configured.
The above command will install the ManagedClusterAddOn resource needed to install the Kuadrant addon into the namespace representing a spoke cluster, and install the Kuadrant data-plane components into the open-cluster-management-agent-addon namespace.
-
The Kuadrant addon will install:
-
-
Kuadrant Operator
-
Limitador (and its associated operator)
-
Authorino (and its associated operator)
-
-
For more details, see the Kuadrant components installed by the kuadrant-operator
The Kuadrant OCM (Open Cluster Management) Add-On depends on the Operator Lifecycle Manager (OLM)'s CatalogSource. By default, this is set to olm/operatorhubio-catalog.
-
In OpenShift environments, OLM comes pre-installed. However, it is configured to use the openshift-marketplace/community-operators CatalogSource by default, not the olm/operatorhubio-catalog.
-
To align the Kuadrant add-on with the OpenShift default CatalogSource, you can patch the add-on's CatalogSource configuration. Run the following command (note it needs to be run for each managed cluster where the add-on is installed):
Existing Istio installations and changing the default Istio Operator name¶
-
In the case where you have an existing Istio installation on a cluster, you may encounter an issue where the Kuadrant Operator expects Istio's Operator to be named istiocontrolplane.
-
The istioctl command saves the IstioOperator CR that was used to install Istio in a copy of the CR named installed-state.
-
To let the Kuadrant operator use this existing installation, set the following:
If you want to use a different channel with the ManagedClusterAddon to install the kuadrant operator. You can do so by overriding the channel with the follow annotation:
You should see the namespace kuadrant-system, and the following pods come up:
-* authorino-value
-* authorino-operator-value
-* kuadrant-operator-controller-manager-value
-* limitador-value
-* limitador-operator-controller-manager-value
Gateway admins, need a way to define the DNS policy for a gateway distributed across multiple clusters in order to control how much and which traffic reaches these gateways.
-Ideally we would allow them to express a strategy that they want to use without needing to get into the details of each provider and needing to create and maintain dns record structure and individual records for all the different gateways that may be within their infrastructure.
-
Use Cases
-
As a gateway admin, I want to be able to reduce latency for my users by routing traffic based on the GEO location of the client. I want this strategy to automatically expand and adjust as my gateway topology grows and changes.
-
As a gateway admin, I have a discount with a particular cloud provider and want to send more of my traffic to the gateways hosted in that providers infrastructure and as I add more gateways I want that balance to remain constant and evolve to include my new gateways.
Allow definition of a DNS load balancing strategy to decide how traffic should be weighted across multiple gateway instances from the central control plane.
Allow different DNS policies for different listeners. Although this may be something we look to support in the future, currently policy attachment does not allow for this type of targeting. This means a DNSPolicy is applied for the whole gateway currently.
-
Define how health checks should work, this will be part of a separate proposal
Provide a control plane DNSPolicy API that uses the idea of direct policy attachment from gateway API that allows a load balancing strategy to be applied to the DNS records structure for any managed listeners being served by the data plane instances of this gateway.
-The DNSPolicy also covers health checks that inform the DNS response but that is not covered in this document.
-
Below is a draft API for what we anticipate the DNSPolicy to look like
-
apiVersion:kuadrant.io/v1alpha1
-kind:DNSPolicy
-spec:
-targetRef:# defaults to gateway gvk and current namespace
-name:gateway-name
-health:
-...
-loadBalancing:
-weighted:
-defaultWeight:10
-custom:#optional
--value:AWS#optional with both GEO and weighted. With GEO the custom weight is applied to gateways within a Geographic region
-weight:10
--value:GCP
-weight:20
-GEO:#optional
-defaultGeo:IE# required with GEO. Chooses a default DNS response when no particular response is defined for a request from an unknown GEO.
-
GEO and Weighted load balancing are well understood strategies and this API effectively allow a complex requirement to be expressed relatively simply and executed by the gateway controller in the chosen DNS provider. Our default policy will execute a "Round Robin" weighted strategy which reflects the current default behaviour.
-
With the above API we can provide weighted and GEO and weighted within a GEO. A weighted strategy with a minimum of a default weight is always required and the simplest type of policy. The multi-cluster gateway controller will set up a default policy when a gateway is discovered (shown below). This policy can be replaced or modified by the user. A weighted strategy can be complimented with a GEO strategy IE they can be used together in order to provide a GEO and weighted (within a GEO) load balancing. By defining a GEO section, you are indicating that you want to use a GEO based strategy (how this works is covered below).
-
apiVersion:kuadrant.io/v1alpha1
-kind:DNSPolicy
-name:default-policy
-spec:
-targetRef:# defaults to gateway gvk and current namespace
-name:gateway-name
-loadBalancing:
-weighted:# required
-defaultWeight:10#required, all records created get this weight
-health:
-...
-
-
In order to provide GEO based DNS and allow customisation of the weighting, we need some additional information to be provided by the gateway / cluster admin about where this gateway has been placed. For example if they want to use GEO based DNS as a strategy, we need to know what GEO identifier(s) to use for each record we create and a default GEO to use as a catch-all. Also, if the desired load balancing approach is to provide custom weighting and no longer simply use Round Robin, we will need a way to identify which records to apply that custom weighting to based on the clusters the gateway is placed on.
-
To solve this we will allow two new attributes to be added to the ManagedCluster resource as labels:
These two labels allow setting values in the DNSPolicy that will be reflected into DNS records for gateways placed on that cluster depending on the strategies used. (see the first DNSPolicy definition above to see how these values are used) or take a look at the examples at the bottom.
The attributes provide the key and value we need in order to understand how to define records for a given LB address based on the DNSPolicy targeting the gateway.
-
The kuadrant.io/lb-attribute-geo-code attribute value is provider specific, using an invalid code will result in an error status condition in the DNSrecord resource.
Custom weighting will use the associated custom-weight attribute set on the ManagedCluster to decide which records should get a specific weight. The value of this attribute is up to the end user.
The above is then used in the DNSPolicy to set custom weights for the records associated with the target gateway.
-
-value:GCP
-weight:20
-
-
So any gateway targeted by a DNSPolicy with the above definition that is placed on a ManagedCluster with the kuadrant.io/lb-attribute-custom-weight set with a value of GCP will get an A record with a weight of 20
DNSPolicy should have a ready condition that reflect that the DNSRecords have been created and configured as expected. In the case that there is an invalid policy, the status message should reflect this and indicate to the user that the old DNS has been preserved.
-
We will also want to add a status condition to the gateway status indicating it is effected by this policy. Gateway API recommends the following status condition
apiVersion:kuadrant.io/v1alpha1
-kind:DNSPolicy
-name:SendMoreToAzure
-spec:
-targetRef:# defaults to gateway gvk and current namespace
-name:gateway-name
-loadBalancing:
-weighted:
-defaultWeight:10
-custom:
--attribute:cloud
-value:Azure#any record associated with a gateway on a cluster without this value gets the default
-weight:30
-
DNSRecord is our API for expressing DNS endpoints via a kube CRD based API. It is managed by the multi-cluster gateway controller based on the desired state expressed in higher level APIs such as the Gateway or a DNSPolicy. In order to provide our feature set, we need to carefully consider how we structure our records and the types of records we need. This document proposes a particular structure based on the requirements and feature set we have.
We want to be able to support Gateway definitions that use the following listener definitions:
-
-
wildcard: *.example.com and fully qualified listener host www.example.com definitions with the notable exception of fully wildcarded ie * as we cannot provide any DNS or TLS for something with no defined hostname.
-
listeners that have HTTPRoute defined on less than all the clusters where the listener is available. IE we don't want to send traffic to clusters where there is no HTTPRoute attached to the listener.
-
Gateway instances that provide IPs that are deployed alongside instances on different infra that provide host names causing the addresses types on each of gateway instance to be different (IPAddress or HostAddress).
-
We want to provide GEO based DNS as a feature of DNSPolicy and so our DNSRecord structure must support this.
-
We want to offer default weighted and custom weighted DNS as part of DNSPolicy
-
We want to allow root or apex domain to be used as listener hosts
For each listener defined in a gateway, we will create a set of records with the following rules.
-
none apex domain:
-
We will have a generated lb (load balancer) dns name that we will use as a CNAME for the listener hostname. This DNS name is not intended for use within a HTTPRoute but is instead just a DNS construct. This will allow us to set up additional CNAME records for that DNS name in the future that are returned based a GEO location. These DNS records will also be CNAMES pointing to specific gateway dns names, this will allow us to setup a weighted response. So the first layer CNAME handles balancing based on geo, the second layer handles balancing based on weighting.
-
shop.example.com
- | |
- (IE) (AUS)
- CNAME lb.shop.. lb.shop..
- | | | |
- (w 100) (w 200) (w 100) (w100)
- CNAME g1.lb.. g2.lb.. g3.lb.. g4.lb..
- A 192.. A 81.. CNAME aws.lb A 82..
-
-
When there is no geo strategy defined within the DNSPolicy, we will put everything into a default geo (IE a catch-all record) default.lb-{guid}.{listenerHost} but set the routing policy to GEO allowing us to add more geo based records in the future if the gateway admin decides to move to a geo strategy as their needs grow.
-
To ensure this lb dns name is unique and does not clash we will use a short guid as part of the subdomain so lb-{guid}.{listenerHost}. this guid will be based on the gateway name and gateway namespace in the control plane.
-
For a geo strategy we will add a geo record with a prefix to the lb subdomain based on the geo code. When there is no geo we will use default as the prefix. {geo-code}.lb-{guid}.{listenerHost}.
-Finally, for each gateway instance on a target cluster we will add a {spokeClusterName}.lb-{guid}.{listenerHost}
-
To allow for a mix of hostname and IP address types, we will always use a CNAME . So we will create a dns name for IPAddress with the following structure: {guid}.lb-{guid}.{listenerHost} where the first guid will be based on the cluster name where the gateway is placed.
An apex domain is the domain at the apex or root of a zone. These are handled differently by DNS as they often have NS and SOA records. Generally it is not possible to set up a CNAME for apex domain (although some providers allow it).
-
If a listener is added to a gateway that is an apex domain, we can only add A records for that domain to keep ourselves compliant with as many providers as possible.
-If a listener is the apex domain, we will setup A records for that domain (favouring gateways with an IP address or resolving the IP behind a host) but there will be no special balancing/weighting done. Instead, we will expect that the owner of that will setup a HTTPRoute with a 301 permanent redirect sending users from the apex domain e.g. example.com to something like: www.example.com where the www subdomain based listener would use the rules of the none apex domains and be where advanced geo and weighted strategies are applied.
-
-
gateway listener host name : example.com
-
example.com A 81.17.241.20
-
-
-
-
Geo Agnostic (everything is in a default * geo catch all)¶
-
This is the type of DNS Record structure that would back our default DNSPolicy.
So in the above example working up from the bottom, we have a mix of hostname and IP based addresses for the gateway instance. We have 2 evenly weighted records that balance between the two available gateways, then next we have the geo based record that is set to a default catch all as no geo has been specified then finally we have the actual listener hostname that points at our DNS based load balancer name.
Once the end user selects to use a geo strategy via the DNSPolicy, we then need to restructure our DNS to add in our geo specific records. Here the default record
-
lb short code is {gw name + gw namespace}
-gw short code is {cluster name}
In the above example we move from a default catch all to geo specific setup. Based on a DNSPolicy that specifies IE as the default geo location. We leave the default subdomain in place to allow for clients that may still be using that and set up geo specific subdomains that allow us to route traffic based on its origin. In this example we are load balancing across 2 geos and 4 clusters
With this CNAME based approach we are increasing the number of DNS lookups required to get to an IP which will increase the cost and add a small amount of latency. To counteract this, we will set a reasonably high TTL (at least 5 mins) for our CNAMES and (2 mins) for A records
AWS supports Weighted(Weighted Round Robin) and Geolocation routing policies https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy.html. Both of these can be configured directly on records in AWS route 53.
Azure supports Weighted and Geolocation routing policies, but requires records to alias to a Traffic Manager resource that must also be created in the users account https://learn.microsoft.com/en-us/azure/traffic-manager/traffic-manager-routing-methods
-
Notes:
-
-
A Traffic Manager Profile is created per record set and is created with a routing method (Weighted or Geographic) https://portal.azure.com/#view/Microsoft_Azure_Network/LoadBalancingHubMenuBlade/~/TrafficManagers
-
Only a singe IP can be added to a DNSRecord set. A traffic manager profile must be created and aliased from a DNSRecord set for anything that involves more than a single target.
-
Significantly more resources to manage in order to achieve functionality comparable with Google and AWS.
-
The modelling of the records is significantly different from AWS Route53, but the current DNSRecord spec could still work. The azure implementation will have to process the endpoint list and create traffic manager policies as required to satisfy the record set.
-
-
Given the example DNSRecord here describing a record set for a geo location routing policy with four clusters, two in two regions (North America and Europe), the following Azure resources are required.
-
Three DNSRecords, each aliased to a different traffic manager:
-
-
-
dnsrecord-geo-azure-hcpapps-net (dnsrecord-geo.azure.hcpapps.net) aliased to Traffic Manager Profile 1 (dnsrecord-geo-azure-hcpapps-net)
-
dnsrecord-geo-na.azure-hcpapps-net (dnsrecord-geo.na.azure.hcpapps.net) aliased to Traffic Manager Profile 2 (dnsrecord-geo-na-azure-hcpapps-net)
-
dnsrecord-geo-eu.azure-hcpapps-net (dnsrecord-geo.eu.azure.hcpapps.net) aliased to Traffic Manager Profile 3 (dnsrecord-geo-eu-azure-hcpapps-net)
-
-
Three Traffic Manager Profiles:
-
-
-
Traffic Manager Profile 1 (dnsrecord-geo-azure-hcpapps-net): Geolocation routing policy with two region specific FQDN targets (dnsrecord-geo.eu.azure.hcpapps.net and dnsrecord-geo.na.azure.hcpapps.net).
-
Traffic Manager Profile 2 (dnsrecord-geo-na-azure-hcpapps-net): Weighted routed policy with two IP address endpoints (172.31.0.1 and 172.31.0.2) with equal weighting.
-
Traffic Manager Profile 3 (dnsrecord-geo-eu-azure-hcpapps-net): Weighted routed policy with two IP address endpoints (172.31.0.3 and 172.31.0.4) with equal weighting.
Google supports Weighted(Weighted Round Robin) and Geolocation routing policies https://cloud.google.com/dns/docs/zones/manage-routing-policies. Both of these can be configured directly on records in Google Cloud DNS and no secondary Traffic Management resource is required.
-
Notes:
-
-
Record sets are modelled as a single endpoint with routing policy embedded. This is a different approach to Route53 where each individual A/CNAME would have its own record entry.
-
Weight must be an integer between 0 - 10000
-
There are no continent options for region, only finer grained regions such as us-east1, europe-west-1 etc...
-
There appears to be no way to set a default region, google just routes requests to the nearest supported region.
-
The current approach used in AWS Route53 for geo routing will work in the same way on Google DNS. A single CNAME record with geo routing policy specifying multiple geo specific A record entries as targets.
-
Geo and weighted routing can be combined, as with AWS Route53, allowing traffic with a region to be routed using weightings.
-
The modelling of the records is slightly different from AWS, but the current DNSRecord spec could still work. The Google implementation of AddRecords will have to process the list of endpoints in order to group related endpoints in order to build up the required API request.
-In this case there would not be a 1:1 mapping between an endpoint in a DNSRecord and the dns provider, but the DNSRecord contents would be kept consistent across all providers and compatibility with external-dns would be maintained.
-
-
Example request for Geo CNAME record:
-
POST https://dns.googleapis.com/dns/v1beta2/projects/it-cloud-gcp-rd-midd-san/managedZones/google-hcpapps-net/rrsets
-
Given the example DNSRecord here describing a record set for a geo location routing policy with four clusters, two in two regions (North America and Europe), the following resources are required.
-
Three DNSRecords, one CNAME (dnsrecord-geo.google.hcpapps.net) and 2 A records (dnsrecord-geo.na.google.hcpapps.net and dnsrecord-geo.eu.google.hcpapps.net)
This directory contains proposals accepted into the MGC. The template for add a proposal is located in this directory. Make a copy of the template and use it to define your own proposal.
Currently, MGC only supports AWS Route53 as a dns provider. A single instance of a DNSProvider resource is created per MGC instance which is configured with AWS config loaded from the environment.
-This provider is loaded into all controllers requiring dns access (ManagedZone and DNSRecord reconciliations), allowing a single instance of MGC to operate against a single account on a single dns provider.
MGC has three features it requires of any DNS provider in order to offer full support, DNSRecord management, Zone management and DNS Health checks. We do not however want to limit to providers that only offer this functionality, so to add support for a provider the minimum that provider should offer is API access to managed DNS records.
-MGC will continue to provide Zone management and DNS Health checks support on a per-provider basis.
-
Support will be added for AWS(Route53), Google(Google Cloud DNS), Azure and investigation into possible adding CoreDNS (intended for local dev purposes), with the following proposed initial support:
-
-
-
-
Provider
-
DNS Records
-
DNS Zones
-
DNS Health
-
-
-
-
-
AWS Route53
-
X
-
X
-
X
-
-
-
Google Cloud DNS
-
X
-
X
-
-
-
-
-
AzureDNS
-
X
-
X
-
-
-
-
-
CoreDNS
-
X
-
-
-
-
-
-
-
-
Add DNSProvider as an API for MGC which contains all the required config for that particular provider including the credentials. This can be thought of in a similar way to a cert manager Issuer.
-Update ManagedZone to add a reference to a DNSProvider. This will be a required field on the ManagedZone and a DNSProvider must exist before a ManagedZone can be created.
-Update all controllers load the DNSProvider directly from the ManagedZone during reconciliation loops and remove the single controller wide instance.
-Add new provider implementations for google, azure and coredns.
- * All providers constructors should accept a single struct containing all required config for that particular provider.
- * Providers must be configured from credentials passed in the config and not rely on environment variables.
Investigation was carried out into the suitability of [External DNS] (https://github.com/kubernetes-sigs/external-dns) as the sole means of managing dns resources.
-Unfortunately, while external dns does offer support for basic dns record management with a wide range of providers, there were too many features missing making it unsuitable at this time for integration.
Run external dns, as intended, as a separate controller alongside mgc, and pass all responsibility for reconciling DNSRecord resources to it. All DNSRecord reconciliation is removed from MGC.
-
Issues:
-
-
A single instance of external dns will only work with a single provider and a single set of credentials. As it is, in order to support more than a single provider, more than one external dns instance would need to be created, one for each provider/account pair.
-
Geo and Weighted routing policies are not implemented for any provider other than AWS Route53.
-
Only supports basic dns record management (A,CNAME, NS records etc ..), with no support for managed zones or health checks.
Add external dns as a module dependency in order to make use of their DNS Providers, but continue to reconcile DNSRecords in MGC.
-
Issues:
-
-
External DNS Providers all create clients using the current environment. Would require extensive refactoring in order to modify each provider to optionally be constructed using static credentials.
-
Clients were all internal making it impossible, without modification, to use the upstream code to extend the provider behaviour to support additional functionality such as managed zone creation.
The MGC has the ability to extend the DNS configuration of the gateway with the DNSPolicy resource. This resource allows
-users to configure health checks. As a result of configuring health checks, the controller creates the health checks in
-Route53, attaching them to the related DNS records. This has the benefit of automatically disabling an endpoint if it
-becomes unhealthy, and enabling it again when it becomes healthy again.
-
This feature has a few shortfalls:
-1. It’s tightly coupled with Route53. If other DNS providers are supported they must either provide a similar feature,
-or health checks will not be supported
-2. Lacks the ability to reach endpoints in private networks
-3. requires using the gateway controller to implement, maintain and test multiple providers
-
This document describes a proposal to extend the current health check implementation to overcome these shortfalls.
As a gateway administrator, I would like to define a health check that each service sitting behind a particular
-listener across the production clusters has to implement to ensure we can automatically respond, failover and
-mitigate a failing instance of the service
Currently, this functionality will be added to the existing MGC, and executed within that component. This will be created
-with the knowledge that it may need to be made into an external component in the future.
The presence of the healthCheck means that for every DNS endpoint (that is either an A record, or a CNAME to an external host),
-a health check is created based on the health check configuration in the DNSPolicy.
-
A failureThreshold field will be added to the health spec, allowing users to configure a number of consecutive health
-check failures that must be observed before the endpoint is considered unhealthy.
The DNSHealthCheckProbe resource configures a health probe in the controller to perform the health checks against an
-identified final A or CNAME endpoint. When created by the controller as a result of a DNS Policy, this will have an
-owner ref of the DNS Policy that caused it to be created.
Address The address to connect to (e.g. IP address or hostname of a clusters loadbalancer)
-
Host The host to request in the Host header
-
Path The path to request
-
Protocol The protocol to use for this request
-
Interval How frequently this check would ideally be executed.
-
AdditionalHeaders Optional secret ref which contains k/v: headers and their values that can be specified to ensure the health check is successful.
-
ExpectedResponses Optional HTTP response codes that should be considered healthy (defaults are 200 and 201).
-
AllowInsecureCertificate Optional flag to allow using invalid (e.g. self-signed) certificates, default is false.
-
-
The reconciliation of this resource results in the configuration of a health probe, which targets the endpoint and
-updates the status. The status is propagated to the providerSpecific status of the equivalent endpoint in the DNSRecord
Currently, the reconciliation loop of this controller creates health checks in the configured DNS provider
-(Route53 currently) based on the spec of the DNSPolicy, separately from the reconciliation of the DNSRecords.
-The proposed change is to reconcile health check probe CRs based on the combination of DNS Records and DNS Policies.
-
Instead of Route53 health checks, the controller will create DNSHealthCheckProbe resources.
When reconciling a DNS Record, the DNS Record reconciler will retrieve the relevant DNSHealthCheckProbe CRs, and consult
-the status of them when determining what value to assign to a particular endpoint's weight.
When a DNS health check probe is failing, it will update the DNS Record CR with a custom field on that endpoint to mark it as failing.
-
There are then 3 scenarios which we need to consider:
-1 - All endpoints are healthy
-2 - All endpoints are unhealthy
-3 - Some endpoints are healthy and some are unhealthy.
-
In the cases 1 and 2, the result should be the same: All records are published to the DNS Provider.
-
When scenario 3 is encountered the following process should be followed:
-
For each gateway IP or CNAME: this should be omitted if unhealthy.
-For each managed gateway CNAME: This should be omitted if all child records are unhealthy.
-For each GEO CNAME: This should be omitted if all the managed gateway CNAMEs have been omitted.
-Load balancer CNAME: This should never be omitted.
-
-
If we consider the DNS record to be a hierarchy of parents and children, then whenever any parent has no healthy
-children that parent is also considered unhealthy. No unhealthy elements are to be included in the DNS Record.
When removing DNS records, we will want to avoid any NXDOMAIN responses from the DNS service as this will cause the
-resolver to cache this missed domain for a while (30 minutes or more). The NXDOMAIN response is triggered when the
-resolver attempts to resolve a host that does not have any records in the zone file.
-
The situation that would cause this to occur is when we have removed a record but still refer to it from other
-records.
-
As we wish to avoid any NXDOMAIN responses from the nameserver - causing the resolver to cache this missed response
-we will need to ensure that any time a DNS Record (CNAME or A) is removed, we also remove any records that refer to the
-removed record. (e.g. when the gateway A record is removed, we will need to remove the managed gateway CNAME that
-refers to that A record).
There will be a DNSHealthCheckProbe CR controller added to the controller. This controller will create an instance of a
-HealthMonitor, the HealthMonitor ensures that each DNSHealthCheckProbe CR has a matching probeQueuer object running.
-It will also handle both the updating of the probeQueuer on CR update and the removal of probeQueuers, when a
-DNSHealthcheckProbe is removed.
-
The ProbeQueuer will add a health check request to a queue based on a configured interval, this queue is consumed by a
-ProbeWorker, probeQueuers work on their own goroutine.
-
The ProbeWorker is responsible for actually executing the probe, and updating the DNSHealthCheckProbe CR status. The
-probeWorker executes on its own goroutine.
Status conditions are used to represent the current state of a resource and provide information about any problems or issues that might be affecting it. They are defined as an array of Condition objects within the status section of a resource's YAML definition.
When multiple instances of a resource (e.g. a Gateway) are running across multiple clusters, it can be difficult to know the current state of each instance without checking each one individually. This can be time-consuming and error-prone, especially when there are a large number of clusters or resources.
To solve this problem, I'm proposing we leverage the status block in the control plane instance of that resource, aggregating the statuses to convey the necessary information.
For example, if the Ready status condition type of a Gateway is True for all instances of the Gateway resource across all clusters, then the Gateway in the control plane will have the Ready status condition type also set to True.
-
status:
-conditions:
--type:Ready
-status:True
-message:All listeners are valid
-
-
If the Ready status condition type of some instances is notTrue, the Ready status condition type of the Gateway in the control plane will be False.
-
status:
-conditions:
--type:Ready
-status:False
-
-
In addition, if the Ready status condition type is False, the Gateway in the control plane should include a status message for each Gateway instance where Ready is False. This message would indicate the reason why the condition is not true for each Gateway.
In this example, the Ready status condition type is False because two of the three Gateway instances (gateway-1 and gateway-3) have issues with their listeners. For gateway-1, the reason for the False condition is that the listener certificate is expired, and for gateway-3, the reason is that no listener is configured for port 80. These reasons are included as status messages in the Gateway resource in the control plane.
-
As there may be different reasons for the condition being False across different clusters, it doesn't make sense to aggregate the reason field. The reason field is intended to be a programmatic identifier, while the message field allows for a human readable message i.e. a semi-colon separated list of messages.
-
The lastTransitionTime and observedGeneration fields will behave as normal for the resource in the control plane.
The Gateway status can include information about addresses, like load balancer IP Addresses assigned to the Gateway,
-and listeners, such as the number of attached routes for each listener.
-This information is useful at the control plane level.
-For example, a DNS Record should only exist as long as there is at least 1 attached route for a listener.
-It can also be more complicated than that when it comes to multi cluster gateways.
-A DNS Record should only include the IP Addresses of the Gateway instances where the listener has at least 1 attached route.
-This is important when initial setup of DNS Records happen as applications start.
-It doesn't make sense to route traffic to a Gateway where a listener isn't ready/attached yet.
-It also comes into play when a Gateway is displaced either due to changing placement decision or removal.
-
In summary, the IP Addresses and number of attached routes per listener per Gateway instance is needed in the control plane to manage DNS effectively.
-This proposal adds that information the hub Gateway status block.
-This will ensure a decoupling of the DNS logic from the underlying resource/status syncing implementation (i.e. ManifestWork status feedback rules)
-
First, here are 2 instances of a multi cluster Gateway in 2 separate spoke clusters.
-The yaml is shortened to highlight the status block.
The MultiCluster Gateway Controller will use a custom implementation of the addresses and listenerers fields.
-The address type is of type AddressType, where the type is a domain-prefixed string identifier.
-The value can be split on the forward slash, /, to give the cluster name and the underlying Gateway IPAddress value of type IPAddress.
-Both the IPAddress and Hostname types will be supported.
-The type strings for either will be kuadrant.io/MultiClusterIPAddress and kuadrant.io/MultiClusterHostname
-
The listener name is of type SectionName, with validation on allowed characters and max length of 253.
-The name can be split on the period, ., to give the cluster name and the underlying listener name.
-As there are limits on the character length for the name field, this puts a lower limit restriction on the cluster names and listener names used to ensure proper operation of this status aggregation.
-If the validation fails, a status condition showing a validation error should be included in the hub Gateway status block.
How to create a MGC OLM bundle, catalog and how to install MGC via OLM¶
-
NOTE: You can supply different env vars to the following make commands these include:
-
* Version using the env var VERSION
-* Tag via the env var IMAGE_TAG for tags not following the semantic format.
-* Image registry via the env var REGISTRY
-* Registry org via the env var ORG
-
-For example
-
-
make bundle-build-push VERISON=2.0.1
- make catalog-build-push IMAGE_TAG=asdf
Create a operator group
-bash
- cat <<EOF | kubectl apply -f -
-apiVersion: operators.coreos.com/v1
-kind: OperatorGroup
-metadata:
- name: og-mgc
- namespace: multi-cluster-gateways-system
-EOF
- For more information on each of these OLM resources please see the offical docs
The multi-cluster gateway controller, leverages the gateway API standard and Open Cluster Management to provide multi-cluster connectivity and global load balancing
-
Key Features:
-
-
Central Gateway Definition that can then be distributed to multiple clusters
-
Automatic TLS and cert distribution for HTTPS based listeners
-
DNSPolicy to decide how North-South based traffic should be balanced and reach the gateways
-
Health checks to detect and take remedial action against unhealthy endpoints
-
Cloud DNS provider integrations (AWS route 53) with new ones being added (google DNS)
-
-
When deploying the multicluster gateway controller using the make targets, the following will be created:
-* Kind cluster(s)
-* Gateway API CRDs in the control plane cluster
-* Ingress controller
-* Cert manager
-* LetsEncrypt certs
Various dependencies installed into $(pwd)/bin e.g. kind, yq etc.
-
Run make dependencies
-
openssl>=3
-
On macOS a later version is available with brew install openssl. You'll need to update your PATH as macOS provides an older version via libressl as well
Build the controller image and load it into the control plane
- sh
- kubectl config use-context kind-mgc-control-plane
- make kind-load-gateway-controller
-
-
-
Deploy the controller(s) to the control plane cluster
-
makedeploy-gateway-controller
-
-
-
-
(Optional) View the logs of the deployed controller
-
Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
http://www.apache.org/licenses/LICENSE-2.0
-
-
Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
It may take a few minutes for the updated RateLimitPolicy to be configured with the modified rate limit.
-
-
-
-
