Read value from Bearer Token and add as RequestHeader for RateLimit

[creib]

Hi,
we are trying to implement the following use-case, where the information on how a user should be rate limited is part of the Bearer Token.

We are using the ExternalAuth Security Policy to implement our own logic for authorization. In this step we are also extracting the value for rate limiting for this user.
The idea was, that we add the rate limit value as a response header to the response of the external auth endpoint.
Then we wanted to add a new request header with this value to the initial request.

Is this approach working at all? Is the auth security policy executed before the rate limiting? How would we be able to use the value from the auth response as a value for a new request header?

Thanks a lot in advance.

[creib]

We where able to add the request header by using the %REQ('Headername')% now.
But now we have the problem, that the rate limiting seems to be evaluated before we add the header. Where can we find a execution order of the policies and even better, is it possible to modify this order?

Another question is the following. We wanted to check the generated nginx configuration on the pod (we are deploying using the helm chart) but we are not able to connect to the pod. Is there a way to provide a value to the helm chart which enables us to debug in such a manner?

[arkodg]

the data plane is based on envoyproxy, you can check out the config using egctl config envoy-proxy more in https://gateway.envoyproxy.io/docs/install/install-egctl/
https://gateway.envoyproxy.io/docs/tasks/operations/egctl/

[arkodg]

there's a lot of things being discussed here, so I'll add some links here which you'll find helpful

• The default filter order is outlined here https://gateway.envoyproxy.io/docs/tasks/operations/customize-envoyproxy/#customize-filter-order where ext_authz is executed before ratelimit, which is what you desire, so you dont need to change this, but if you ever want to, you can, using this field in the EnvoyProxy resource
• make sure to set headersToBackend with the header you are using for ratelimiting if you are using the HTTP variant of ext auth https://gateway.envoyproxy.io/docs/api/extension_types/#httpextauthservice
• if you are directly retrieving the value to ratelimit on from a claim, you'll find this useful https://gateway.envoyproxy.io/docs/tasks/traffic/global-rate-limit/#rate-limit-jwt-claims
• if you want to add headers before any filter reads them, check out the early header feature https://gateway.envoyproxy.io/docs/tasks/traffic/http-request-headers/#early-header-modification

[gid-fieldcode]

@arkodg creib and I are colleagues, working on the same issue for several weeks now. As you may remember, we have recently discussed our case, where we need to rate-limit based on a JWT claim. We can't use the security policy approach because we have 50+ tenants and therefore we need as many providers and Huabing Zao has told us that there's a limit of 4 providers. So we have implemented External Auth, where the ExtAuth service not only returns the validation result, but also returns the JWT claim as a header. Now, using v1.2.1 and EarlyHeaderModification, we have added this required header into the original request. However, when we submit the client request to our Envoy URL, even though we do see the new header in the request, we see no rate-limit activity. It's as though rate-limiting decisions have been already made, and therefore "retro-adding" the required header made no difference. Which surprises us because in the filter sequence we have seen that rate-limiting takes place just before routing, i.e. practically last in sequence. What would you suggest here?

Also as creib mentioned, we cannot open a shell into the three pods (main envoy-gateway, pod-for-our-own-gateway and envoy-ratelimit), and therefore we cannot see the code you are using inside NginX so we can perhaps understand it and figure out a solution. Is there something we can do to enable this kind of access?

Thanks for your help!

[arkodg]

However, when we submit the client request to our Envoy URL, even though we do see the new header in the request, we see no rate-limit activity

can share your SecurityPolicy, BackendTrafficPolicy, ClientTrafficPolicy, HTPRoute (and remove any sensitive info)

[gid-fieldcode]

Sure. For reference, x-extracted-erp is the header that is returned from the ExtAuth service, and we take it and early-add it to the initial request as x-acme-erp, so that we can then rate-limit based on its value. Which fails because it seems it's too late to affect any rate-limiting decisions.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: ext-auth-example
  namespace: test
spec:
  extAuth:
    failOpen: false
    http:
      backendRefs:
      - group: ""
        kind: Service
        name: keycloak-proxy
        namespace: keycloak
        port: 4901
      headersToBackend:
      - x-current-user
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: time-zone-gsl

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: time-zone-gsl-httproute
  namespace: test
spec:
  hostnames:
  - '*.acme.club'
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: apps
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: time-zone-gsl
      port: 4900
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: /
          type: ReplacePrefixMatch
    - requestHeaderModifier:
        set:
        - name: X-Forwarded-Proto
          value: https
        - name: X-Forwarded-Port
          value: "443"
        - name: X-Forwarded-Host
          value: '%REQ(Host)%'
      type: RequestHeaderModifier
    matches:
    - path:
        type: PathPrefix
        value: /api/public/time-zone-gsl

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: policy-time-zone-gsl
  namespace: test
spec:
  rateLimit:
    global:
      rules:
      - clientSelectors:
        - headers:
          - invert: false
            name: x-acme-erp
            type: Distinct
        limit:
          requests: 3
          unit: Minute
    type: Global
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: time-zone-gsl-httproute

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: enable-early-headers
  namespace: test
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: apps
  enableProxyProtocol: true
  headers:
    earlyRequestHeaders:
      add:
        - name: "x-acme-erp"
          value: "%REQ(x-extracted-erp)%"

[arkodg]

    earlyRequestHeaders:
      add:
        - name: "x-acme-erp"
          value: "%REQ(x-extracted-erp)%"

won't work as you expect it to because this logic is being executed before ext auth is kicking in

[creib]

Thanks a lot. If we use the default RequestHeader (https://gateway.envoyproxy.io/docs/tasks/traffic/http-request-headers/#adding-request-headers) it seems to be too late, basically after the ratelimit is applied.
Any suggestion to workaround? Should we use a lua filter, will it be executed at the right place between ext auth and ratelimit?

[arkodg]

if your ext auth is adding a new header, you dont really need to explicitly re-add it in the config, you can directly using x-extracted-erp in ratelimit

[arkodg]

you can test it by manually injecting it with curl to make sure ratelimit works, and then try adding it via ext auth

[arkodg]

in your config I see another mismatch

      headersToBackend:
      - x-current-user

shouldnt this be x-extracted-erp ?

[gid-fieldcode]

HI @arkodg, thanks again for your help. Here is where we stand:

With our current setup, when we submit the request with curl and manually add the critical header x-extracted-erp we get rate-limiting just fine.

When we submit the same request properly (without the manual header), we don't get rate-limiting, it is not triggered. The critical header is returned to us with the response, both as "x-extracted-erp" and as "x-acme-erp", though of course this is not our goal here. We tried also without the "early-header" trick, and saw that rate-limiting stopped working altogether, i.e. even with the manual header.

Here is the setup:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: time-zone-gsl-httproute-adhoc
  namespace: test
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: apps
  hostnames:
    - "*.acme.club"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: time-zone-gsl
          namespace: test
          port: 4900
          weight: 1          
      matches:
        - path:
            type: PathPrefix
            value: /api/public/time-zone-gsl/
      filters:
        - type: URLRewrite
          urlRewrite:
            path:
              type: ReplacePrefixMatch
              replacePrefixMatch: /
        - type: RequestHeaderModifier
          requestHeaderModifier:
            set:
              - name: X-Forwarded-Proto
                value: "https"                            
              - name: X-Forwarded-Port
                value: "443"          
              - name: X-Forwarded-Host
                value: "%REQ(Host)%"
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: ext-auth-example
  namespace: test
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: time-zone-gsl-httproute-adhoc
  extAuth:
    http:
      backendRefs:
        - name: keycloak-proxy
          port: 4901
          namespace: keycloak
      headersToBackend: ["x-extracted-erp"]
---	  
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: enable-early-headers
  namespace: test
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: apps
  enableProxyProtocol: true
  headers:
    earlyRequestHeaders:
      add:
        - name: "x-acme-erp"
          value: "%REQ(x-extracted-erp)%"
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy 
metadata:
  name: policy-time-zone-gsl
  namespace: test
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: time-zone-gsl-httproute-adhoc
  rateLimit:
    type: Global
    global:
      rules:
      - clientSelectors:
        - headers:
          - name: x-acme-erp
            type: Distinct    
          - name: x-extracted-erp
            type: Distinct    
        limit:
          requests: 3
          unit: Minute		  

Can you see a way out??

[arkodg]

I dont see x-acme-erp ever being set

    earlyRequestHeaders:
      add:
        - name: "x-acme-erp"
          value: "%REQ(x-extracted-erp)%"

is not going to work because x-extracted-erp doesnt exist yet , this logic kicks in before the filters are applied and x-extracted-erp is appended to the request by a filter (ext auth)

[gid-fieldcode]

How would you suggest we set it then?

[arkodg]

https://gateway.envoyproxy.io/docs/tasks/traffic/http-request-headers/#adding-request-headers
and the REQ formatter should work here as well

[gid-fieldcode]

We have tried the following on the httproute and it doesn't help. You probably mean something different....?

        - type: RequestHeaderModifier
          requestHeaderModifier:
            add:
              - name: "x-acme-erp"
                value: "%REQ(x-extracted-erp)%"

[arkodg]

thats weird, also what's the point of copying the value again into another header ?
isnt

      - clientSelectors:
        - headers: 
          - name: x-extracted-erp
            type: Distinct  

working ?

[gid-fieldcode]

Our idea was that we receive the x-extracted-erp header from the ext-auth service and then we add it to the request with the new name.

Are you suggesting we completely drop the x-acme-erp header?

Can you please tell us what YAML we should be using? I think it will be faster that way.

[gid-fieldcode]

We removed the EarlyHeaders trick and rate-limiting worked. Thanks for your help!