[Question] Deploying behind NGINX ingress controller

[pboers1988]

Hi All,

More a question. At this point in time I'm attempting to deploy the gnmi-server behind an NGINX-ingress inside kubernetes. I'm struggling to tweak the ingress in such a way that it works. When I do a port-forward to the gnmi-server I'm able to query the server with a client. However when I do the same query behind the ingress (TLS enabled) I get the following error:

2024/08/06 16:14:23.062224 /home/runner/work/gnmic/gnmic/app/logging.go:21: [gnmic] rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR
rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR
Error: one or more requests failed

The query I'm attempting is:

Behind the ingress - Error

gnmic -a <address>:443 sub --path "/components" --target <target> --mode once --debug

Port forward - Works

gnmic -a localhost:57400 sub --path "/components" --target <target> --mode once --debug

Ingress status

The ingress is configured correctly and works.

❯ k describe ingress -n streaming gnmic
Name:             gnmic
Labels:           app.kubernetes.io/instance=gnmic
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=gnmic
                  app.kubernetes.io/version=0.34.3
                  helm.sh/chart=gnmic-0.1.0
Namespace:        streaming
Address:          *****
Ingress Class:    nginx-production
Default backend:  <default>
TLS:
  tls-routers-secret terminates gnmi.routers.****
Rules:
  Host                   Path  Backends
  ----                   ----  --------
  gnmi.routers.**  /   gnmic-collector-gnmic-api:57400 (10.246.2.127:57400,10.246.4.115:57400,10.246.6.102:57400 + 1 more...)
Annotations:             cert-manager.io/issuer: letsencrypt
                         meta.helm.sh/release-name: gnmic
                         meta.helm.sh/release-namespace: streaming
                         nginx.ingress.kubernetes.io/backend-protocol: GRPC
                         nginx.ingress.kubernetes.io/service-upstream: true
                         nginx.ingress.kubernetes.io/whitelist-source-range: ******
Events:
  Type    Reason  Age                   From                      Message
  ----    ------  ----                  ----                      -------
  Normal  Sync    8m1s (x20 over 121m)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    8m1s (x20 over 121m)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    8m1s (x20 over 121m)  nginx-ingress-controller  Scheduled for sync

Certificate status

Name:         tls-routers-secret
Namespace:    streaming
Labels:       app.kubernetes.io/instance=gnmic
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=gnmic
              app.kubernetes.io/version=0.34.3
              helm.sh/chart=gnmic-0.1.0
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2024-08-06T12:29:26Z
  Generation:          1
  Owner References:
    API Version:           networking.k8s.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  gnmic
    UID:                   8c9485af-f5bf-4522-a99b-215fda9f331f
  Resource Version:        431199365
  UID:                     89da4cd4-f10f-4a37-9960-a9ac9dec5713
Spec:
  Dns Names:
    gnmi.*****
  Issuer Ref:
    Group:      cert-manager.io
    Kind:       Issuer
    Name:       letsencrypt
  Secret Name:  tls-routers-secret
  Usages:
    digital signature
    key encipherment
Status:
  Conditions:
    Last Transition Time:  2024-08-06T12:39:51Z
    Message:               Certificate is up to date and has not expired
    Observed Generation:   1
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2025-08-06T23:59:59Z
  Not Before:              2024-08-06T00:00:00Z
  Renewal Time:            2025-04-06T23:59:59Z
  Revision:                1
Events:                    <none>

The routers I'm attempting to query are very fast, results ususally return in as few ms so I shouldn't be hitting this timeout.

Has anyone had a similar experience? Thanks.

[hellt]

Could it be that your ingress terminates tls and sends http towards the server that expects tls? Or vice versa