GKE HTTP2

[hadim]

I use argo v3.0.1 on GKE (configured with an IAP proxy for auth).

The "user" <-> "GKE LB" communication uses HTTP2 and I would like to have the "LB" <-> "argo server" to use HTTP2 too. I have correctly configured the GKE backend service to use HTTP2 but enabling that prevent any more communication with argo (502 errors + no logs on the argo server-side).

As a sanity check, I connected manually (using Kubernetes proxy CLI) to the argo server service and when browsing to it I can see it's using HTTP1 (using Chrome Dev Tools). I guess this is the reason why the above setup is failing but I am confused because I think argo is configured to serve HTTP2 requests right?

The motivation to make it to work is to avoid this UI error (net::ERR_HTTP2_PROTOCOL_ERROR 200):

GET https://xxxxxxxx.com/api/v1/workflow-events/argo?listOptions.resourceVersion=94239078&fields=result.object.metadata.name,result.object.metadata.namespace,result.object.metadata.resourceVersion,result.object.metadata.creationTimestamp,result.object.metadata.uid,result.object.status.finishedAt,result.object.status.phase,result.object.status.message,result.object.status.startedAt,result.object.status.estimatedDuration,result.object.status.progress,result.type,result.object.metadata.labels,result.object.spec.suspend net::ERR_HTTP2_PROTOCOL_ERROR 200

Which does not prevent using the UI but is not ideal.

EDIT: this is the continuation of #4297 (comment)

[hadim]

In short, has anyone been able to enable full HTTP2 up to a GKE powered argo server (using IAP)? Full HTTPS (up to the argo server) already works perfectly.

[hadim]

It might also be related to #5523

[hadim]

The described error is also possibly related to #5124

[hadim]

I have done more tests in case it can help. I have followed the GKE documentation to set up HTTP2 in between argo and the LB (https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-http2).

When using the same configuration but replacing the Deployment with the one in the documentation (image: k8s.gcr.io/echoserver:1.10) instead of image: argoproj/argocli:v3.0.1), it works. The response from the echoheaders server confirms that it's using HTTP2. Same when calling the server not from the LB but from the Service directly (using a k8s proxy).

When switching back the Deployment to argo-server, calling it from the Service (using a k8s proxy), the request still uses HTTP/1.

So at this point, I think I have misconfigured something on the argo side because the full GKE setup is working for HTTP2 (unless I am missing something obvious here).

The argo server configuration is very close to the namespace-install one with:

          image = "argoproj/argocli:v3.0.1"
          name  = "argo-server"
          args = [
            "server",
            "--configmap",
            "workflow-controller-configmap",
            "--auth-mode",
            "server",
            "--namespaced",
            # "--verbose",
          ]

Also, the current argo server on GKE is already configured and uses HTTPS.

Let me know if I can do more tests.

[hadim]

It turns out the issue was due to the LB closing the eventsource connection but it looks like #5124 does not fix it (at least for the setup and provider I use).

The solution was too increase the GCP BackendConfig timeout from 30s to 3h:

apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: my-backendconfig
spec:
  timeoutSec: 10800  # 3h

In the end, it has nothing to do with HTTP2 and I think I was just confused because user <-> LB uses HTTP/2 but not the argo server.