Failures in integration tests against Kong nightly image

[randmonkey]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When running integration test cases requiring TLS certificates against the Kong nightly image (kong/kong:nightly), the certificate failed the TLS verification:

ingress_https_test.go:224: WARNING: error while waiting for https://foo.example:443/foo: Get "https://foo.example:443/foo": tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match foo.example

While the same tests passed for Kong 3.7, and also passed for tests with DB backed gateway.

Expected Behavior

The tests passes without certificate verification failures.

Steps To Reproduce

In the KIC repo, use the following command to run integration test case TestIngressHTTPS:
GOTESTFLAGS="-run=TestHTTPSIngress$" TEST_KONG_IMAGE="kong/kong" TEST_KONG_TAG="20240805-ubuntu" TEST_KONG_EFFECTIVE_VERSION="3.8.0" make test.integration.dbless

Kong Ingress Controller version

latest main, may also affect all KIC versions.

Kubernetes version

Not relevant

Anything else?

First observed in https://github.com/Kong/kubernetes-ingress-controller/actions/runs/10174112861 run in July 31st. The PR in Kong Kong/kong#13357 may be related.

[randmonkey]

Tested with the 2 images:
kong/kong:pr-13357-ubuntu failed (the image built after Kong/kong#13357 merged)
kong/kong:pr-13354-ubuntu passed (the image built after Kong/kong#13354 merged, one PR before the PR above)
So it is highly likely that the PR Kong/kong#13357 changed something that made the certificate fail in the tests.

[randmonkey]

Some new progress of re-producing: I tested with two images before and after the PR Kong/kong#13357.
The image before the PR kong/kong:264341db658ab9d0a17000ffb65bea7960348556 and the image after the PR kong/kong:cb8d04674319237f0d8070a6638ed9ac273052c3 behaves differently with SNIs attached to certificates.

TL;DR: New image has some problems in processing SNIs attached to certificate when they are represented as an attribute of certificates where they are attached to.

In the Kong config dumped from the old image, the SNIs are in a separate section and attached to certificates in their certificate field, like:

{
   "certificates":[{
        "cert": "----BEGIN CERTIFICATE-----....",
        "key": "-----BEGIN RSA PRIVATE KEY-----....",
        "id": "abcd1234"
    }],
    "snis":[{
      "name": "foo.example",
      "id": "fedc0987",
      "certificate":"abcd1234"
    }]
}

Where SNIs are listed in a separate section and attached to certificate by their certificate field.

While the config dump from the new image looks like

{
    "certificates":[{
        "cert": "----BEGIN CERTIFICATE-----....",
        "key": "-----BEGIN RSA PRIVATE KEY-----....",
        "id": "abcd1234",
        "snis":[{
            "name": "foo.example",
            "id": "fedc0987" 
        }]
    }]
}

Where SNIs are represented as an attribute of the certificate they are attached to.

Also, when we apply the config with the "attached" format to the old image, the certificate and SNIs are configured correctly, and the "separate" mode is OK too. However, when we apply the "attached" format to the new image, we can reproduce the error where the client receives tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match foo.example. The config with "separate" format is OK with the new image.

In the declarative config generated by KIC, the "attached" format is generated. While Kong gateway has the ability to process it and convert to "separate" format in dumping config before the Kong/kong#13357 PR. After this PR, Kong gateway "lost" the ability to deal with the "attached" format of SNIs.

[randmonkey]

Kong/kong#13516 and https://github.com/Kong/kong-ee/pull/10000 should fix it. The nightly image built today (2024-08-19) should pass the integration tests.
Update (2024-08-20): https://github.com/Kong/kubernetes-ingress-controller/actions/runs/10465337759/job/28981647787 still fails, but enterprise-dbless passes. After checking Kong docker repos, CE nightly build on 20240820 in kong/kong is not uploaded yet: https://hub.docker.com/r/kong/kong/tags?page=&page_size=&ordering=&name=202408

[randmonkey]

Close it as the integration tests against Kong nightly images passed.