No cache eviction in Vault Proxy after updating from v1.16.3 to v1.17.6

[dominik-bln]

Describe the bug

We are using vault proxy with enabled static secret caching. When running with v1.16.3, updating and accessing a cached secret looks like the following:

Sep 26 15:14:26 SOMESERVERNAME vault[1439420]: 2024-09-26T15:14:26.315+0200 [DEBUG] proxy.cache.staticsecretcacheupdater: received update static secret request: path=qa-jenkins/data/job-credentials/data-engineering/test indexId=85df31f4aaade9c2ca71e196d22abd06210fb93619d8f6bcd14d9a560550891e
Sep 26 15:14:26 SOMESERVERNAME vault[1439420]: 2024-09-26T15:14:26.392+0200 [DEBUG] proxy.cache.staticsecretcacheupdater: evicting response from cache: path=qa-jenkins/data/job-credentials/data-engineering/test
Sep 26 15:15:08 SOMESERVERNAME vault[1439420]: 2024-09-26T15:15:08.243+0200 [INFO]  proxy.apiproxy: received request: method=GET path=/v1/qa-jenkins/data/job-credentials/data-engineering/test
Sep 26 15:15:08 SOMESERVERNAME vault[1439420]: 2024-09-26T15:15:08.243+0200 [DEBUG] proxy.apiproxy: using auto auth token: method=GET path=/v1/qa-jenkins/data/job-credentials/data-engineering/test
Sep 26 15:15:08 SOMESERVERNAME vault[1439420]: 2024-09-26T15:15:08.243+0200 [DEBUG] proxy.cache.leasecache: forwarding request from cache: method=GET path=/v1/qa-jenkins/data/job-credentials/data-engineering/test
Sep 26 15:15:08 SOMESERVERNAME vault[1439420]: 2024-09-26T15:15:08.243+0200 [INFO]  proxy.apiproxy: forwarding request to Vault: method=GET path=/v1/qa-jenkins/data/job-credentials/data-engineering/test
Sep 26 15:15:08 SOMESERVERNAME vault[1439420]: 2024-09-26T15:15:08.243+0200 [DEBUG] proxy.apiproxy.client: performing request: method=GET url=https://vault.host/v1/qa-jenkins/data/job-credentials/data-engineering/test
Sep 26 15:15:08 SOMESERVERNAME vault[1439420]: 2024-09-26T15:15:08.320+0200 [DEBUG] proxy.cache.leasecache: storing static secret response into the cache: method=GET path=/v1/qa-jenkins/data/job-credentials/data-engineering/test id=85df31f4aaade9c2ca71e196d22abd06210fb93619d8f6bcd14d9a560550891e

When updating to v1.17.5 the cached secret is not evicted anymore and the response is outdated:

Sep 26 15:30:16 SOMESERVERNAME vault[1441847]: 2024-09-26T15:30:16.307+0200 [DEBUG] proxy.cache.staticsecretcacheupdater: received update static secret request: path=qa-jenkins/data/job-credentials/data-engineering/test indexId=936c1f40ca9966d0e65306100b8a7f64212e7bd3b2da63d0d9c9c02bc73fb1f0
Sep 26 15:30:26 SOMESERVERNAME vault[1441847]: 2024-09-26T15:30:26.629+0200 [INFO]  proxy.apiproxy: received request: method=GET path=/v1/sys/internal/ui/mounts/qa-jenkins
Sep 26 15:30:26 SOMESERVERNAME vault[1441847]: 2024-09-26T15:30:26.629+0200 [DEBUG] proxy.apiproxy: using auto auth token: method=GET path=/v1/sys/internal/ui/mounts/qa-jenkins
Sep 26 15:30:26 SOMESERVERNAME vault[1441847]: 2024-09-26T15:30:26.629+0200 [DEBUG] proxy.cache.leasecache: forwarding request from cache: method=GET path=/v1/sys/internal/ui/mounts/qa-jenkins
Sep 26 15:30:26 SOMESERVERNAME vault[1441847]: 2024-09-26T15:30:26.629+0200 [INFO]  proxy.apiproxy: forwarding request to Vault: method=GET path=/v1/sys/internal/ui/mounts/qa-jenkins

To Reproduce

Steps to reproduce the behavior:

1. Start Vault Proxy with enabled static secret caching (see below for full config)
2. Run on an existing secret VAULT_ADDR=http://localhost:8100 vault kv get -mount=qa-jenkins job-credentials/data-engineering/test
3. Update secret in Vault
4. Run VAULT_ADDR=http://localhost:8100 vault kv get -mount=qa-jenkins job-credentials/data-engineering/test again
5. Received secret is the same as in 2. even though the cache should have been evicted

Expected behavior

The 2nd request should have received the updated secret data.

Environment:

• Vault Server Version: Vault 1.16.2+ent
• Vault CLI Version: Vault v1.17.6 (69a720d), built 2024-09-24T19:48:40Z
• Server Operating System/Architecture: Ubuntu 22.04

Vault server configuration file(s):

pid_file = "/etc/vault/proxy-pidfile"
log_level = "debug"

vault {
  address = "<REMOVED>"
  retry {
    num_retries = 5
  }
  namespace = "namespace/subnamespace"
}

auto_auth {
  method "aws" {
    config = {
      type = "iam"
      role = "jenkins-reader"
    }
  }
}

cache {
  cache_static_secrets = true
  static_secret_token_capability_refresh_interval = "1h"
}

api_proxy {
  use_auto_auth_token = "force"
  enforce_consistency = "always"
  prepend_configured_namespace = true
}

listener "tcp" {
  address = "127.0.0.1:8100"
  tls_disable = true
}

Additional context