Secret engine not being disabled

[samuelvgo]

Describe the bug
Vault is unable to disable completely a secret engine, which is now stuck in the middle of not being usable anymore and not completely disabled.

To Reproduce
Steps to reproduce the behavior:

1. Login to vault
2. Run vault secrets disable /path/to/secret/engine
3. See error: error":"1 error occurred:\n\t* invalid request\n\n"}

Expected behavior
It was expected that secret engine would be completely disabled.

Environment:

• Vault Server Version (retrieve with vault status): Version 1.15.6
• Vault CLI Version (retrieve with vault version): Vault v1.15.6 (615cf6f), built 2024-02-28T17:07:34Z
• Server Operating System/Architecture: Debian GNU/Linux 12 (bookworm)

Vault server configuration file(s):

listener "tcp" {
  address       = "10.100.32.73:8200"
  tls_cert_file = "/etc/vault.d/ssl/certs/vault-server-cert.pem"
  tls_key_file  = "/etc/vault.d/ssl/private_keys/vault-server-key.pem"
  tls_cipher_suites = "TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
}
listener "tcp" {
  address       = "127.0.0.1:8200"
  tls_disable   = true
  max_request_duration = "900s"
}
storage "raft" {
  path = "/var/lib/vault"
  node_id = "hcvault-cde-cde-europe-west4-z44s"
}
service_registration "consul" {
  address = "127.0.0.1:8500"
  token = "
  service = "cdevault"
}
cluster_addr = "http://10.100.32.73:8201"
api_addr = "http://10.100.32.73:8200"
ui = true

Additional context
The vault server was version 1.14.10, and recently updated to version 1.15.6.
The disable initially was triggered via UI, and the secret engine had a significant number of records, with around 21499 pages in the search via UI.

Nothing with that secret engine works anymore, and the vault disable is failing constantly with:

Vault audit: delete failed - reason: 1 error occurred:
* invalid request

And when trying to create a secret it says:

route entry is tainted.

We have tried the leases force removal but it also didn't work

[miagilepner]

Hi, was the endpoint that you used to remove leases sys/leases/revoke-force/[:prefix]? If so, did you receive any errors from that endpoint? Are there any server logs with error messages?

You may be able to get help faster by asking on the discuss forum

[samuelvgo]

Hi @miagilepner, no I ran that slice differently using the vault lease revoke command:

[miagilepner]

Hi @samuelvgo I suggest then looking at the server logs to see if there is an error log from either the revoke command or the disable command.

[samuelvgo]

Hi @miagilepner Sorry for taking longer on this, the revoke command was successfull, but the disable command show this error:

{"time":"2024-10-30T13:30:11.415282308Z","type":"response","auth":{"client_token":"hmac-sha256:0d0dbdc7e6d7c6db317cca7dfe3b3371e715bd21505a239046b2912252e68390","accessor":"hmac-sha256:994ad73e98749a0bcd6f48c88424c022df78d4358508bd4dde31c9480748931c","display_name":"root","policies":["root"],"token_policies":["root"],"policy_results":{"allowed":true,"granting_policies":[{"name":"root","namespace_id":"root","type":"acl"}]},"token_type":"service","token_issue_time":"2020-05-06T08:08:39Z"},"request":{"id":"2ffc8a62-bc89-c3dd-b8dd-62ccc67c2a99","client_id":"0DHqvq2D77kL2/JTPSZkTMJbkFVmUu0TzMi0jiXcFy8=","operation":"delete","mount_point":"sys/","mount_type":"system","mount_accessor":"system_7e1ce005","mount_running_version":"v1.15.6+builtin.vault","mount_class":"secret","client_token":"hmac-sha256:0d0dbdc7e6d7c6db317cca7dfe3b3371e715bd21505a239046b2912252e68390","client_token_accessor":"hmac-sha256:994ad73e98749a0bcd6f48c88424c022df78d4358508bd4dde31c9480748931c","namespace":{"id":"root"},"path":"sys/mounts/opg/data","remote_address":"127.0.0.1","remote_port":40114},"response":{"mount_point":"sys/","mount_type":"system","mount_accessor":"system_7e1ce005","mount_running_plugin_version":"v1.15.6+builtin.vault","mount_class":"secret","data":{"error":"hmac-sha256:c94ca45a1e36d9f548e03b4453339326ce9e70fa5de774203b5b819d2e9cf107"}},"error":"1 error occurred:\n\t* invalid request\n\n"}

Vault audit: delete failed - reason: 1 error occurred:
* invalid request

Is there any other way to force that deletion? We are looking of deleting and re-creating it properly

[miagilepner]

@samuelvgo it looks like that is an audit log of the response to the mount deletion operation. What I would check is if there are vault server logs (see https://support.hashicorp.com/hc/en-us/articles/360002046068-Where-are-My-Vault-Logs-and-How-do-I-Share-Them-with-HashiCorp-Support for more details) that show errors when you attempt to do a lease revoke.

[samuelvgo]

@miagilepner So we didn't have any errors while trying to do the lease revoke, the logs suggest that everything worked:

{"time":"2024-11-05T09:19:02.038441926Z","type":"request","auth":{"client_token":"hmac-sha256:0d0dbdc7e6d7c6db317cca7dfe3b3371e715bd21505a239046b2912252e68390","accessor":"hmac-sha256:994ad73e98749a0bcd6f48c88424c022df78d4358508bd4dde31c9480748931c","display_name":"root","policies":["root"],"token_policies":["root"],"policy_results":{"allowed":true,"granting_policies":[{"name":"root","namespace_id":"root","type":"acl"}]},"token_type":"service","token_issue_time":"2020-05-06T08:08:39Z"},"request":{"id":"bb04a459-73e9-4e83-1a84-e16e0e7c2d11","client_id":"0DHqvq2D77kL2/JTPSZkTMJbkFVmUu0TzMi0jiXcFy8=","operation":"update","mount_point":"sys/","mount_type":"system","mount_accessor":"system_7e1ce005","mount_running_version":"v1.15.6+builtin.vault","mount_class":"secret","client_token":"hmac-sha256:0d0dbdc7e6d7c6db317cca7dfe3b3371e715bd21505a239046b2912252e68390","client_token_accessor":"hmac-sha256:994ad73e98749a0bcd6f48c88424c022df78d4358508bd4dde31c9480748931c","namespace":{"id":"root"},"path":"sys/leases/revoke-force/opg/data","remote_address":"127.0.0.1","remote_port":22898}}

Vault audit: update succesfull

The following two commands were issued, and both returned as successful:

vault lease revoke -force -prefix opg/data/
Warning! Force-removing leases can cause Vault to become out of sync with
secret engines!
Success! Force revoked any leases with prefix: opg/data/

and

vault lease revoke -force -prefix sys/mounts/opg/data/
Warning! Force-removing leases can cause Vault to become out of sync with
secret engines!
Success! Force revoked any leases with prefix: sys/mounts/opg/data/