not able to clear alarm in etcd 3.5.3 version by using etcdctl alarm disarm command

[rahulbapumore]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via [email protected].
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

Alarm was raised on ETCD cluster, but later DB size got reduced lesser than ETCD_BACKEND_QUOTA_BYTE, but still cant put values into DB because Nospace alarm still exist and not cleared.
When We try to clear it using etcdctl alarm disarm command it logs timeout error inside the ogs

What did you expect to happen?

Alarm should be cleared and data should be able to be inserted

How can we reproduce it (as minimally and precisely as possible)?

NA

Anything else we need to know?

No response

Etcd version (please run commands below)

ETCD version 3.5.3

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

No response

[rahulbapumore]

Hi @ahrtr ,
We faced this issue and blocking to insert data, Any inputs for clearing the alarm?

Thanks

[ahrtr]

Please provide detailed steps to reproduce the issue. Or the detailed steps what you did.

[ahrtr]

Please ensure you have correctly compacted & defragmented the db, and finally disalarmed.

References:

• https://etcd.io/docs/v3.5/op-guide/maintenance/#space-quota
• https://etcd.io/blog/2023/how_to_debug_large_db_size_issue/

@Elbehery could you followup this ticket until it's closed?

[Elbehery]

On it 🙏🏽

[rahulbapumore]

Hi @Elbehery ,
We think our issue is similar to #14379 this ticket
Any workaround to resolve this issue by performing any procedure or steps?

Thanks

[ahrtr]

Please also double confirm whether you can still see this issue on latest release 3.5.11.

[rahulbapumore]

@Elbehery
Latest version we havent seen it in 3.5.11, but we need any work around for live deployment which is having 3.5.3.
Because even if we have db space available, due to alarm we are not able to put data in etcd

[rahulbapumore]

@Elbehery ,
Also etcdctl member list does not show any alarm
also etcdctl endpoint health does not show any alarm
but etcdctl endpoint status --write-out=table shows nospace alarm in output

[Elbehery]

@rahulbapumore

can you please give details how to reproduce ?

also can u paste some logs ?

[rahulbapumore]

Can deleting wal folder will resolve this issue?
@Elbehery

[Elbehery]

did you try to restart the etcd pod on the node which raised the alarm ?

so when the pod will start, it will re-read the snapshot and the wal, this might help

[Elbehery]

@rahulbapumore ^^

[rahulbapumore]

Yes I tried deleting pods but same issue

[Elbehery]

can u upgrade to the latest release ??

also please describe the environment you use, and if some logs are available would be gr8

[rahulbapumore]

Hi @Elbehery ,
By upgrading
will alarm issue get resolved?

[rahulbapumore]

We are using ETCD inside kubernetes pods controlled by statefulset.
We will try to get logs and provide you

[rahulbapumore]

Hi @Elbehery , By upgrading will alarm issue get resolved?

anything on this?

[Elbehery]

i really cant help without logs

did u try restarting the node that caused the alarm ?