Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Alerts break and alerting lock index becomes unrecoverable #1734

Open
rkbennett opened this issue Nov 28, 2024 · 1 comment
Open

[BUG] Alerts break and alerting lock index becomes unrecoverable #1734

rkbennett opened this issue Nov 28, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@rkbennett
Copy link

Describe the bug

When using alerting the .opensearch-alerting-config-lock shard isn't rotated and can grow past a recoverable shard size which then breaks all alerting. Because this is a shard that isn't managed by the user, a rotation strategy should be default

Related component

Storage

To Reproduce

  1. Create an alert
  2. Restart the cluster once the shard size has grown past the limit_size_in_bytes setting
  3. .opensearch-alerting-config-lock shard will never recover and alerts will no longer work

Expected behavior

System shards shouldn't get into permanently broken states

Additional Details

Plugins
Security

Host/Environment (please complete the following information):

  • OS: Opensearch 2.18 container image
  • Version 2.18
@rkbennett rkbennett added bug Something isn't working untriaged labels Nov 28, 2024
@dblock dblock transferred this issue from opensearch-project/OpenSearch Dec 2, 2024
@dblock dblock removed the untriaged label Dec 16, 2024
@dblock
Copy link
Member

dblock commented Dec 16, 2024

[Catch All Triage - 1, 2, 3]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: 🆕 New
Development

No branches or pull requests

2 participants