[BUG] Policy Managed Indices failed to get status of snapsho

[justin-samuel]

Describe the bug

Implemented an index lifecycle management policy to transition indices older than 3 days from a hot to a cold state, in alignment with our specified requirements. Upon entering the cold state, the policy is designed to trigger a snapshot creation process followed by the deletion of the indices. This process appears to be functioning as intended, with all created snapshots displaying a "success" status within the snapshot repository. However, an issue has been observed within the Policy Managed Indices interface. For all indices that are older and have been transitioned by this policy, the following error message is being generated:
{ "cause": "[<storage repo>:<snapshot> is missing", "message": "Failed to get status of snapshot [index=xxxxxxxx]" }

Upon conducting a manual search for the snapshot using its full name, (ie security-auditlog-2024.02.20_snapshot-2024.02.23-00:06:31.687), the search yields no results, indicating that the snapshot cannot be retrieved by its complete identifier. However, when the search criteria is truncated (ie security-auditlog-2024.02.20_snapshot-2024.02.23-00), excluding the precise timestamp, the snapshot becomes visible and can be successfully retrieved. This discrepancy suggests an issue with how snapshot identifiers are being indexed, queried, or both, potentially involving the handling or parsing of detailed timestamp information in the snapshot naming convention.

Related component

Indexing

To Reproduce

1. Create Managed Policy and assign it to all indices
   { "id": "hot_to_cold_policy", "seqNo": 7151128, "primaryTerm": 58, "policy": { "policy_id": "hot_to_cold_policy", "last_updated_time": 1708659192697, "schema_version": 18, "error_notification": null, "default_state": "hot", "states": [ { "name": "hot", "actions": [], "transitions": [ { "state_name": "cold", "conditions": { "min_index_age": "1d" } } ] }, { "name": "cold", "actions": [ { "retry": { "count": 3, "backoff": "exponential", "delay": "1m" }, "snapshot": { "repository": "<Cold Storage repository>", "snapshot": "{{ctx.index}}" } }, { "retry": { "count": 3, "backoff": "exponential", "delay": "1m" }, "delete": {} } ], "transitions": [] } ], "ism_template": [ { "index_patterns": [ "*" ], "priority": 75, "last_updated_time": 1696882227220 } ] } }

2. Once the cold state occurs, view each index information

Expected behavior

During the cold state, the information should be able to search successfully for the snapshot

Additional Details

Host/Environment (please complete the following information):

• OS: ubuntu 22.04
• Opensearch v2.9.0

[peternied]

[Triage - attendees 1 2 3 4 5]
@justin-samuel Thanks for filing this issue, we'd be happy to accept a pull request to address the bug.