Fixing _list/shards API for closed indices

[gargharsh3134]

Description

By default, the Broadcast requests use IndicesOptions.strictExpandOpenAndForbidClosed(). For _list/shards API, concrete indices from strategy are being passed to IndicesStats Action with the default indicesOptions.
Details around IndexNameExpressionResolver which are causing issues with the list/shards API:

1. Concrete indices if specified in request with indices options as IndicesOptions.strictExpandOpenAndForbidClosed(), it would result in an error (index_closed_exception).
2. Concrete index even if is hidden, will get evaluated and fetched. Basically, if wildcard is to be expanded the IndexOption will ignore hidden indices, but if concretely specified it will get included. Similar is the behaviour with existing _cat/indices API, which by default (curl localhost:9200/_cat/indices), omits the hidden indices but if one of them is explicitly queried, it will return the stats (curl localhost:9200/_cat/indices/hidden-index).

_list/shards API throws index_closed_exception whenever the cluster has closed indices. This change is to resolve and ignore the closed indices from getting explicitly passed to IndicesStatsAction.

Functional Testing

On a local cluster with 2 indices:
open-test-index -> Open index with 1P and 1R.
new-test-idx -> closed index with 1P and 2 R.

gkharsh@bcd07443f159 OpenSearch % curl localhost:9200/_cat/indices
green open  open-test-index rvZzxo5BSnK3TFlmlPQMkw 1 1 0 0 416b 208b
green close new-test-idx    eHm4npFbTQ-dfU8abSLJoQ 1 2 

Existing _cat/shards behaviour:

Unfiltered default query outputs closed shards without stats.

gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_cat/shards"                                           
open-test-index 0 p STARTED 0 208b 127.0.0.1 data1
open-test-index 0 r STARTED 0 208b 127.0.0.1 data2
new-test-idx    0 p STARTED        127.0.0.1 data1
new-test-idx    0 r STARTED        127.0.0.1 data2
new-test-idx    0 r STARTED        127.0.0.1 data3

Explicitly querying for a concrete closed index, throws error.

gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_cat/shards/new-test-idx"   
{"error":{"root_cause":[{"type":"index_closed_exception","reason":"closed","index":"new-test-idx","index_uuid":"eHm4npFbTQ-dfU8abSLJoQ"}],"type":"index_closed_exception","reason":"closed","index":"new-test-idx","index_uuid":"eHm4npFbTQ-dfU8abSLJoQ"},"status":400}%    

Wildcards only match open indices and response is empty if only closed indices match the expression

gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_cat/shards/new-test-idx*"
gkharsh@bcd07443f159 OpenSearch % 

Fixed _list/shards behaviour:

Unfiltered default query will output closed shards without stats.

gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_list/shards"             
new-test-idx    0 p STARTED        127.0.0.1 data1
new-test-idx    0 r STARTED        127.0.0.1 data2
new-test-idx    0 r STARTED        127.0.0.1 data3
open-test-index 0 p STARTED 0 208b 127.0.0.1 data1
open-test-index 0 r STARTED 0 208b 127.0.0.1 data2
next_token null

Explicitly querying for a concrete closed index, will output closed shards without stats instead of throwing error.

gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_list/shards/new-test-idx" 
new-test-idx 0 p STARTED   127.0.0.1 data1
new-test-idx 0 r STARTED   127.0.0.1 data2
new-test-idx 0 r STARTED   127.0.0.1 data3
next_token null

Wildcards continue to match only open indices and response is empty if only closed indices match the expression

gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_list/shards/new-test-idx*"
next_token null

Related Issues

Resolves #16626

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for c62e5f2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 691803c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 537fb2d: UNSTABLE

• TEST FAILURES:

      2 org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

✅ Gradle check result for 40b2641: SUCCESS

[github-actions]

✅ Gradle check result for 10d0701: SUCCESS

[github-actions]

✅ Gradle check result for 77c7c9f: SUCCESS

[github-actions]

❌ Gradle check result for 152b182: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[rajiv-kv]

LGTM !

[github-actions]

✅ Gradle check result for f4fb43d: SUCCESS

[github-actions]

✅ Gradle check result for 790b8c0: SUCCESS