Migrations fail on single node clusters due to unavailable shards exception

[rudolf]

We are seeing frequent migration-related failures on CI with a message like:

Not enough active copies to meet shard count of [ALL] (have 1, needed 2)

E.g. #156117 (comment)

After speaking to the Elasticsearch team this appears to be a race condition in Elasticsearch that only happens on single node clusters. We create indices with "auto_expand_replicas": "0-1" and wait for a shards_acknowledged=true response. On a single node cluster this creates an index with 0 replicas.

info [o.e.c.m.MetadataCreateIndexService] [node-01] [.kibana_8.8.0_reindex_temp] creating index, cause [api], templates [], shards [1]/[1]

However, even if the create index API responds that all shards are available, there is a brief time where the index actually has 1 replica assigned which cannot/has not been assigned. ES then immediately adjusts the replicas down to 0

info [o.e.c.r.a.AllocationService] [node-01] updating number_of_replicas to [0] for indices [.kibana_8.8.0_reindex_temp]

However, in the brief time between these two messages, if Kibana indexes any data or searches against the index we'll get "Not enough active copies to meet shard count of [ALL] (have 1, needed 2)" errors.

This should be fixed upstream by ES, but in the meantime we can work around this problem by always waiting for the index status to turn "green".

Note: the error message is similar to #127136 but this is a different issue.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[rudolf]

To do this we'd need to remove the if (res.acknowledged && res.shardsAcknowledged) check, something like:

--- a/packages/core/saved-objects/core-saved-objects-migration-server-internal/src/actions/create_index.ts
+++ b/packages/core/saved-objects/core-saved-objects-migration-server-internal/src/actions/create_index.ts
@@ -146,25 +146,20 @@ export const createIndex = ({
       AcknowledgeResponse,
       'create_index_succeeded'
     >((res) => {
-      if (res.acknowledged && res.shardsAcknowledged) {
-        // If the cluster state was updated and all shards started we're done
-        return TaskEither.right('create_index_succeeded');
-      } else {
-        // Otherwise, wait until the target index has a 'green' status meaning
-        // the primary (and on multi node clusters) the replica has been started
-        return pipe(
-          waitForIndexStatus({
-            client,
-            index: indexName,
-            timeout: DEFAULT_TIMEOUT,
-            status: 'green',
-          }),
-          TaskEither.map(() => {
-            /** When the index status is 'green' we know that all shards were started */
-            return 'create_index_succeeded';
-          })
-        );
-      }
+      // Otherwise, wait until the target index has a 'green' status meaning
+      // the primary (and on multi node clusters) the replica has been started
+      return pipe(
+        waitForIndexStatus({
+          client,
+          index: indexName,
+          timeout: DEFAULT_TIMEOUT,
+          status: 'green',
+        }),
+        TaskEither.map(() => {
+          /** When the index status is 'green' we know that all shards were started */
+          return 'create_index_succeeded';
+        })
+      );
     })
   );
 };

[pgayvallet]

@gsoldevila I guess #157973 should have closed this one?

EDIT: or maybe not given the last messages from @dmlemeshko on slack

[rudolf]

Second attempt to fix this #158182

[rudolf]

Closing as we have not seen further failures on CI