[Remote Store] Remote Store failures can prevent empty translogs from getting created, causing failed recoveries

[Bukhtawar]

Describe the bug

RecoveryFailedException[[xxx-000002][15]: Recovery failed on {8fb5b3dfd37b0ff02f2c3a52a4e66570}{gX97tZZQQ0exYv5PSbVt2w}{hXpPKrmMRkyKYSss2FDuwg}{172.16.38.237}{172.16.38.237:9300}{dir}{dp_version=20210501, adv_sec_enabled=true, cross_cluster_transport_address=2600:1f14:3178:6282:d9b4:50bd:7587:e6d3, awareness_features_enabled=true, search_backpressure_feature_present=true, plugins.security.custom_serialization.enabled=false, box_type=hot, distributed_snapshot_deletion_enabled=false, cold_enabled=true, zone=us-west-2b, global_cpu_usage_ac_supported=true, shard_indexing_pressure_enabled=true, di_number=1}]; nested: IndexShardRecoveryException[failed recovery]; nested: TranslogCorruptedException[translog from source [/path/data/nodes/0/indices/10KCAO5o1OSb-lV4riL9yMvg/15/translog/translog.ckp] is corrupted]; nested: NoSuchFileException[/hath/data/nodes/0/indices/10KCAO5o1OSb-lV4riL9yMvg/15/translog/translog.ckp];
        at org.opensearch.index.shard.IndexShard.lambda$executeRecovery$30(IndexShard.java:3800)
        at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)
        at org.opensearch.index.shard.StoreRecovery.lambda$recoveryListener$8(StoreRecovery.java:541)
        at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)
        at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:347)
        at org.opensearch.index.shard.StoreRecovery.recoverFromRemoteStore(StoreRecovery.java:134)
        at org.opensearch.index.shard.IndexShard.restoreFromRemoteStore(IndexShard.java:2830)
        at org.opensearch.index.shard.IndexShard.lambda$startRecovery$24(IndexShard.java:3683)
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: [xxx-000002/10KCAO5o1OSb-lV4riL9yMvg][[xxx-000002][15]] IndexShardRecoveryException[failed recovery]; nested: TranslogCorruptedException[translog from source [/path/data/nodes/0/indices/10KCAO5o1OSb-lV4riL9yMvg/15/translog/translog.ckp] is corrupted]; nested: NoSuchFileException[/path/data/nodes/0/indices/10KCAO5o1OSb-lV4riL9yMvg/15/translog/translog.ckp];
        ... 12 more
Caused by: TranslogCorruptedException[translog from source [/path/data/nodes/0/indices/10KCAO5o1OSb-lV4riL9yMvg/15/translog/translog.ckp] is corrupted]; nested: NoSuchFileException[/path/data/nodes/0/indices/10KCAO5o1OSb-lV4riL9yMvg/15/translog/translog.ckp];
        at org.opensearch.index.translog.Checkpoint.read(Checkpoint.java:210)
        at org.opensearch.index.shard.StoreRecovery.recoverFromRemoteStore(StoreRecovery.java:570)
        at org.opensearch.index.shard.StoreRecovery.lambda$recoverFromRemoteStore$1(StoreRecovery.java:136)
        at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:344)
        ... 9 more
Caused by: java.nio.file.NoSuchFileException: /path/data/nodes/0/indices/10KCAO5o1OSb-lV4riL9yMvg/15/translog/translog.ckp
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:181)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:298)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:357)
        at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:78)
        at org.opensearch.index.translog.Checkpoint.read(Checkpoint.java:202)
        ... 12 more

Related component

Storage:Remote

To Reproduce

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

Ensure empty translog creation in the sync path of shard creation to avoid failing recoveries later

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[peternied]

[Triage - attendees 1 2 3 4 5 6 7 8]
@Bukhtawar Thanks for creating this issue; however, it isn't being accepted due to not having clear steps to reproduce or context on how to get into this state. Please feel free to re-open after addressing the reason