Pluggable alternatives to translog functionality using existing market solutions

[anasalkouz]

Instead of having the translog saved into the file system and care of replicating it to other replica nodes, or care of backing up files (Once segement replication introduced). we can make an extend of the functionality and provide the flexibility to OpenSearch's clients to choose one of the existing solutions in the market to backup or store OpenSearch operations. For example we can use Kineses or Kafka.

[muralikpbhat]

Isn't object store (like S3) also an option? Queues can be really expensive at scale, especially when these messages are throw away unless the primary fails and then we need it for recovery. A cheaper storage for logging is better choice. However making it work for every single document write is challenging and we might have to suggest a bigger bulk size or grouping of documents before translog is committed (to avoid object storage write for every single doc)

[muralikpbhat]

We should also consider whether we want to just replace the current translog with other solutions from market or rethink about the whole durability story. It might be useful to have a persistent stream in-front so that clients can just keep streaming the data. We can internally consume from the queue and index at the possible rate. This would also avoid the requirement for the current translog since we can do checkpointing on this queue and re-drive the messages incase it is failed to persist the index durably.

[anasalkouz]

Closing this issue, since we are tracking pluggable translog on this issue #1319