Custom Storage Backends?

[only-cliches]

I noticed that Meilisearch has high availability planned for the near future, I wonder if there's an easier solution.

Would it be possible to support different storage backends? This way the application developer can decide on the tradeoffs by choosing the right backend for their use. Running on a local device? SQLite is perfect. Running on a web server? RocksDB is tuned for that. Need high availability? ScyllaDB or MongoDB has you covered.

This abstracts the problem of high availability and effective high performance file system storage from Meilisearch. You could still provide a default backend, but there are already a host of data storage libraries that provide various tradeoffs for storage, why not let them do the heavy lifting?

[gmourier]

Hi @only-cliches 👋

Thanks for the feedback!

It is not something we consider profitable for several reasons despite the intrinsic advantage of being able to take advantage of the native high availability of some of the tools mentioned here.

Indeed, it seems difficult to us to develop an abstraction layer to fits each potential backend. Moreover, some backends do not allow to have a data structure that allows proposing all the features of MeiliSearch. They are not built as full-text search engines that manage typo tolerance or the concept of tie-breaking of our ranking rules. It forces in a way to store our data in a relational format for which MeiliSearch is not made.

It seems to me to be compromising for the performances to have a pass between MeiliSearch and a backend other than LMDB in addition to adding a lot of complexity.

We could propose in the future a different backend like RocksDB in addition to LMDB. But that would not allow benefiting from a native high availability.

That said, I could be wrong and I would love to have more information or concrete examples so we can look at this!