How can I run the indexing and searching in memory instead of on disk？

[Wenze7]

[lintool]

Hi @Wenze7 - this is an ill-formed question. What exactly are you trying to do? Dense/spare already try to make effective use of memory, but in different ways.

[Wenze7]

Sorry. According to my guess, unlike faiss, sparse's main bottleneck is hard disk. Can I run sparse in memory in the same way as the faiss？

[lintool]

Nope, it doesn't work like that. There's an in-memory option in Anserini on the Lucene end, but based on my experience, it doesn't increase retrieval speed by much. The standard way to increase speed is to partition the dataset and search each partition in parallel.

[TylerRankin]

Im running the pyserini LuceneSearcher on azure web app, so there is latency when loading in indexes from azure storage. Is there anyway to pre-load in all the indexes I need ? Thanks !