Issue Loading my Dense Retriever Index

[jonsaadfalcon]

Hello, after I trained my own index, I tried to load a SimpleDenseRetriever with the following line:

encoder = TctColBertQueryEncoder('IndexingExperiments/tct_colbert-msmarco')
searcher = SimpleDenseSearcher('IndexingExperiments/indexes/TREC_PM_Document_Collection', query_encoder=encoder)

However, it gave me the following error after running this line:

RuntimeError: Error in faiss::(anonymous namespace)::FileIOReader::FileIOReader(const char *) at index_io.cpp:136: Error: 'f' failed: could not open IndexingExperiments/indexes/TREC_PM_Document_Collection/index for reading: No such file or directory

However, the indexing command I ran never gave me any file titled "index". I used the following indexing command for generating my own index on my own data:

python -m pyserini.index \ --input TREC_PM_Document_Collection \ --collection JsonCollection \ --generator DefaultLuceneDocumentGenerator \ --index indexes/TREC_PM_Document_Collection \ --threads 1 \ --storePositions --storeDocvectors --storeRaw

Any suggestions on how to address the error?

[MXueguang]

Hi @jonsaadfalcon ,

after I trained my own index

I am assuming you trained your dense retriever model?

However, the pyserini.index is creating Lucene sparse index.
SimpleDenseSearcher requires dense index based on FAISS.

Please check this document to encode the corpus and create dense index:
https://github.com/castorini/pyserini/blob/master/docs/usage-encode.md#encode-documents-with-dense-encoder

[jonsaadfalcon]

For setting the --devices tag for the command, I tried specifying --devices [cuda:0, cuda:1] but it doesn't seem to work. I was wondering what the proper command is for specifying more than one cuda device for indexing