Dense Retrieval on big models

[MarwaEssam]

Hi, I am using Bigbird for encoding my documents. Does pysernini support dense retrieval using tokenizers for long sequences (4096 specifically)? If yes, how to setup bigbird encoder with it? I

Thanks

[lintool]

Hi @MarwaEssam - Have you looked at this guide? https://github.com/castorini/pyserini/#how-do-i-index-and-search-my-own-documents

Please let us know if you run into specific issues.

[MarwaEssam]

I read it. But for indexing my documents, I need to pass an encoder. I need this encoder to be the bigbird encoder. This is not shown in the examples provided. Alternatively, I could create my own .jsonl files that have the vectors of the documents encoded by bigbird to create the index. But how will I provide the query to the searcher without a supported encoder for "on-the fly" query encoding in the code? I need this piece of code because I generate a query based on some parameters in my work, and I further process the search results. Is there a way to pass an encoded query vector to the searcher?

[MXueguang]

Hi @MarwaEssam, yeah, we don't have a encoder with bigbird right now.
you probably have to setup pyserini in development mode and implement an encoder class like this https://github.com/castorini/pyserini/blob/master/pyserini/encode/_dpr.py
by replacing BertModel (DprModel) with corresponding Bigbird model.
Then use that query encoder during search.