Approach described in Cohere document

[aamir-s18]

Hey,

I want to try the approach described in the Cohere document.

Let's say I take the English dataset of MIRACL as an example. Are these steps here correct?

1. Download the BM25 index of the English dataset

from pyserini.search.lucene import LuceneSearcher

searcher = LuceneSearcher.from_prebuilt_index('miracl-v1.0-en')

2. Obtain the dev queries and hits from huggingface and load them

queries = pd.read_csv("data/queries.tsv", sep="\t", header=None, names=["qid", "query"])
results = pd.read_csv("data/hits.tsv", sep="\t", header=None, names=["qid", "q", "docid", "relevant"])

3. Search and get top-100 hits and get the texts of the hits

hits = searcher.search(query, k=100)
texts = [json.loads(hit.raw)["text"] for hit in hits]

4. Get embeddings of the hits and query

embeddings = get_embedding(texts)
query_embeddings = get_embedding(query)

5. Get the similarities of query and texts and get the top-10

from sklearn.metrics.pairwise import cosine_similarity

similarities = cosine_similarity(query_embeddings, embeddings_formatted)
similarity_hit_list = []
    for hit, similarity in zip(hits, similarities[0]):
        similarity_hit_list.append((similarity, hit.docid))
top_10 = similarity_hit_list[:10]

6. Get nDCG

from sklearn.metrics import ndcg_score

golden_hit = results.loc[(results["qid"] == queries["qid"][i]) & (results["relevant"] == 1)]
golden_doc_ids = golden_hit["docid"].to_list()
true_relevance = []
predicted_scores = []

# Iterate through your top 10 results
for similarity, docid in top_10:
    # If the document is in your list of golden hits (relevant documents)
    if docid in golden_doc_ids:
        # Append a relevance score of 1
        true_relevance.append(1)
    else:
        # Otherwise append a score of 0
        true_relevance.append(0)
    # Add the similarity as the predicted score
    predicted_scores.append(similarity)

ndcg = ndcg_score([true_relevance], [predicted_scores])
print(ndcg)

It would be nice if you could publish the code. This would help to try out other models with MIRACL without encoding the whole dataset, which can become expensive super fast (cost-, time- and resource-wise).

Thank you!

[lintool]

Sorry, somehow missed this until now. Dunno if you've see our paper about this? https://aclanthology.org/2023.acl-industry.50/

Please re-open issue and follow up if desired?