Skip to content

Commit

Permalink
additional notes
Browse files Browse the repository at this point in the history
  • Loading branch information
jxnl committed Feb 6, 2024
1 parent 989f2b6 commit 1f51783
Showing 1 changed file with 6 additions and 0 deletions.
6 changes: 6 additions & 0 deletions docs/writing/posts/lgtmk.md
Original file line number Diff line number Diff line change
Expand Up @@ -227,3 +227,9 @@ I hope this post has provided you with better intuition on how to think about re
4. Define metrics, conduct tests, investigate areas of poor performance, and then reevaluate the system.
5. Explore simple metrics for relevance and ranking, such as MAR, MAP, MRR, and NDCG.
6. Remember that these metrics should ultimately align with desired business outcomes.

## Additional Notes

1. Notice that for MAR and MAP, They do not depend on the rank only the presence of the relevant document in the top K. This is why they are often used in the context of retrieval.
2. Notice that for MRR and NDCG, they depend on the rank of the relevant document. This is why they are often used in the context of ranking. If you end up building a sophisticated RAG application, you'll find that a lot of the time, many of the queries are just asking for documents which great opportunity to consider a ranking metric above just a regular context retrieval mechanism. If each document is 20 pages, you'll likely really care about which document shows up first.
3. Showing your work is super important for products that need to gain the user's trust. Again, ranking becomes really relevant even though language models themselves might not care.

0 comments on commit 1f51783

Please sign in to comment.