Skip to content

Commit

Permalink
added leaderboard
Browse files Browse the repository at this point in the history
  • Loading branch information
harsha-simhadri committed Dec 7, 2023
1 parent 48b04f1 commit 8785b9d
Show file tree
Hide file tree
Showing 2 changed files with 20 additions and 21 deletions.
21 changes: 0 additions & 21 deletions neurips23/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -252,27 +252,6 @@ There are several ways to get help as you develop your algorithm using this fram
* You can submit an issue at this github repository.
* Send en email to the competition's googlegroup, [email protected]

### Leaderboard

This leaderboard is based on the standard recall@10 vs throughput benchmark that has become a standard benchmark when evaluating and comparing approximate nearest neighbor algorithms.
The recall of the baselines at this QPS threshold is listed [above](#measuring_your_algorithm).

For tasks "Filter", "Out-of-Distribution" and "Sparse" tracks, algorithms were ranked on the QPS they achieve on the track dataset, as long as the recall@10 is at least 90%.
These results files for [Azure D8lds_v5](Azure_D8lds_v5_table.md) and [AWS EC2 c6i.2xlarge](ec2_c6i.2xlarge_table.md) list the maximum QPS measured for each algorihtm with at least 90% recall@10.

For the Streaming track, algorithms will be ranked on recall@10, as long as each algorithm completes the runbook within the alloted 1 hour.
The [result file](streaming/res_final_runbook_AzureD8lds_v5.csv) lists measurements on Azure D8lds_v5.

QPS vs recall@10 plots for tracks based on public queries on Azure D8lds_v5:
**Filter track**
![yfcc-10M](filter/plot_public_queries_AzureD8lds_v5.png)

**OOD track**
![text2image-10M](ood/plot_public_queries_AzureD8lds_v5.png)

**Sparse track**
![sparse-full](sparse/plot_public_queries_AzureD8lds_v5.png)

## Custom_Setup

While we encourage using our framework for all steps of the evaluation, we consider open-source submissions that diverge from the proposed setup.
Expand Down
20 changes: 20 additions & 0 deletions neurips23/leaderboard.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
### Leaderboard

This leaderboard is based on the standard recall@10 vs throughput benchmark that has become a standard benchmark when evaluating and comparing approximate nearest neighbor algorithms.
The recall of the baselines at this QPS threshold is listed [above](#measuring_your_algorithm).

For tasks "Filter", "Out-of-Distribution" and "Sparse" tracks, algorithms were ranked on the QPS they achieve on the track dataset, as long as the recall@10 is at least 90%.
These results files for [Azure D8lds_v5](Azure_D8lds_v5_table.md) and [AWS EC2 c6i.2xlarge](ec2_c6i.2xlarge_table.md) list the maximum QPS measured for each algorihtm with at least 90% recall@10.

For the Streaming track, algorithms will be ranked on recall@10, as long as each algorithm completes the runbook within the alloted 1 hour.
The [result file](streaming/res_final_runbook_AzureD8lds_v5.csv) lists measurements on Azure D8lds_v5.

QPS vs recall@10 plots for tracks based on public queries on Azure D8lds_v5:
**Filter track**
![yfcc-10M](filter/plot_public_queries_AzureD8lds_v5.png)

**OOD track**
![text2image-10M](ood/plot_public_queries_AzureD8lds_v5.png)

**Sparse track**
![sparse-full](sparse/plot_public_queries_AzureD8lds_v5.png)

0 comments on commit 8785b9d

Please sign in to comment.