-
Notifications
You must be signed in to change notification settings - Fork 115
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
48b04f1
commit 8785b9d
Showing
2 changed files
with
20 additions
and
21 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -252,27 +252,6 @@ There are several ways to get help as you develop your algorithm using this fram | |
* You can submit an issue at this github repository. | ||
* Send en email to the competition's googlegroup, [email protected] | ||
|
||
### Leaderboard | ||
|
||
This leaderboard is based on the standard recall@10 vs throughput benchmark that has become a standard benchmark when evaluating and comparing approximate nearest neighbor algorithms. | ||
The recall of the baselines at this QPS threshold is listed [above](#measuring_your_algorithm). | ||
|
||
For tasks "Filter", "Out-of-Distribution" and "Sparse" tracks, algorithms were ranked on the QPS they achieve on the track dataset, as long as the recall@10 is at least 90%. | ||
These results files for [Azure D8lds_v5](Azure_D8lds_v5_table.md) and [AWS EC2 c6i.2xlarge](ec2_c6i.2xlarge_table.md) list the maximum QPS measured for each algorihtm with at least 90% recall@10. | ||
|
||
For the Streaming track, algorithms will be ranked on recall@10, as long as each algorithm completes the runbook within the alloted 1 hour. | ||
The [result file](streaming/res_final_runbook_AzureD8lds_v5.csv) lists measurements on Azure D8lds_v5. | ||
|
||
QPS vs recall@10 plots for tracks based on public queries on Azure D8lds_v5: | ||
**Filter track** | ||
![yfcc-10M](filter/plot_public_queries_AzureD8lds_v5.png) | ||
|
||
**OOD track** | ||
![text2image-10M](ood/plot_public_queries_AzureD8lds_v5.png) | ||
|
||
**Sparse track** | ||
![sparse-full](sparse/plot_public_queries_AzureD8lds_v5.png) | ||
|
||
## Custom_Setup | ||
|
||
While we encourage using our framework for all steps of the evaluation, we consider open-source submissions that diverge from the proposed setup. | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
### Leaderboard | ||
|
||
This leaderboard is based on the standard recall@10 vs throughput benchmark that has become a standard benchmark when evaluating and comparing approximate nearest neighbor algorithms. | ||
The recall of the baselines at this QPS threshold is listed [above](#measuring_your_algorithm). | ||
|
||
For tasks "Filter", "Out-of-Distribution" and "Sparse" tracks, algorithms were ranked on the QPS they achieve on the track dataset, as long as the recall@10 is at least 90%. | ||
These results files for [Azure D8lds_v5](Azure_D8lds_v5_table.md) and [AWS EC2 c6i.2xlarge](ec2_c6i.2xlarge_table.md) list the maximum QPS measured for each algorihtm with at least 90% recall@10. | ||
|
||
For the Streaming track, algorithms will be ranked on recall@10, as long as each algorithm completes the runbook within the alloted 1 hour. | ||
The [result file](streaming/res_final_runbook_AzureD8lds_v5.csv) lists measurements on Azure D8lds_v5. | ||
|
||
QPS vs recall@10 plots for tracks based on public queries on Azure D8lds_v5: | ||
**Filter track** | ||
![yfcc-10M](filter/plot_public_queries_AzureD8lds_v5.png) | ||
|
||
**OOD track** | ||
![text2image-10M](ood/plot_public_queries_AzureD8lds_v5.png) | ||
|
||
**Sparse track** | ||
![sparse-full](sparse/plot_public_queries_AzureD8lds_v5.png) |