You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The benchmark rules describe that submissions also need to perform well on held-out workloads. This is currently not automatically accounted for in the scoring code.
Specifically, (quoting from the benchmark rules):
For a submission to receive a finite training time on a fixed workload, it needs to:
Reach the validation target on the fixed workload within the maximum runtime.
Reach the validation target fixed workload within 4x of the fastest submission.
Reach the validation target on the held-out workload (corresponding to the fixed workload) within the maximum runtime.
Reach the validation target on the held-out workload (corresponding to the fixed workload) within 4x of the fastest submission. To determine the fastest submission on a held-out workload, we only consider submissions that reached the target on the corresponding fixed workload. This protects us against extremely fast submissions that only work on a specific held-out workload and are useless as general algorithms.
The text was updated successfully, but these errors were encountered:
The benchmark rules describe that submissions also need to perform well on held-out workloads. This is currently not automatically accounted for in the scoring code.
Specifically, (quoting from the benchmark rules):
For a submission to receive a finite training time on a fixed workload, it needs to:
The text was updated successfully, but these errors were encountered: