Notes on the CARLA leaderboard

[Kin-Zhang]

some discussion collections on the CARLA leaderboard.

---

Notes:

1. question about some calculations, details: Bug in infractions/km calculation in compute_global_statistics carla-simulator/leaderboard#117

   What we want to compute is the total number of collisions / total number of km driven so the correct formula is:

   

2. weather matters for performance, details: Poor evaluation result autonomousvision/transfuser#37 (comment)

   The official CARLA leaderboard evaluation considers a secret set of weathers and towns, which are unknown to the public for benchmarking purposes.

3. event trigger is not so great, some times is hard to brake, details: Question about loss in the training phase dotchen/LAV#9 (comment)

   I found that better experts for IL to train could reduce the problem. (better rules)

   slower velocity as the author said will cause some timeout.

---

Technical:

4. CARLA will crash. reason: memory leak, haven't solved yet, details: Memory leak issue when loading a new world carla-simulator/leaderboard#81

   pls, try to make the number of route in routes.xml less than 50-100.

5. From Kait, Lens distortion attributes carla-simulator/carla#3130 Another CARLA bug that is relevant for the leaderboard depending on how your approach works.

welcome to add more. 😄

---

great open-source code for CARLA leaderboard (personally like 💖):

💖 https://github.com/autonomousvision/transfuser clear codes, easy to get points, recommend to freshman

💖 https://github.com/dotchen/LAV really big system but the greatest, recommend to seniors.

💖 https://github.com/Kin-Zhang/carla-expert kind of experts for CARLA leaderboard

[Kait0]

carla-simulator/carla#3130 Another CARLA bug that is relevant for the leaderboard depending on how your approach works.