Origin LAV v1 version Results [~8pts]

[Kin-Zhang]

Below is the v1 version, not! the v2 version, I didn't test on the update version. If someone did, please comment in the discussion.

---

Even after discussing with the origin author, it still cannot reproduce the LAV result in the online leaderboard. Reasons list on the behind.

Related issue comments:
dotchen/LAV#9 (comment)
dotchen/LAV#9 (comment)

Here is the resulting picture with origin LAV repo provided weights files' link and default parameters config link:

expand to see the result image (Driving score: 8.188, Route completion:52.401)

Please note again. This result is from origin LAV repo provided weights files' link and default parameters config link

Reasons:

• As the author mentioned in the LAV paper, the LAV results in the online leaderboard trained at the whole dataset.

• Appendix on arxiv version 2 [section B.2]:

  Our online CARLA leaderboard entry additionally process temporal information by concatenating LiDAR scans normalized to the current vehicle pose. Vehicles poses are tracked across time via GPS signals, smoothed by an Extended Kalman Filter (EKF).

[dotchen]

Hello,

LAV author here. I came across this link on the CARLA leaderboard website.
Please do note that the provided weights are trained on a subset of all towns and are indeed different from our leaderboard submission.
Our paper has also detailed the differences in the appendix: https://arxiv.org/abs/2203.11934.

We are currently running some last experiments/improvements over certain design choices, and I will refactor these updates, as well as codes that you can directly submit to reproduce our leaderboard numbers to our github repo. This will be done after my thesis proposal exam is over, by the end of May.

Thank you for your patience.

[Kin-Zhang]

Thanks for replying, Yes, that's why I highlight the origin weight files provided link.

But even with all dataset training, I still cannot reproduce. related issue: dotchen/LAV#9. We discuss it in this issue, for some training loss and default settings etcs.

Never mind, since the provided code is enough for us to do a reference, it's a great backbone. better than others. I'd like to thank you for providing LAV to the CARLA leaderboard community. That's why after combining the results greater than I did before.

Best wishes for your thesis preparation.

[dotchen]

Sorry for the delay, the leaderboard weights are now released (also with a faster agent option for improved inference speed).

[dotchen]

Thank you for your understanding. I would also like to highlight our current leaderboard entry uses a stack of LiDAR points (described in the paper), and this is currently not in the Github repo.

Once I am done with the exam I will work on refactoring it among other things to the repo.

[Kin-Zhang]

I would also like to highlight our current leaderboard entry uses a stack of LiDAR points (described in the paper), and this is currently not in the Github repo.

Sorry to bother you. But after I review the paper, I still didn't get the meaning of a stack of LiDAR and paper here corresponds to codes here

expand to see screenshot of paper

https://github.com/dotchen/LAV/blob/2d00c6276d1a72f45627b4b9b418cd2ccad0b994/lav/models/point_pillar.py

[Kait0]

He was referring to section B.2. CARLA Leaderboard in the appendix:

Our online CARLA leaderboard entry additionally pro-
cess temporal information by concatenating LiDAR scans
normalized to the current vehicle pose. Vehicles poses are
tracked across time via GPS signals, smoothed by an Ex-
tended Kalman Filter (EKF).

Note that this section got only added in version 2 of the arxiv release. If you are reading version 1 it won't be in there.

[Kait0]

I personally thought that was quite an interesting change, because methods that use imitation learning usually perform better with single time step inputs. So I am quite curious about the details of this implementation, why it works in this architecture or if you have ablations that actually show that this helps the performance of the system.

[dotchen]

Unfortunately due to several constraints we have not run a detailed ablation over this. Intuitively, it provides denser point clouds and temporal information which is critical to predict other vehicles' trajectories. It is true that it could potentially cause "causal confusion" for ego planning. However, with a distillation framework like the one we used here, you might be able to get away with certain tricks.

[Kin-Zhang]

He was referring to section B.2. CARLA Leaderboard in the appendix:

Our online CARLA leaderboard entry additionally process temporal information by concatenating LiDAR scans normalized to the current vehicle pose. Vehicles poses are tracked across time via GPS signals, smoothed by an Extended Kalman Filter (EKF).

Note that this section got only added in version 2 of the arxiv release. If you are reading version 1 it won't be in there.

Thanks! I didn’t find that they update to version two on arxiv😅.

[dotchen]

Check the appendix and our demo video.

…

On Wed, May 18, 2022, 6:00 AM Kin ***@***.***> wrote: I would also like to highlight our current leaderboard entry uses a stack of LiDAR points (described in the paper), and this is currently not in the Github repo. Sorry to bother you. But after I review the paper, I still didn't get the meaning of a stack of LiDAR and paper here corresponds to codes here [image: IMG_1213] <https://user-images.githubusercontent.com/35365764/169044258-4075f2c9-a24a-4f6d-a27f-dd91000e30a0.jpg> https://github.com/dotchen/LAV/blob/2d00c6276d1a72f45627b4b9b418cd2ccad0b994/lav/models/point_pillar.py — Reply to this email directly, view it on GitHub <https://github.com/Kin-Zhang/LAV/discussions/1#discussioncomment-2775761>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACPV4FFEBZQZONAOU2Y7TLTVKTSYFANCNFSM5U5XPOQA> . You are receiving this because you commented.Message ID: ***@***.***>

[Kin-Zhang]

I personally thought that was quite an interesting change, because methods that use imitation learning usually perform better with single time step inputs. So I am quite curious about the details of this implementation, why it works in this architecture or if you have ablations that actually show that this helps the performance of the system.

Our online CARLA leaderboard entry additionally process temporal information by concatenating LiDAR scans normalized to the current vehicle pose.

I also have the same question, since the sentence didn't illustrate more about it. For example, how many timestamps of LiDAR points should we concat? And for IL, usually, we adopt the same setting as previous training. Maybe we could try multip images of time into the training also.

[Kait0]

Temporal data in imitation learning is a tricky topic. Naive approaches like simply concatenating images often don't work.
If you are interested in the topic this paper from Nips explains the problem well: https://proceedings.neurips.cc/paper/2019/file/947018640bf36a2bb609d3557a285329-Paper.pdf

[Kin-Zhang]

yes, I knew this paper. Here are more ideas and references:

• Maybe it's not simply concatenating but try to understand as the video understand field did. I3D paper
• And sometimes just adding channels may also have effects as the prediction field did. But I may prefer combining with velocity information one like MP3 motion field warps the occupancy over time. but it didn't open-source and hard to reproduce completely for me.
• Some BEV detection perceptions also have ideas on how to use the temporal in training. BEVFormer: Learning Bird’s-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers