Evaluation performance for Town05 Long from the paper table

[JunyongYun-SPA]

The above table is the performance table for town05 Long in the paper supplement. However, when evaluated with the pretrain weight provided by GitHub, the corresponding performance is not achieved. (driving score: 57.447 / route completion: 89.142)

How can I get that performance?

[deepcs233]

Hi!
The released model was only trained on a subset of the training dataset. You could collect more dataset from other towns and train it. Some hyper-parameters in the controllers may be adjusted according to the autual performance.

[JunyongYun-SPA]

Thank you for your quick response!

[YilinGao-SHU]

Thank you very much for your reply, I have another question here, why the accuracy in the ablation experiments seems to be less
than the one in the appendix when the same comparison is done on the Town05 Long, is there some detail in the paper that I am missing here?

[deepcs233]

Hi! Our experiments are conduct on LangAuto-Long, instead of Town05-long. May be some misunderstanding here :) YilinGao-SHU ***@***.***>于2023年12月22日 周五15:03写道：

…

Thank you very much for your reply, I have another question here, why the accuracy in the ablation experiments seems to be less than the one in the appendix when the same comparison is done on the Town05 Long, is there some detail in the paper that I am missing here? image.png (view on web) <https://github.com/opendilab/InterFuser/assets/98305137/52bec58a-e4a7-4171-b930-90885d39f2c6> — Reply to this email directly, view it on GitHub <#80 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEFTRR6EP3ZZQBOACCMCAKDYKUWDPAVCNFSM6AAAAAA73IEZE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRXGMYTMNZRHE> . You are receiving this because you commented.Message ID: ***@***.***>

[YilinGao-SHU]

Hi! Our experiments are conduct on LangAuto-Long, instead of Town05-long. May be some misunderstanding here :) YilinGao-SHU @.>于2023年12月22日 周五15:03写道：
…
Thank you very much for your reply, I have another question here, why the accuracy in the ablation experiments seems to be less than the one in the appendix when the same comparison is done on the Town05 Long, is there some detail in the paper that I am missing here? image.png (view on web) https://github.com/opendilab/InterFuser/assets/98305137/52bec58a-e4a7-4171-b930-90885d39f2c6 — Reply to this email directly, view it on GitHub <#80 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEFTRR6EP3ZZQBOACCMCAKDYKUWDPAVCNFSM6AAAAAA73IEZE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRXGMYTMNZRHE . You are receiving this because you commented.Message ID: @.>

Hi！
This table comes from page 7 of "Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer". Should there be no LangAuto yet?

[deepcs233]

Hi!
Sorry for the wrong reply, i just used my phone to reply you. And I thought I replied to you in that LMDrive issue. The table in pgae7 didn't share the same setting in the appexdix. Considering we need to conduct many ablation experiments, we only used a subset of our training set for the ablation studies, which may cause a performance drop.

[YilinGao-SHU]

Hi! Sorry for the wrong reply, i just used my phone to reply you. And I thought I replied to you in that LMDrive issue. The table in pgae7 didn't share the same setting in the appexdix. Considering we need to conduct many ablation experiments, we only used a subset of our training set for the ablation studies, which may cause a performance drop.

Thanks so much for you reply!