2024-09-30 ODD working group weekly meeting (208)

[Igata-ctrl]

Attendee

• Daniel Shih (National Taiwan University/TIER IV/Reference WG)
• A K wagle (Sahyadri College of Engineering & Management)
• Nithesh Shetty (Sahyadri College of Engineering & Management)
• Tran Huu Nhat Huy (Student/Tokyo Institute of Technology)
• Zain Khawaja (AWF)
• Berkay Karaman (Leo Drive)
• Ahmed Ebrahim (Leo Drive)
• David Walmroth (PIX Moving)
• M. Fatih Cirit (AWF)
• @mitsudome-r Ryohsuke Mitsudome (AWF TSC/TIER IV)
• @Igata-ctrl Hiroshi IGATA (TIER IV)

Agenda/Minutes

• On September 22, 2024, 298/340 "public road bus" and "pull out" scenarios succeeded on Autoware.universe main branch.

  • 111/142 public road bus scenarios were successful. Identical to the result of recent weeks.

    • UC-23-004-0001-With2Obstacles-ShalunRS : Ego looks like ignoring the traffic from its left.
    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops though "isBLIND:true" is set.
    • In addition to the above, 30 scenarios (including parameter variations) failed due to system errors (Simulation Health Check, Time Out etc.).

  • 187/198 pull out scenarios were successful. Two scenarios became successful after explicitly setting Ego's max speed of 4.2m/s.

    • UC-v2-F-15-00003_001_case01 : Failed for (D1=8, D2=4) case. Ego was not able to back up due to "obstacle stop".
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : Failed -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).
    • UC-v2-F-15-00103_001_case01_cmn_general : "Obstacle stop" prevents EGO from proceeding. -> The scenario intention is to use the space outside of the lane, but current Autoware.universe main is not capable of it.

• On September 22, 2024, 62/96 Dense Urban scenarios were successful on Autoware.universe scenario-dev branch.

  • Test-SafetyPool-map1 : Has been successful for a month, but failed this time. (A simple lane change did not happen.)
  • Test-dense-urban_ITRI(2|3|4) : EGO cannot use the space in th opposite lane when it passes/overtakes NPC/Pedestrians.
  • Test-roundabout1_Shalun : Ego cannot avoid collision with the NPC in the round about. -> It looks like NPC is not detected (as the red bounding box does not appear in Rviz screen).
  • Test_MRM_Fault_Injection1 : MRM not conducted upon the fault injection.
  • UC-NTR-001-0003_case1 : Vn=5.55 (slowest) case failed for the excess deceleration (<-2.499m/s2).
  • UC-NTR-001-0005 : All 3 cases failed for "Ego moved before Npc takeover finished.
  • UC-NTR-001-0008 : Failed for Vslow= 3.472 case as Ego decelerated to < 80% of Vslow. -> Became successful after the map version-up and fixing Npc start timing, etc.)
  • UC-NTR-001-0009 : Failed for as Ego decelerated to < 80% of Vslow. (After explicitly setting Ego max speed to 4.2m/s, it then causes a crash against Npc.)
  • UC-NTR-002-0001_case(1|2), UC-NTR-002-0003_case(1|2) : EGO cannot overtake NPC when Npc is fast (=2.77m/s) and failed with the excess deceleration.
  • UC-NTR-002-0004 : Failed with the excess deceleration.
  • UC-NTR-003-0002 : Failed as Ego stops twice and one of the failure conditions was met.
  • UC-NTR-004-0001 : EGO did not avoid the NPCs coming in the wrong direction. ("max_negative_acc" was not modified.)
  • UC-VRU-001-0001_case1 : Failed with unnecessary stop, or "act_check_ego_stopped_in_overtaking".
  • UC-VRU-001-0001_case2 : Failed with the excess deceleration.
  • UC-VRU-002-0001_case(1|2) : Failed with the excess deceleration.
  • UC-VRU-002-0002_case(1|2) : Failed with the excess deceleration.
  • UC-VRU-002-0003_case(1|2) : Failed with the excess deceleration.
  • UC-VRU-002-0005 : Failed with the excess deceleration.

• On September 29, 2024, 294/340 "public road bus" and "pull out" scenarios succeeded on Autoware.universe main branch.

  • 111/142 public road bus scenarios were successful. Identical to the result of recent weeks. (Strictly speaking this number was from a retry on September 30 with "Retry=1" option, though the number of success cases increased by two without any retries.)

    • UC-23-004-0001-With2Obstacles-ShalunRS : Ego looks like ignoring the traffic from its left.
    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops though "isBLIND:true" is set.
    • In addition to the above, 30 scenarios (including parameter variations) failed due to system errors (Simulation Health Check, Time Out etc.).

  • 185/198 pull out scenarios were successful. Two scenarios became successful after explicitly setting Ego's max speed of 4.2m/s.

    • UC-v2-F-15-00002_001_case03 : 1/6 case failed as Ego stopped unnecessarily. -> Suceeded after explicitly setting Ego max speed = 4.2m/s.
    • UC-v2-F-15-00003_001_case01 : 3/4 cases failed as Ego was not able to back up as intended".
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : Failed -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).
    • UC-v2-F-15-00103_001_case01_cmn_general : "Obstacle stop" prevents EGO from proceeding. -> The scenario intention is to use the space outside of the lane, but current Autoware.universe main is not capable of it. -> Succeeded this week! (The scenario has not been modified.)

• On September 29, 2024, Dense Urban scenarios failed with build error. @mitsudome-r and @brkay54 are kindly retrying them.

• [recap of the discussions until last week]

• Discussed the improvement requests for Evaluator.

  • Will continue discussion next week, Armağan's initial ides included embedding the use case definition (slides) and the scenario (NPC behavior) preview by the time-slider in the scenario editor screen, etc.
  • @Igata-ctrl explained that Evaluator is not an OSS but a product from TIER IV, so the development team has a long wish list form TIER IV and OEM projects.
  • May need to clarify the content of the wish list from AWF before discussing the priority with TIER IV CI/CD team. @Igata-ctrl to try to collect recent situation with the team.

• Regarding the needs for calculating the accurate RSS safety distance from Vf(Npc) and Vr(Ego), TIER IV Mapping and Localization team will be able to add the functionality using UserDefinedValueCondition like UC-v2-F-15-01001_001_case01_cmn_general case. It may take more than two weeks depending on their resource margin/availability. -> @brkay54 to kindly submit an issue regarding this request mentioning Takeuchi-san of TIER IV (@kosuke55) who is working on UserDefinedValueCondition.

• For the current scenario suite for weekly execution above, the RSS definition for minimum safety distance and 2.499m/s^2 for maximum comfortable deceleration have been applied.

• @Igata-ctrl got the information in Simulation working group meeting that the resource allocated to Scenario Simulation v2 on the cloud may not be sufficient and it can make the simulation result not trustworthy enough. We may need to analyze the Simulation Log file to see if there are many waring messages in there when we get unexpected simulation result.

• @Igata-ctrl to investigate if we can use real time speed of Ego and Npc to calculate RSS Safe Distance more accurately.

• Discussion (RSS distance and comfortable deceleration) pages have been added to the Use cases for Dense Urban Area 2024 presentation.

• Reached a consensus to use RSS definition for minimum safety distance and 2.499m/s^2 for maximum comfortable deceleration for low speed shuttle.

• Discussion of the condition/threshold setting so that we can better track the Autoware maturity through scenario simulation.

• After the discussion, it turned out "min_acc" parameter should better be reverted to original value of -1.5 (this week).

• Current discussion with Safety Pool

  • [Map] Successfully converted a sample OpenDRIVE map from Safety Pool database to Lanelet2, but we have following issues. Though Mohammad (Deepen.ai/AWF Safety Assurance WG lead) proposed in May TSC that Autoware should consider accepting OpenDRIVE maps in a direct fashion as it solves all these issues, given that we may need to map-less autonomy, it may not worth switching map format from now.
    • Speed limit, lane change flag and LHT/RHT are lost through the conversion.
    • The converted Lanelet2 map contains a lot of unnecessary points on line strings. (e.g. 16,000points on 1 km one way straight road)
  • [Scenario] Autoware requires EGO's destination in each scenario, while all of the Safety Pool scenarios do not have the destination. Though Safety Pool once suggested to run Autoware in the “roaming” mode which does not require the destination in the scenario, it sounds unrealistic for Autoware to operate without destination as the destination is almost always an important part of the use case/scenario intention. (e.g. turning left/right).
    • In the Safety Pool Webinar, they demonstrated Autoware running with Safety Pool scenario, but they say that it requires some special settings for it and it is not scalable.
  • [Scenario file format] Though both Safety Pool and Autoware (Scenario Simulator v2) use OpenSCENARIO v1.x (a.k.a. OpenSCENARIO XML) standard, Autoware (Scenario Simulator v2) uses ”.yaml” file to better handle the parameters, while Safety Pool uses “.xosc” file, which is the default of the OpenSCENRIO standard. If the number of scenarios is not too big, we might be able to convert them from .xosc to .yaml format manually. (Currently, no automated converter is available.) If the scenario simulator v2 is installed on-premise, it will accept the .xosc file in a direct fashion.
  • [Scenario availability] After signing up for the Safety Pool database, the number of the accessible/visible scenarios are limited. 32,588 scenarios are visible but most of the scenarios are marked with "access restricted" and only 1957 scenarios are accessible. The total number of the scenario which Safety Pool claims is one million. Still waiting for an answer from Safety Pool

Action Item

Documents

• The bus ODD use case list which was assigned to the members is here.

• The bus ODD use case list has been also uploaded here to make it visible to anyone on the net.

• The ODD working group shared document folder is here. [Restored!]

• Discussions and Q&As in AWF Discord ODD WG channel are also encouraged.

Tools

• [Autoware Evaluator (CI/CD pipeline)]

  • Cloud based DevOps (integration of the development tools including the scenario editor and the scenario simulator below)
  • A product from TIER IV and offered for the official Autoware Foundation projects like Cargo Delivery and Public Road Bus, etc.
  • The user guide is available TIER IV document site
  • As it consumes AWS resource, the (batch) weekly execution of the scenario simulation is managed by the Software/ODD WG leads

• [Scenario Simulator V2 (Scenario testing framework)]

  • Stand alone scenario simulation tool
  • An OSS from TIER IV freely available for any Autoware developer/researcher
  • The documentation of the Scenario Testing Framework (open sourced from Tier IV) is on GitHub

• The GUI Scenario editor

  • Web based GUI scenario editor freely available from TIER IV
  • You can create and export scenarios with this web interface
  • The user guide is available TIER IV document site

• [TIER IV account]

  • The working group members who are interested in creating and testing scenarios are advised to create a free TIER IV account here.
  • Once you have created your account, please let @Igata-ctrl know the (long) User ID which appears on your login page. After @Igata-ctrl registered you to the AWF group in the Evaluator (CI/CD pipeline), you can go to AWF Autoware Evaluator page to see the simulation results, create/edit scenarios, etc.
  • If you already have a TIER IV account, your 4-digit User ID continues to work, so you do not need to register to TIER IV account again.

Administrative

• The two meetings on December 25th (Christmas day) and January 1st (New Year day) have been cancelled.

• Please check the ODD WG wiki page.

• Recurring weekly meetings have been scheduled. Please check Autoware Foundation events calendar and add this calendar to your own Google calendar by clicking the right bottom button and/or add your contact to ODD WG invitation group to receive invitations for future meetings.

• ODD WG meetings are held weekly in the following single time slot.

  • 7:00am, Monday (PST) / 6:00am, Monday (PDT) US Pacific Time
  • 10:00am, Monday (EST) / 9:00am, Monday (EDT) US Eastern Time
  • 4:00pm, Monday (CEST) / 3:00pm, Monday (CEDT) Poland time
  • 5:00pm, Monday (TRT) Turkey time
  • 10:00pm, Monday (CST) Taiwan time
  • 11:00pm, Monday (JST) Japan Time