Summary Report of VMAD-SG1(scenario) 15 - PowerPoint Presentation

1. Summary Report of VMAD-SG1(scenario) 15th meetingJuly 12-13, 2021 Takashi NAONO (MLIT, Japan)on behalf of SG1 leader1VMAD-19-03

2. Summary Report 1/221. Date, Time and Participants Date: 29 June 2021 Time: 20:30-22:20 (JST), 13:30-15:20 (CET) Participants: about 30 members 2. Discussion of each “Outstanding issues” SG1 discussed issues in outstanding issues list (7,8, and 10-14). SG1 leader introduced “NATM initial draft” as a proposal for future update of NATM MD. Some of the comments were as follow. - ODD issue can be dealt with “Coverage” issue. - Any template for scenarios can be valuable in order to combine different scenarios in a structured way. Nonetheless, the technology neutral perspective should be considered. - How to update the scenario catalogue depends on how the scenario catalogue is used. SG1 continue to discuss the issues at the following SG1 meetings.  

3. Summary Report 2/233. Discussion about Abstract scenario SG1 will continue discussion in the next meeting. (No discussion due to time limitation) SAFE will provide some example of abstract scenario in order to share the image and purpose of the proposal.4. Change of SG1 leader SG1 leader is changing from Shumpei MIYAZAKI to Hidenori NONAKA.  5. Next meeting will be held on 3th August 12:30-14:30 CEST. (after 15th session, Leader invited additional session on 13th July 14:00-14:30) [Request to members]- Submit any comments to the open issues including NATM initial draft in excel sheet (VMAD-SG1-15-02-rev1). (deadline: 26th July)- SAFE will provide some example of scenario template and abstract scenario based on Annex 2 of NATM MD (ECE-TRANS-WP29-2021-61e) in order to share the image and purpose of the proposal.- The submission should be sent to Hidenori NONAKA (h-nonaka@jasic.org) and Yuko BABBISH (babbish@jasic.org)

4. (reference) List of “Outstanding issues” 4copy from VMAD-SG1-15-02　 Possible new issuesTarget Completion dateCommentsNATM initial draft1Which pillars (simulation, track test, real-world test, audit) should be used to assess scenarios?　[7*][8]2021.9(JP) Scenarios that are used to assess comprehensiveness and scenarios that have risk to cause accident should be assessed by simulation pillar, not by track test nor real-world test.It should be considered that real-world test cannot test specific scenarios.Although this issue cannot be concluded within SG1, SG1 can report the comments to VMAD.(RU) It is premature to discuss this issue now. The general methodology on which pillar to use to assess the specific scenarios could be discussed at the VMAD level once the scenario catalogue becomes available. The decision should be based on feasibility.(JP) Since the pros and cons of each pillar are described in NATM MD, it is not necessary to discuss this issue within SG1.(DE) All pillars can potentially be used to assess scenarios. It might also be necessary to use multiple pillars to assess a certain scenario, either because different output parameters might be necessary or for reinforcing validation.(SAFE) From a technical capability perspective, all pillars can be used to asses scenarios. scenarios can me created on a test track, and scenarios can be identified during analysis of real world test drive. The usage of the scenario based assessment per pillars, and what to actually assess with each pillar, should be determined by VMAD and FRAV. No text for NATM (This issue should not be described in SG1 section.)2What is "coverage" of scenario?What is "sufficient" coverage?How to ensure sufficient coverage?[10][13]2021.9(JP) Coverage means how much proportion scenario catalogue cover compared to real world situations.Regrading highway case, 24 scenarios in NATM MD Annex2 "Interaction with other vehicles/objects" can provide sufficient coverage by considering various aspects by parameters. (By setting appropriate parameters, situations other than cut in can be described )Sufficient coverage can be ensured by setting appropriate parameters.(RU) We should not question the "coverage" of a specific scenario. We should consider a package of scenarios, which should cover traffic situations of a high probability of occurrence. This issue could be addressed when the scenario catalogue becomes available.(DE)„Coverage“ as a theoretical quantity is necessary to make a justified statement towards the residual risk, resulting from introducing certain automated driving functions into traffic. Coverage can belong to the scenario database / collection of scenarios in total, but can also belong to a certain scenario in order to express the scenarios ability to represent similar scenarios.Coverage = share of considered cases out of basic population / total basic populationSufficient with regard to residual risk has to be decided outside of VMAD and FRAV.Any judgement on coverage need to be preceded by a sound definition of the basic population.(SAFE) Agree with (DE) . Coverage gives you a metric of "What was tested" and "What is left over as not tested" ( which can mapped to residual risk ). Coverage can be measured once you define the basic population. The basic population can be defined by using granularity of Interest. As currently VMAD/UNECE is looking only at functional scenarios, which have no physical parameters specified , initial coverage can be measured probably per the content of the scenario catalog ( how much of the catalog have you tested ) Once specific regulation introduces parameter ranges for testing ( e.g. UN REG 157 introduced 0-60Kph as a range ) , more coverage parameters can be define. The definition of sufficient coverage is to be set by FRAV - as a functional requirement for safety threshold. 5.18 “Coverage” of scenario catalogue, which means considered cases out of total cases, is an important aspect in order to estimate the effectiveness of the scenario catalogue. Since it is desirable to ensure as wide coverage as possible, it is important to enhance the scenarios.

5. (reference) List of “Outstanding issues” 5copy from VMAD-SG1-15-02　 Possible new issuesTarget Completion dateCommentsNATM initial draft3Which scenarios are required to validate the functional safety requirements established by FRAV?[11]2021.9Need FRAV output(JP) Avoiding accidents and collision mitigation should be considered in the first step.Other aspects (e.g. HMI, failure) should be considered in the second step.(RU) It is not reasonable to discuss this issue until FRAV introduces the functional safety requirements. But in general, any specific scenario will allow validating the requirements partially. A combination of the assessment results would give more or less a full picture of vehicle behaviour and compliance with the requirements.(DE) Will depend on FRAV requirements.(SAFE) This can be answered once the functional safety requirements are introduced. I would assume that once Functional requirements are known, regulation threshold for sufficient performance is set, and ODD is understood - one will be able to map the scenarios and their content to the required testingNo text for NATM (This issue should not be described in SG1 section.)4Scenarios not covered by scenario catalogue (Should authority require evaluation of scenarios that are not covered by scenario catalogue?)[12]2021.9(JP) It should be possible that authority can request additional scenarios in addition to predefined scenario catalogue.In this case, the request from authority should be reasonable.If "scenarios not covered by scenario catalogue" are identified, they should be included in the scenario catalogue.(RU) In general, the authority has a right to require validation regarding any scenario (included or not included in the catalogue), because the authority must be certain that a vehicle is safe on the road. To address this issue and not cause the authority to introduce its own scenarios, the scenario catalogue should be extended as much as possible and all new scenarios introduced by the authorities should be included in the general scenarios catalogue.(SAFE) The answer to this issue should address the question of geographically or country unique scenarios. ( think about emergency vehicles differences - for example ) . In general, the UNECE catalog is introducing a common set or scenarios for bi-lateral certification recognition. Once a local authority is introducing additional requirement , it should be justified , and potentially introduced into the common catalog. (DE) Scenarios not covered by the catalogue can be requested by the authority if such scenarios belong to the basic population. Such scenarios should be deemed „candidates“ for future scenario database updates.5.19 Scenarios not covered by scenario catalogue should be assessed if authority considers they are necessary for ADS safety, but there are nonetheless concerns that unidentified scenarios may be required during certification. If scenarios not covered by scenario catalogue are identified, they should be included in the scenario catalogue.5How to deal with "overfitting" problem?[14]2021.9(JP) Random sampling in the track test should be available to authority/TS with the condition that the request is reasonable and not too much burden to manufacturers.(RU) This is not a problem. It is expected and not bad, if manufacturers would "train" their vehicles to pass the scenarios. If a vehicle passes all the scenarios from the catalogue, one can assume that vehicle is safe. To reduce the associated risks, a vehicle should pass randomly selected scenarios, and the scenario catalogue should be extended as much as possible.(DE) In general overfitting can be reduced by using cross validation. Using one (or preferable multiple) random samples – serving as validation data sets - should avoid overfitting.5.20 Random sampling can be justified in order to avoid overfitting. Although the more cases of random sampling are preferable for the credibility perspective, the burden to manufacturers and authority (e.g. technical service) should be considered reasonably.

6. (reference) List of “Outstanding issues” 6copy from VMAD-SG1-15-02　 Possible new issuesTarget Completion dateCommentsNATM initial draft6Should scenario catalogue includes unusual situations (e.g. wrong way driver)?[16]2021.6(JP) Inappropriate actions of other road users (e.g. wrong way driver, sudden crossing) should be included in scenario catalogue.However, it does not mean that every collision should be avoided because the requirement for ADS depends on the situation and required level of safety.In addition, it is not practical that scenario catalogue include every situation, scenarios should be defined with some assumptions.(RU) Yes, such unusual situations are scenarios, as well. The reason for the inclusion of those in the catalogue is the relatively high probability of their occurrence.(SAFE) The leading principle is that the scenario catalogue should include foreseeable, reasonable situations in which the ADS is to demonstrate safe behavior. A definition on the threshold ( probability of occurrence) for "unusual situation" should be developed. The example of the wrong way is one example that is exercising the perception system of the ADS, One can think of more reasonably foreseeable situation - like animals on highway. In general, you need to test for reasonably foreseeable , unusual situation. It is up to VMAD and FRAV to define these. (DE) „Unusual“ is a subjective expression. Question can be answered by quantifying / objectivation of this expression. If such scenarios need to be included it will (also) be a question of coverage (see 2).5.21 Inappropriate actions of other road users (e.g. wrong way driver, sudden crossing, and significant excess of speed limit) are not necessarily excluded from scenario catalogue. This does not mean that all collision should be avoided because the requirement for ADS depends on the situation and required level of safety.7What aspect should be dealt with functional/logical scenarios?2021.9(JP) The detail of functional scenario should be defined in logical scenario. (e.g. curve, light, weather)(RU) In the case of a dispute around the content of the functional scenario, the arguable elements could be shifted to the next level of specification, i.e., logical scenario, because the final goal is the elaboration of a catalogue of concrete scenarios.(DE) Functional scenarios should be defined at a high abstraction level, trying to minimize country specific particularities.(SAFE) Functional scenarios should focus on the specific interaction between the participants of the scenario, and any unique characteristics of the environment that may be of importance ( e.g. Scenario happens within a tunnel ). The transformation of the functional scenario to ABSTRACT scenario ( formalization ) and LOGICAL scenario ( parameterization range ) is to be done per specific regulation usage and safety requirements. No text for NATM (NATM already includes the description that logical scenario deals parameters.)8How to deal with various parameters (e.g. perception, vehicle disturbance(e.g. road surface condition, slope, wind)) during certification?2021.9Need FRAV output(JP) Parameters should be dealt in logical scenario.(RU) Those are variables characterising scenarios. At the compliance assessment process, they should be selected randomly as the vehicle must be safe in all conditions that might happen.(DE) Parametrization of scenarios needs to fit with FRAV requirementsNo text for NATM (NATM already includes the description that logical scenario deals parameters.)

7. (reference) List of “Outstanding issues” 7copy from VMAD-SG1-15-02　 Possible new issuesTarget Completion dateCommentsNATM initial draft9How to deal with different scenarios for different countries/regions?2021.6(JP) Scenarios for every country/region which are aimed for mutual recognition should be assessed.It should be allowed that the manufacturers can limit the target country/region in order to limit the applicable scenarios. (e.g. UNR157(ALKS) can limit the target country.)(RU) Different countries/regions may introduce specific scenarios reflecting on the traffic situation in those countries/regions. Those scenarios should be considered relevant for the vehicles supplied to the markets of those countries/regions. However, the number of such scenarios should be reduced as much as possible. As a countermeasure, the scenario catalogue should be extended as much as possible and all local scenarios should be included in the general scenarios catalogue.(DE) Country specific scenarios need to be respected. It is expected that country specific scenarios will become more relevant turning to logical and concrete scenarios. For functional scenarios country characteristics should be kept at a minimum.5.22 Country specific scenarios should be respected. There is a possibility that manufacturers can limit the target country/region in order to limit the applicable scenarios (e.g. UNR157 can limit the target country.). In order to harmonise globally, country/region specific scenarios should be minimised.10What is pass/fail criteria for scenario-based testing?2021.9Need FRAV output(JP) SG1 should wait the conclusion of FRAV. SG1 can discuss after the progress in FRAV.(RU) It is premature to discuss this issue now. In general, the pass criteria could be no accident occurrence and vehicle motion parameters (e.g., longitudinal deceleration) should not exceed the acceptable values. The Guidance on traffic disturbance critical scenarios for ALKS (UN R 157) should be taken into account as an example.(DE) Need FRAV input. Scenarios deliver the base for testing ADS. Pass/ Fail criteria are an issue of NATM as a whole.No text for NATM (This issue should not be described in SG1 section.)11ODD based scenario generation and/or linking scenario coverage as a function of ODD (How to generate scenario based on ODD? How to consider coverage based on ODD?)2021.9(JP) Regarding the coverage, the situation is the same as "coverage issue (2 of this outstanding issues)".Regarding the ODD exit, the situations should be assessed as necessary. (e.g. "Highway ODD" can exclude urban road because ADS can exclude any possibility to run in urban road. "With lane marking ODD" cannot exclude disappearance of lane marking because lane marking can suddenly disappear.)(RU) In general, the scenarios should not be based on ODD. However, to verify the vehicle performance in the particular scenario, the variable parameters of that scenario should be adjusted as such to ensure that the vehicle would be within its ODD. Of course, a vehicle should go through different scenarios within its ODD. Besides that, the vehicle performance in some specific scenarios should be validated outside ODD and when reaching ODD limits to ensure the proper DDT fallback of the vehicle. (DE) Should be dealt with in 2 – Coverage.(SAFE) Scenarios should be defined independent of ODD, and can be applied to specific ODDs. Scenarios may be categorized per ODD ( i.e. declared as usable for specific ODD ) - since there will be ADSes defined per ODD, for example, Scenarios for ALKS may not match and can not be used in an urban ODD. The discussion should not focus on how to define ODD, but rather on how to categorize the scenarios per ODD. As for Coverage, - same answer as issue #2. ( you can defined coverage per ODD ). No text for NATM (This issue is included in "coverage" issue.)

8. (reference) List of “Outstanding issues” 8copy from VMAD-SG1-15-02　 Possible new issuesTarget Completion dateCommentsNATM initial draft12Update of functional scenarios (i.e. Annex2 of NATM MD).[15]Anytime(JP) This issue should be discussed after SG1 concluded above issues.(RU) The top priority and the general objective of SG1 is the development of traffic scenarios starting from functional scenarios to elaborate on a full catalogue of those. This is a complicated and long-lasting task, however, we should develop a methodology and start this activity ASAP. As the initial step please refer to the contribution by the Russian Federation VMAD-SG1-07-03. All other SG1 "outstanding issues" are considered of lesser priority and discussing those or seeking new ones is counterproductive time consumption for the time being.(DE) Should be foreseen. List of candidates should be collected during vehicle operation. New functional scenarios should have the ability to be harmonized across contracting parties.(SAFE) This issue relates to #14. The Russian federation proposal of a methodic way to construct scenarios by separating layers and independently filling each on of them can serve as a basis, while the challenge is to fill layer 4 with various interactions ( as can be seen in the contribution from Japan ). However, there needs to be a way to map these to FRAV requirements, and understand if the there is any correlation between the two. Obviously, the CP need to agree on this contentNo text for NATM (No concrete proposal at this moment.)13Should any template for scenario be defined? If so, how will it be?　(JP) It is not necessary to define template of scenarios because the regulations should be technology neutral under WP29. Any template for scenarios should be discussed in the standard field.(SAFE) A template implies a structure or an outline of a document. A template will enable unified representation of the functional scenarios in the proposed UNECE catalog. As functional scenarios are defined as "natural language" with no formal structure - one must have some structure in place. In order for the scenario catalog to be useable, there should exist a definition of what content and in what form a scenario will be represented there. If one looks at the 5 different contributions that were made to the scenario catalog, they are all very different - and can not be combined. Thus the proposed scenario catalog should articulate how a functional scenario looks like ( this is functional scenario, in free language , but the outline should be defined). A naive example for a template can be : Title: Type of Road: Type of Actors: Environment Conditions: Interaction and actions: 　14What is the maintenance procedure (e.g. regular update) of scenario catalogue? By whom?　(JP) The scenario catalogue which belongs to WP29 should be discussed and updated by WP29. (SAFE) Agree that that WP.29 Should set and define maintenance procedure. Some consideration that should be elevated to VMAD, GRVA and WP.29: The Scenario Catalog should be maintain by an entity which will be capable of adding , replacing and modifying scenarios in the catalog. A decision process ( agreed by all CP) of applying changes and modifications to the catalog should be defined. Such an Entity should be authorized to make decision on acceptance and rejections. This will be required with the evolution of the technology and as additional ADS regulations will be developed. ( Today, for amending a regulation, one has to get WP.29 approval , do we expect every change to the scenario catalog to go to WP.29 voting ? )