The European Union (EU) released a draft version of its legislation for vehicles with an Automated Driving System (ADS) in early April 2022. This column is focused on providing an overview of the ADS compliance assessment.
The EU ADS legislation draft has two main parts: ADS performance requirements and ADS compliance assessment. The ADS performance requirements specify what capabilities an autonomous vehicle must have to receive a type–approval in Europe. My previous column provided an overview of ADS performance requirements.
The ADS compliance assessment specifies how an autonomous vehicle (AV) will be evaluated, audited, and tested before it will get type–approval.
The next table has a short overview of how the two parts differ. The chapter name in the legislation draft is listed in the table — Annex 2 for the ADS performance requirements specifications and Annex 3 for the ADS compliance assessment specifications.
The ADS capabilities description accounts for 10 pages in the legislative draft. The ADS compliance assessment description section accounts for 50 pages of the ADS legislation draft or 5 times larger than the ADS performance requirements. It is interesting that the ADS evaluation, assessment, and testing has much more complexity and specifications than the ADS performance requirements.
ADS performance requirements summary
The ADS requirements use five traffic scenarios to describe the capabilities needed. The ADS must perform the entire dynamic driving tasks (DDT) for all traffic scenarios.
It also specifies functional and operational safety conditions to be met. Minimum risk maneuvers (MRM) are important for avoiding crashes and dangerous situations.
The ADS must also have built–in cybersecurity and software management systems. An event recorder is needed to keep track of the performance of the ADS. A detailed operating manual is required as part of the type–approval application.
ADS compliance assessment
The above table has summary information on ADS compliance assessment. The key is that the assessment will be performed by a type–approval organization. Any requirement of the ADS performance may be checked by tests performed by the type–approval authority as specified in the ADS compliance assessment. Every vehicle with ADS capabilities must pass these tests before they can be sold in EU countries.
The compliance assessment is quite complex with detailed descriptions of how each ADS will be evaluated, as shown in the below table. The table includes the sections in the legislation draft where the assessment descriptions are contained.
There are five assessment categories: Operational design domain (ODD) scenarios; ADS system design and documentation; on–road testing of ADS safety; ADS modeling and simulation (M&S) capabilities; and safety performance during the ADS lifetime. Each assessment category is summarized below. However, it is not feasible to describe the many assessment details of the legislative draft. For such information, the reader must analyze the ADS legislation draft themselves.
ODD scenario assessment
The ODD scenario assessment includes details on safety parameters for operations such as lane change, merging, turning, cross–traffic, and other driving operations. The assessment covers different driving environments including highways/motorways, urban and rural roads, and behavior of road users.
The ADS legislation draft has extensive description of the traffic scenarios included in the previous column on traffic scenarios: normal, critical, and failure–emergency operation. Much of the discussion is expected behavior during on–road ADS testing performance.
An example of the assessment details included in the legislative draft is shown in the next figure.
For the ODD of an ADS, both a data–based and a knowledge–based approach can be used to generate corresponding traffic scenarios. A knowledge–based approach utilizes expert knowledge to identify hazardous events systematically and create scenarios. A data–based approach utilizes the available data to identify and classify occurring scenarios.
ADS safety concept assessment and audit
This section is focused on assessing the ADS safety concept and auditing the manufacturer safety management system. The type–approval authority shall verify through targeted spot checks and tests that the ADS performance requirements are actually implemented based on submitted documentation.
This means the ADS operates in its ODD in a way that is free of unreasonable safety risks to the vehicle occupants and other road users under fault (functional safety) and non–fault (operational safety) conditions.
- ADS documentation — The manufacturer shall provide documentation which provides access to the ADS design and other vehicle systems the ADS directly controls, as well as off–board hardware/software and remote capabilities. Documentation shall be brief yet provide evidence that the design and development had the benefit of expertise from all the ADS fields which are involved. The documentation section discusses a long list of requirements for all aspects of the ADS software, hardware, human–machine interface (HMI), data elements, algorithms, schematics, functional operation, and much more. About 80 documentation items are listed. Copy of proprietary data is not requested, but inspection of such information is required.
- Verification and tests — Based on the ADS documentation, the type–approval authority shall request tests to be performed including ADS functional operation and ADS safety concept verification. The type–approval authorities shall check scenarios that are critical for the object and event detection and response as well as the decision–making and HMI functions of the ADS. The verification results shall correspond to the documented summary of the hazard analysis to confirm that the safety concept and execution follow regulation requirements.
- Safety management system — The manufacturer shall demonstrate that its safety management system (SMS) has effective processes, methodologies, training, and tools that are up to date and being followed within the organization to manage the safety and continued compliance throughout the ADS lifecycle. The design and development process shall be established and documented including SMS, requirements management, requirements’ implementation, testing, failure tracking, remedy, and release. The manufacturer shall also ensure effective communication channels between manufacturer departments responsible for functional/operational safety, cybersecurity, and other relevant vehicle safety disciplines. The Certificate of Compliance for SMS remains valid for three years unless it is withdrawn. In due time, the manufacturer must apply for a new or for an extension of the existing SMS Certificate of Compliance.
- Safety concept reporting provision —The manufacturer’s reporting of the safety assessment of the ADS safety concept and audit of the safety management system shall be performed in a manner that allows traceability. The type–approval authority shall issue the safety assessment results to be annexed to the type–approval certificate based on the documentation provided by the manufacturer.
ADS pass–fail testing
Pass/fail criteria to assess ADS safety shall be based on the ADS capability requirements described in the previous column (Annex 2 in the legislation draft). The requirements are defined in such a way that the pass/fail criteria can be derived for a specific set of test parameters, for all safety–relevant combinations of parameters and the specified operating range (e.g., speed range, longitudinal and transverse acceleration range, radii of curvature, brightness, number of lanes).
The test site shall comprise characteristics that correspond to the specified ODD of the ADS. The tests must be carried out safely and without any risk to other road users. Tests shall be carried out under different environmental conditions, within the limits of the defined ODD for the ADS.
The subject vehicle shall be tested with any permissible vehicle load. No load alteration shall be made once the test procedure has begun. The manufacturer shall demonstrate, through the use of documentation, that the ADS works at all load conditions.
Type–approval testing may be carried out based on simulations, maneuvers on the test track, and driving tests on real road traffic. However, it may not be based solely on computer simulations.
A large variety of test scenarios will be done on a test track to assess the performance of the ADS — a list of 13 tests is included with many variations for some tests. The specified tests include lane keeping, lane changing, collision avoidance, emergency braking avoidance, following lead vehicle, lane cut–in by another vehicle, lane cut–out due to stationary vehicle in lane, parking, parking facility navigation, and specific motorway scenarios.
ADS M&S assessment
The key to this section is establishing a framework for credible assessment of developing and using virtual toolchains in ADS validation. The legislative draft lists five properties to achieve credibility of M&S:
- Capability: What can the M&S do, and what risks are associated with it?
- Accuracy: How well does M&S reproduce the target data?
- Correctness: How sound and robust are M&S data and algorithms?
- Usability: What training and experience is needed?
- Fit for purpose: How suitable is the M&S for the ODD and ADS assessment?
At the same time, the credibility assessment framework must be general enough to be used for different M&S types and applications. It also defines the envelope of how a virtual tool can be used for assessing ADS safety and capabilities.
The legislation draft includes a credibility assessment framework. This framework presents a way to assess and report the credibility of M&S based on quality assurance criteria. A graphical representation of the relationship between the components of the credibility assessment framework is shown in the next figure.
After this figure, the ADS legislation draft has over six pages of discussions on M&S technical issues including multiple management issues, data inputs, data quality, modelling assumptions, model verification, code verification, validation, and accuracy.
The last M&S portion is a discussion of the documentation structure, where two items stand out:
- The manufacturer shall produce a document (“simulation handbook”) structured to provide evidence for the topics presented.
- The documentation shall be maintained throughout the whole lifecycle of the M&S utilization.
This section contains an extensive discussion on M&S technologies, and it is not feasible to cover all the details in this column. It should be studied by experts in this field.
ADS lifetime safety performance assessment
This section focuses on what is required during the lifetime deployment of the ADS. Essentially, it is the manufacturers’ reporting duties when unexpected ADS events happen. There are two types of reporting events: non–critical occurrences and critical occurrences.
Critical occurrence means an occurrence in which the ADS is engaged at the time of a collision with at least one person suffering injury that requires medical assistance, or with physical damages above a certain threshold to the ADS vehicle, other vehicles, or stationary objects. The manufacturer must notify critical occurrences immediately to the type–approval authorities, market surveillance authorities, and the EU Commission.
The manufacturer shall report within one month any short–term occurrences, as described in the table below, that must be remedied by the manufacturer. The next table is a copy from the table in legislation draft.
I have one disagreement with this table: cybersecurity breaches are too important and should be reported within a month instead of within a year.
Summary of the ADS compliance assessment
The ADS compliance assessment section details how the type–approval organization will evaluate the ADS capabilities and assess its compliance with EU legislation.
The first task is to assess ODD road scenarios including urban, rural, and highways. ODD assessment includes common operations such as lane changes, merging, turning against traffic, cross traffic, and other driving tasks.
The second part is audit and assessment of the ADS safety concept including all ADS documentation the manufacturer provides. The manufacturer’s safety management system is also audited.
The third element is a description of driving tests that will be performed. These are pass or fail tests that assess the safety of the ADS in a variety of driving situations.
The fourth assessment is looking at the manufacturer’s M&S that has been performed and the capabilities of the M&S systems. This section is very comprehensive with extensive technical suggestions for gaining successful type–approval.
The last compliance assessment is to track the ADS performance after it is deployed. It includes information of what has to be tracked and reported to the type–approval organization to retain the vehicle’s type–approval.
The ADS compliance assessment is excellent and very complete. I have not seen anything close to the quality and completeness of this ADS legislative draft. Experts in various AV segments will surely find topics for improvements, but this is a solid proposal.
What is the impact?
The EU ADS legislation draft address all AV use cases including personal AVs and makes it a very ambitious and impactful regulation proposal. It specifies the ADS requirements and performance for auto OEMs and service providers, but not what technology to use. The ODD of the ADS will determine AV use cases.
The descriptions of how an ADS assessment will be performed are very complete and is likely to generate many sleepless nights for several AV engineers and executives. The M&S section is particularly impressive from my perspective and is likely to impact how the AV industry uses simulation and modelling across the board for designing, developing, testing, and deploying ADS vehicles.
Unlike the EU, The U.S. does not use type–approval procedures to determine if a vehicle can be sold. Rather, each auto OEM self–certifies that a vehicle model is ready for safe use and sale. Hence the type–approval procedures will have limited direct impact for vehicles sold in the U.S. But the indirect impact will be substantial as U.S. OEMs will participate in the European ADS market segments and will need type–approval.
Maybe the introduction of AVs is the time for using a type–approval procedure in the U.S. It would be a very unpopular decision for the OEMs, and they would likely put up a big fight to stop it. Perhaps it is a question that needs a lot of discussion and could be a future topic.