TestArmy ebook QA trends 2019

Be sure to download our new report and follow the latest trends in QA, Cybersec, IoT, DevOps and more!


Autonomous cars. How to test things that have never been seen before?


The biggest challenge for delivery teams, in the current environment of repeating disruptions is to prepare for testing products of the future. Those are solutions that no one has seen, are part of daily operations for developers and testers. In the age of continuous digitalization the technology that changes the driver interaction with a car and prepares to give the algorithm total control over it are totally different challenges than testing web systems or mobile applications. Those web and mobile products are pretty common and most companies know how to tackle that. However, things like machine learning, artificial intelligence, autonomous cars, augmented and virtual reality are areas where You have to be innovative in the space of design and implementation, but also in quality assurance.

Cars are moving to real world

As recently confirmed, during July and August the United Nations had agreed the regulations for autonomous vehicles. It has been signed by 53 countries around the world making it one step closed to have autonomous cars a common sight on our public roads. According to Japan Times, the scope of signed countries include Japan, South Korea, members of European Union and the United Nations Economic Commission for Europe. This is the first multi-national regulation dedicated for autonomous cars of level 3 in national laws to expand the growth of autonomous vehicles on the market. However we already hear about premieres of cars produced in the EU and Japan that have these abilities. That means the designers and testers had to work with them a lot of time before.

What are autonomous cars levels?

Let’s start with the definition of the autonomous driving levels, it is agreed to have a split to 5 levels:

  • level 0 – No Automation – the full-time performance by the human driver of all aspects of the dynamic driving task
  • level 1 – Driver Assistance -the driving mode-specific execution by a driver assistance system using information about the driving environment and with the expectation that the human driver performs all remaining aspects of the dynamic driving task – in example adaptive cruise control
  • level 2 – Partial Automation – the driving mode-specific execution by one or more driver assistance systems  using information about the driving environment and with the expectation that the human driver performs all remaining aspects of the dynamic driving task – in example traffic jam assistant or automatic parking system
  • level 3 – Conditional Automation – the driving mode-specific performance by an automated driving system of all aspects of the dynamic driving task with the expectation that the human driver will respond appropriately to a request to intervene – in example when cal can drive fully autonomous in selected road conditions like highways.
  • level 4 – High Automation – the driving mode-specific performance by an automated driving system of all aspects of the dynamic driving task even if a human driver does not respond appropriately to a request to intervene the car can pull over safely by guiding system – in example when car can handle multiple road situations and only unexpected challenges might be caused by weather conditions.
  • level 5 – Full Automation – The driving mode-specific performance by an automated driving system of all aspects of the dynamic driving task under all roadway and environmental conditions that can be managed by a human driver – in example here we can have cars that do not even have pedals or steering wheel.


Definitions according to: “Automated Driving – Levels of Driving Automation are Defined in New SAE International Standard J3016” (PDF). SAE International. 2014. Archived (PDF)from the original on 1 July 2018.

Challenge for product delivery

While designing and building autonomous vehicles projects the biggest change is to move Yourself to the future. Companies want to have ready products once the appropriate law is created. For example recent agreement for level 3 cars that will be enabled in many countries at the start of 2021 is already fulfilled with cars planed to be released in 2021 like Honda Legend, Mercedes S-Class, Tesla. That requires product designers,developers and testers to implement and validate the software thinking about tomorrow while working on project. Companies focused on delivery of such products and verification services providers like Test Army have to prepare processes to fulfill the requirements of highly processed delivery. It has to be compliant with Automotive SPICE framework, Functional Safety ISO norm and other standards. There a lot of required metrics at code and verification levels for example for coverage by tests. There have to be millions of kilometers covered in simulations and thousands of road maneuvers and situations to be checked. This requires an effective strategy not only for verification, but also test automation and simulation. Meanwhile the design teams prepare solutions for tomorrow in advance before those laws are signed and must keep a significant scope of agility. Tenable looking beyond here and now towards the future sometimes it requires to also take step in re-shaping the norms and processes, event laws of modern world to enable the product to be usable. In example many organizations work on autonomous cars a step ahead of regulations to make their products ready. However some new players are working straight on level 4 and 5 solutions as a platform to enable it to multiple car manufactures once the regulations are ready. It is a long term race with billions of Euros investments in huge market to be served. There is also a constant pressure on technical experts i.e. testers should be ready for such challenges in their projects to build the practices for verification of such solutions.

How to prepare?

To prepare themselves it is beneficial to try and understand how product managers and designers think. Business often does not focus on the current stage, but plans a few steps ahead. It is important to learn from the shape of the past and presence for the benefit of the ongoing project, but the main goals are visible in forward-looking. The same way testers need to understand all the best engineering and verification practices they know, however they need to design a way to utilize that knowledge in the environment of the product for the future. Sometimes it requires a new usage of existing tools or practices, often You have to step beyond the norm or processes that handle Your project. Just to later support Your customer to alternate “old” documents or prepare a way to combine the product and new law proposals.

Transportation of the future

Solutions like autonomous vehicles it is not only the segment of private owned cars, but also:

  • Mobility as a service, for private and public passenger transport
  • Transportation as a service, for cargo trucks
  • RoboTaxis or car sharing services, for personal transportation

They not only take over the responsibility from humans, but in parallel need to interact with them in a seamless way. All of those machine learning algorithms need to not only work on a daily basis, but also grow for the future. They will not only take over the drivers responsibility, but in level 5 just only interact with cargo or passengers without a driver’s role at all. It is the testers responsibility to switch their thinking to such solutions and find the best way to verify it. They need to be able to shift the focus from judging the current behaviour and observations to think about the learning mechanism and design test scenarios to verify if it works properly. If they think the machine learns now will enable correct behaviours in the future. The same challenge goes with bots connected with machine learning or artificial intelligence, and all different products that will be enabled on the market with ML and AI utilized under the surface.

The tester of the future

Our job as the testers is to think about the era that lies in the future, looking at designs, specifications and consider how these products will be used and could shape the future. We can use all the tools and practices that exist today, but also we need to remember that strategies for usage of heuristics and model-based testing. Those solutions where we can validate the software under test against preconditions made by the model. 


To empower the delivery of the product and enablement of the customer vision in real world, a few strengths are required:

  • Be a learner – be open for new ideas and solutions and happy to dig more to understand them better, the need that does not exist today might be crucial in the future
  • adapt to changes – during the project delivery there will be new research and conclusions made, some of them might totally alter the business case or purpose of the project
  • rationalization of the imagination – the ultimate goal is to deploy a business product not fly away with a vision constantly chasing the horizon, be prepared to help the customer to keep the focus on rolling out the product to the market

When we think about the future it is a space full of possibilities. In contrast to the presence that we feel as a space full of constraints. This optimistic approach to the day of tomorrow enables the business to shape the solution, with the option to shift priorities and re-evaluation of the idea. Space where we lack alternatives today might be reopened. Testers of the solutions for the future must be open to take a passenger seat in that journey toward the future. They have to be ready to step out from that image of being pessimistic and focused on presence. They need to be open to take the vision and try to travel in time to wear the end user’s shoes. This travel through time changes the traveller, too. It has to be that change as the goal for testers to be ready to work with those products. It gives new perspectives of how we might use cars, phones, and interact with technology. Enriched with that knowledge the whole product team including the testers can travel through time building the product roadmap and verification plans.

Prepare Yourself

As a tester to be ready for such challenges in your innovative projects, You not only have to be ready for tomorrow, but also for the scenery of the day after. This will be required to do their job inside the delivery team and move the product from vision to a concrete shape that can be released to the market. As each step it is required to grow with the product to keep the logical continuation of what already was built  and what is designed on the roadmap. All of those aspects have to fit together in a shape of the final product fully integrated, with the highest quality standards and efficient feature delivery. Product of tomorrow face much more public audience and no-one wants to fail at the first few steps. The best testers can intuitively think far in advance, but not in a straight line, they build the image in a turbulent way. This enables them to be much better with software verification on undiscovered ground.

The race is on

The market delivery of such disruptive innovations requires tremendous effort form design and product management divisions supported with technical teams built from great developers and testers who keep their heads open to be ready for that challenge. With their work we can observe continuous growth in technological solutions that not only suggest to us the fastest path to reach our destination, but also support us in decisions. For example our cars from static navigation got enriched with:

  • interactive HMIs
  • head-up displays, 
  • dynamic rerouting based on traffic data
  • augmented reality projections on front window

Many of us had already seen or used the solutions like Tesla’s autopilot, Mercedes Drive Pilot, Cadillac Super Cruise and these are solutions for level 1.5 and 2. We already got tests from  American Consumer Reports where 17 solutions were compared and top eight were:
Super Cruise from  Cadillac, Tesla Autopilot ,then we have Lincoln/Ford CoPilot 360, Audi Pre Sense, Hyundai Smart Sense/Kia Drive Wise, Mercedes Driver Assistance, Subaru Eyesight i BMW Active Driving Assistance Pro. This race will soon restart with level 3 and we will get Honda as a new player on the field, extended systems form Tesla, Mercedes and Audi are already confirmed. Future is unknown as we do not know what manufactures will enter at which level and race will restart up to the autonomous driving level 5. Just fasten Your seat belts and get ready to enjoy the ride both as a customer, tester or engineering team. Based on the benefits from level 1.5 and 2 of autonomous vehicles studies from https://www.iihs.org/ declare that switching from human driven accidents to single car would be moved to autonomous mode a third of road crashes would be prevented, if both cars are autonomous then almost 80% would not happen. Isn’t that an exciting future and challenge for our testing teams?

Rate the article:

Michał Buczko

Michał Buczko 18.12.2020

Chief Testing Officer w TestArmy, od 15 lat związany ze środowiskiem testowania oprogramowania. Prelegent na wielu międzynarodowych konferencjach testerskich. Na co dzień skupia się na współpracy działów technicznych i sprzedażowych oraz prowadzi usługi konsultingowe dla klientów TestArmy. W wolnym czasie stawia na blogowanie, jazdę na snowboardzie, triathlon i wędkarstwo. Obecnie rozwija swoje zainteresowanie drukiem 3D, autonomicznymi pojazdami i elektryfikacją rynku samochodowego.

Comments are closed.