While vague on the surface, an Apple white paper detailing the company's approach to self-driving cars reveals some additional insights into its approach to autonomous vehicle development.
As above, so below
Publication of the report followed a February 2019 disclosure that revealed Apple’s safety drivers to be taking control of experimental self-driving cars approximately one time for every mile driven.
That sounds like a lot, but I rather suspect the truth is this is connected with Apple’s different approach to artificial intelligence (AI) rather than any weakness in its systems.
My opinion (based on Apple’s general approach to AI, which is to deploy intelligence at the edge rather than in the cloud) is that the company hopes to build a truly autonomous vehicle intelligence.
You don’t want a car that know it should turn left now, you want a car that knows that and also knows why it should do so – a level of contextual intelligence that also makes the vehicle more autonomous. And much harder to train.
That’s my opinion. You probably want some facts.
Here’s what Apple has to say:
Beyond transportation
Apple says it is “excited” at the potential of automation in many sectors, with transportation being one of these.
Automation for the people
Apple sees the potential for automation to improve road safety, boost mobility, and “realize broader social benefits.”
What happens inside these cars?
Apple has two people inside its vehicles: a safety driver and an operator. The safety driver scans the road and is ready to take control of the vehicle at any time. The operator watches the car’s automated driving system. Their job is to tell the driver what the vehicle has detected and what it will do next – they can also disengage the system with a touch on their screen.
What about the sensors?
The 63-plus Apple text vehicles you may see on roads use a combination of LiDAR sensors, radars, and cameras to understand the environment they are driving through. These deliver high-resolution 360-degree 3D coverage around the car.
The cars aren’t fully in control yet
Apple reveals that commands made by the automated driving system aren’t yet fully immediate. This is so the safety driver can override them – so if the car wanted to change lanes, the driver will get a chance to take control before the lane change takes place.
How does the safety driver take control?
The safety driver can take control of the vehicle by using the wheel, brake, throttle or gear lever, or by tapping an override button. Drivers must keep two hands on the wheel when the car is in autonomous mode.
Sense, plan, drive
Apple’s self-driving system is essentially based on three main components that work together in a kind of loop. That loop repeats multiple times each second and the components include sensing the environment, planning, and executing vehicle response.
Plays nice with others
Apple’s sensing system can place the vehicle and will monitor surroundings for others, including cars, pedestrians, and cyclists
Apple Maps are super-important
We all know Apple Maps cars have been driving around the world to capture high-resolution street images. Well, those images form part of the driving system Apple uses in its cars. It uses maps and location together with data from the sensors to plan the vehicle’s next move.
Apple is already making self-driving software
Apple has already created some form of self-driving software. Those daily driver’s meetings also see updates to that software reviewed.
Apple cars try to predict the future
Several times a second the car monitors surrounding objects and then attempts to predict where they will be “many seconds” in future. This predictive intelligence also informs the driving system.
While Apple’s report makes no mention of this, it is interesting to note that researchers at the University of Michigan are currently developing a system that teaches self-driving cars to predict pedestrian movement, this system also works in 360-degrees using a neural network.
The system isn’t perfect yet
Apple says that control is passed to the human driver instantaneously if a problem is detected. Safety drivers in Apple’s test suite must always have their hands on the wheel, which rather suggests development has some way to go.
There’s lots of safety features
Apple’s driving system uses predictive intelligence and sensor data to manage the drive, but it also works with both hardware and software safety systems to ensure the drive is safe. The vehicles are crash-tested and assessed to maintain the highest possible level of protection.
“Any malfunctions and scenarios that require the safety driver to take control of the vehicle to address a hazard are further studied to confirm that the safety driver could regain manual control of the vehicle safely. This includes simulations and vehicle testing with induced faults at proving grounds,” the company states.
Vehicles are tested daily.
Apple deploys software incrementally
Apple also tells us that even once it has been tested, new software is introduced incrementally across the test fleet.
Apple tests systems on a component basis
In a discussion on system verification, Apple explains that it tests every new hardware and/or software component extensively. These tests include simulations, on test benches, and on closed courses before these components are let loose on public roads – and even then, these additions are closely monitored.
What about the drivers?
As it develops its technologies, Apple puts both a safety driver and an observer in its cars. Drivers must be proficient, with clear driving records and no serious accidents, DUI convictions, or any license suspensions or revocations within the last 10 years. They also need to take drug and background checks. There are daily driver’s meetings, and Apple actively encourages staff to raise any safety concerns.
Yes, what about safety?
Any safety concerns raised are checked, can be escalated, and the whole fleet grounded if necessary.
What happens if something goes wrong?
If an incident takes place, all testing is paused, data is reviewed, and an investigation takes place. If the system or driver contributed to the incident, then corrective steps will take place.
You are not the product
Apple is taking the same approach to self-driving vehicle technologies as it does to any other mobile product. Privacy and security are critical, and it gathers no information beyond the bare minimum required to make its systems work. That means when you drive an Apple Car, you won’t find yourself being re-routed by your local burger bar or find ads for places you’ve visited appear when you next go online. You are not the product.
Apple will share what it knows
The company says it “stands ready” to be a resource on current and future technological, regulatory, and public policy matters, which implies:
- It intends to continue development of self-driving cars.
- It intends lobbying for any necessary legal changes that may be required to put its vehicles on the road.
- It may contribute data to support future vehicle insurance models, such as that in development at Avinew.
What happens next?
Apple’s disclosure that its sensors rely on LiDAR and imaging data may prove important in the future. After all, how well will your autonomous vehicle perform in rain, fog, or smog? Sensors capable of handling this are only now in development at MIT.
I’d caution that this means there’s some way to go before any company successfully puts truly autonomous vehicles safely on public roads, no matter what the press releases or optimistic government representatives claim.
Please follow me on Twitter, or join me in the AppleHolic’s bar & grill and Apple Discussions groups on MeWe.