Teaching a car to see and decide
A self-driving car has to do something humans do effortlessly and yet find almost impossible to explain: perceive a complex, changing world and make safe decisions in real time. To pull this off, engineers break the challenge into three broad jobs — sensing the environment, understanding it, and acting on that understanding. Each relies on a different set of technologies working in concert.
The result is a car that reads the road many times per second, predicts what other people and vehicles will do, and steers, accelerates, and brakes accordingly. Understanding those three jobs is the key to seeing why the technology is both impressive and still unfinished.
The senses: how a car perceives
A self-driving car cannot rely on a single sensor, because each has weaknesses. Instead it blends several, a technique called sensor fusion, so that the strengths of one cover the blind spots of another.
- Cameras — see colour, read signs and lane markings, and recognise traffic lights, but struggle in glare or darkness.
- Radar — measures the distance and speed of objects and works well in rain or fog, but sees the world in low detail.
- Lidar — bounces laser pulses to build a precise 3D map of surroundings, but is costly and can be affected by heavy weather.
- Ultrasonic sensors — handle close-range tasks like parking and detecting nearby obstacles.
By combining these, the car builds a single, reliable picture of everything around it — far more consistent than any one sensor could provide alone. Much of this processing has to happen inside the car itself — a textbook example of edge computing.
The brain: understanding the scene
Raw sensor data is just numbers until software makes sense of it. This is where artificial intelligence takes over. Trained on vast amounts of driving data, the car's software identifies pedestrians, cyclists, other vehicles, road edges, and signs, then tracks how each is moving.
Crucially, the system also predicts what will happen next. Will that pedestrian step off the kerb? Is that car about to change lanes? Good prediction is what separates a safe autonomous system from a dangerous one, and it is one of the hardest parts of the whole problem because human behaviour is not always logical. It is the same challenge facing the wider field of robotics: making sense of a messy, unpredictable world.
Perceiving the road is difficult. Predicting what unpredictable humans will do next is harder still.
The hands: making it move
Once the car understands the scene and has predicted how it will evolve, it must decide on an action — maintain speed, slow down, change lanes, stop — and then execute it smoothly. This planning layer weighs safety, comfort, and the rules of the road, then sends precise commands to the steering, throttle, and brakes.
Detailed digital maps often assist here, telling the car about lane layouts, speed limits, and junctions before its sensors even see them. The car constantly compares the live sensor view against the map to know exactly where it is, down to a few centimetres.
The levels of autonomy
The industry uses a scale defined by the Society of Automotive Engineers, from Level 0 to Level 5:
- Level 0 — No automation. The human drives everything, all the time.
- Level 1 — Driver assistance. The car handles one task at a time — steering or speed — as with adaptive cruise control. The human supervises everything else.
- Level 2 — Partial automation. The car steers and controls speed together, but the human must watch constantly and intervene instantly. Most driver-assistance systems on sale today sit here.
- Level 3 — Conditional automation. The car drives itself in limited conditions, but the human must take over when asked — a handover so risky that some companies skip this level entirely.
- Level 4 — High automation. No human attention needed, but only inside a defined area or conditions. Today's city robotaxis are Level 4.
- Level 5 — Full automation. The car drives anywhere a human could, with no steering wheel needed. No production vehicle has reached this yet.
This matters enormously for safety: a system that drives well most of the time but occasionally needs a human to grab the wheel demands constant attention — exactly the vigilance humans are bad at maintaining when they feel the car is doing the work.
This is why the middle of the autonomy scale is, in some ways, the most dangerous place to be. A system that is good but not fully trustworthy invites complacency: the more capable it appears, the more tempting it is to stop paying attention, exactly when attention is still required. Engineers and regulators are acutely aware of this trap. Some companies have chosen to skip the awkward middle ground entirely, aiming straight for full autonomy in limited, well-mapped areas rather than selling half-measures to ordinary drivers. Others take the opposite view, arguing that gradual improvement, with the human always responsible, is the safer path to earning trust. Both approaches are being tested on real roads today, and which one proves wiser is still an open question — one that will shape not just the technology, but how comfortable the public feels sharing the road with it. Who is responsible when an automated car makes a mistake is one of the hardest questions in the ethics of artificial intelligence.
Who is actually on the road today
Driverless cars are no longer just test vehicles. Several companies now run real services, each taking a different route to the same goal.
- Waymo (owned by Alphabet) is the clear leader: a commercial, fully driverless robotaxi service across around eleven US metro markets — Phoenix (since 2020), the San Francisco Bay Area, Los Angeles, Austin, and Atlanta — with more cities coming online.
- Tesla launched its Robotaxi service in Austin in mid-2025, expanding to more US metros in 2026 — but fleets remain tiny. When it covered the whole Austin metro in mid-2026, only around twenty vehicles were actually operating there.
- Zoox (owned by Amazon) launched commercial service in late 2025 with purpose-built vehicles that have no steering wheel at all.
- Beyond the US, Baidu's Apollo Go operates across more than ten Chinese cities.
The pattern is consistent: real driverless service exists, but it is confined to limited areas with small fleets, expanding city by city.
Why full autonomy is taking so long
Building a car that drives well in good conditions is achievable. Building one that handles every rare, strange, and dangerous situation — the so-called edge cases — is extraordinarily hard. A ball rolling into the road, an unusual construction layout, a police officer waving traffic through a red light: humans handle these with common sense that machines still lack.
Because a driving mistake can cost lives, the bar for safety is rightly very high. That is why fully driverless cars have rolled out slowly, city by city, rather than arriving everywhere at once. The technology is genuinely remarkable, but the last few percent of reliability is proving to be the hardest — and most important — part of the journey. Fast, reliable 5G networks could eventually help cars share what they see with each other and with traffic systems.
Where things honestly stand
Fully driverless rides are a real, bookable service today — but only in a handful of cities, on well-mapped streets, with fleets in the hundreds or thousands, not millions. Two philosophies are being tested: Waymo and Zoox bet on expensive sensor suites, lidar included, in mapped areas; Tesla bets on cameras and AI alone, arguing it can scale everywhere faster. Both are making progress; neither has proved the other wrong. Safety, not demos, sets the pace — and the last few percent of reliability will decide everything.
