A self-driving car must do what a human driver does, only without tiredness or distraction: see the road, understand what is happening, predict what others will do and steer safely. Doing this reliably in every condition is one of the hardest problems in AI. Here is how the technology is organised.
Levels of driving automation
A widely used scale from the engineering society SAE describes six levels.
| Level | Name | Who is responsible |
|---|---|---|
| 0 | No automation | Human does everything |
| 1 | Driver assistance | Human drives; system helps with steering or speed (e.g. adaptive cruise control) |
| 2 | Partial automation | System steers and controls speed; human must supervise constantly |
| 3 | Conditional automation | System drives in limited conditions; human must take over when asked |
| 4 | High automation | System drives itself within a defined area and conditions, with no human fallback needed there |
| 5 | Full automation | System drives anywhere a human could |
Most cars sold today offer level 1 or 2 features. Some robotaxi services operate at level 4 in selected cities and areas. True level 5 remains a goal.
The sensors
No single sensor is perfect, so vehicles combine several (sensor fusion).
- Cameras: provide rich colour and detail for reading signs, lane markings and traffic lights. They struggle in glare, darkness and heavy rain or fog.
- Radar: uses radio waves to measure distance and speed, working well in poor weather and for tracking vehicles ahead, but with lower detail.
- Lidar: emits laser pulses to build a precise 3D point map of surroundings. Accurate shape and distance, but costly and affected by some weather.
- Ultrasonic sensors: short-range detection for parking.
- GPS and inertial sensors: estimate position and motion, supported by high-definition maps in some systems.
Some companies rely mostly on cameras plus AI, while others add lidar and radar. The best design is debated and depends on goals and cost.
The software stack
1. Perception
Neural networks interpret sensor data to detect and classify objects: cars, pedestrians, cyclists, signs, lanes and free space. They estimate positions, sizes and speeds. Vision models handle images, while other networks process lidar point clouds.
2. Localisation
The car works out exactly where it is, often within centimetres, by matching sensor data against maps and using GPS and motion sensors.
3. Prediction
The system forecasts what others will do next: will the pedestrian step off the kerb, will the car beside me change lanes? Prediction models use motion history and context, and usually output several possible futures with probabilities.
4. Planning
The planner chooses a safe, comfortable and legal path and speed that reaches the destination. It considers traffic rules, obstacles and the predicted behaviour of others, balancing caution with progress.
5. Control
Controllers convert the planned path into steering, acceleration and braking commands, many times per second.
Safety systems, monitoring and fallback behaviours run alongside, such as slowing down and stopping safely if a failure occurs.
How the AI is trained
Perception models learn from millions of labelled images and sensor frames showing objects in diverse conditions. Companies also use simulation to test rare and dangerous situations safely, and real-world driving data to find and fix failures. Some systems use end-to-end neural networks that map sensor input more directly to driving actions, while others keep separate modules that are easier to inspect.
A tiny illustration: braking distance
Planning needs physics as well as AI. This Python snippet estimates stopping distance from speed, reaction time and braking deceleration.
def stopping_distance(speed_kmh: float, reaction_s: float = 0.5, decel_ms2: float = 6.0) -> float:
v = speed_kmh / 3.6 # convert km/h to m/s
reaction_distance = v * reaction_s # distance travelled before braking starts
braking_distance = v ** 2 / (2 * decel_ms2)
return reaction_distance + braking_distance
for speed in (30, 50, 80):
print(f"{speed} km/h -> {stopping_distance(speed):.1f} m")The numbers show why speed matters so much: stopping distance grows with the square of speed, and a system must detect hazards early enough to stop.
Why it is so hard
- The long tail: rare situations, like unusual vehicles, hand signals, debris or construction zones, are endless.
- Weather and lighting: rain, snow, fog, glare and night driving degrade sensors.
- Human unpredictability: drivers and pedestrians break rules and communicate with eye contact and gestures.
- Local differences: road layouts, signs and driving culture vary widely. Roads with mixed traffic, animals and informal lanes, as in many South Asian cities, are particularly demanding.
- Proving safety: showing a system is safer than humans requires enormous testing and agreed standards.
- Responsibility: laws and insurance must clarify who is liable in a crash.
Benefits and concerns
Potential benefits include fewer crashes caused by human error, mobility for people who cannot drive and more efficient traffic. Concerns include accidents and over-trust in partial automation (drivers who stop paying attention), cybersecurity, job impact for drivers, data privacy and ethical decisions in unavoidable crash scenarios.
What it means for drivers today
Level 2 features are driver aids, not self-driving. Keep your hands ready, eyes on the road and understand the limits stated by the manufacturer.
Frequently Asked Questions
Are self-driving cars safe?
Safety depends on the system, the area and conditions. Some limited services report good records, but broad claims need independent evidence. Driver-assistance systems still require full attention.
Why do some companies avoid lidar?
They argue cameras and AI can be enough and are cheaper. Others value lidar's precise 3D measurement. Both approaches continue to be developed.
Could self-driving cars work in Nepal?
Dense mixed traffic, narrow roads, mountain terrain and weather make it very challenging. Driver-assistance features are more likely to arrive first.
What happens if sensors fail?
Systems are designed with redundancy and fallback behaviour, such as slowing and stopping safely, but no system is perfect.
Conclusion
Self-driving cars combine sensors, perception, prediction, planning and control. Progress is real, especially in limited areas, but handling every road, weather and human behaviour remains difficult. Understanding the levels and limits helps you see past the hype.