{"author":{"@type":"Person","name":"Alexander Bock","url":"https://abxk.ai/en/autor/alexander-bock/"},"categories":["Physical AI"],"content":{"markdown":"A human driver uses two eyes, two ears and years of experience. A self-driving car replaces them with a ring of sensors and software that turns millions of measurements per second into three decisions: where to steer, when to accelerate and when to brake. In several US cities this already works without anyone behind the wheel, and every crash of these cars is reported to the federal road safety authority.\n\nThis is part 2 of Level 0 in our Physical AI learning path. You will learn what the levels of driving automation mean, which sensors a self-driving car uses and what each one sees, how the software gets from sensor data to steering and what public crash reports say about how these cars actually get into accidents.\n\n## Six levels of automation\n\n\"Self-driving\" is used for very different systems. The standard SAE J3016 from SAE International, an association of automotive engineers, sorts them into six levels. The key question is who is responsible for watching the road.\n\n![A staircase of six levels from Level 0 no automation to Level 5 full automation; up to Level 2 the human drives and watches the road, from Level 3 the system drives in defined conditions](/images/learn/en/how-self-driving-cars-see-levels.webp \"The six SAE levels. The big step is between Level 2 and Level 3: who watches the road.\")\n\nAt Levels 1 and 2, the car helps: it keeps its distance and its lane, but you drive and must watch the road at all times, even with your hands off the wheel. From Level 3, the system drives itself in defined conditions, and at Level 4 it needs no human at all inside its **operational design domain (ODD)**: the area, roads, speeds, times and weather it was built and tested for. A robotaxi that serves one city is a Level 4 vehicle. Level 5, a car that drives anywhere a human could, does not exist yet.\n\n## The sensors and what they see\n\nNo single sensor sees everything, so most self-driving vehicles combine several types. Each has strengths where another is weak.\n\n| Sensor | What it measures | Strengths | Weaknesses |\n|---|---|---|---|\n| Camera | Color images, like an eye | Reads signs, traffic lights and lane markings; cheap | Needs light; dazzled by low sun; distance must be estimated |\n| Radar | Distance and speed via radio waves | Works in rain, fog and darkness; measures speed directly | Coarse image; hard to tell what an object is |\n| Lidar | Distance via laser pulses, as a 3D point cloud | Precise 3D shape of everything around the car | Expensive; weaker in heavy rain, snow and fog |\n| Ultrasonic | Distance via sound, very close | Parking and maneuvering | Range of only a few meters |\n| GNSS and inertial sensors | Position and motion of the car itself | Knows where the car is and how it moves | Satellite signal lost in tunnels and between tall buildings |\n\n![Top view of a car with overlapping sensor fields: long-range radar and cameras to the front, lidar all around, short-range sensors at the corners](/images/learn/en/how-self-driving-cars-see-sensors.webp \"Overlapping sensor fields around the car. Where one sensor is weak, another covers the gap.\")\n\nCombining the readings is called **sensor fusion**. If the camera is dazzled by the sun, radar still measures the car ahead. If radar sees an object but cannot tell what it is, the camera can. Whether a car could drive on cameras alone, like a human, is one of the big open debates in the industry; most robotaxis that drive without a safety driver today combine cameras, radar and lidar.\n\n## From sensor data to steering\n\nThe software of a self-driving car has four classic stages. **Perception** finds and labels objects in the sensor data: cars, cyclists, pedestrians, cones. **Prediction** estimates where each of them will be in the next seconds. **Planning** chooses a path and a speed that are safe and comfortable. **Control** turns that plan into steering angle, throttle and brake pressure. Most stages use machine learning, especially perception, where neural networks detect objects in camera images and lidar point clouds.\n\nA newer approach replaces several stages with one large model that goes from camera images straight to driving commands, called **end-to-end** driving. It learns from many hours of recorded driving, much like a language model learns from text. It can handle situations that are hard to describe in rules, but it is harder to check why it made a decision.\n\nMany robotaxis also carry a detailed **HD map** of their area, with every lane, curb and traffic light. The car compares what it sees with the map to know exactly where it is, which is one reason why such services start in one city and expand street by street.\n\n## When the car is unsure\n\nEven a Level 4 vehicle meets situations it cannot resolve on its own: a road blocked by a fallen tree, a police officer giving hand signals, a construction site that is not on the map. Two safety nets exist for such moments. First, the car can perform a **minimal risk maneuver**: it slows down and stops in a safe place, ideally at the side of the road. Second, most robotaxi operators have staff in a remote center who can answer questions from the car, for example by confirming which way to drive around an obstacle, while the car still steers itself. For passengers, this means that a robotaxi will sometimes stop and wait. That is safer than guessing, and it may be one reason why so many reported crashes involve a vehicle that was standing still.\n\n## What the crash reports show\n\nIn the US, every crash of a vehicle with an automated driving system has to be reported to the National Highway Traffic Safety Administration (NHTSA). We analyzed all {{\u003c num \"av\" \"reports\" \u003e}} reports filed since June 2025. In {{\u003c num \"av\" \"driverless_pct\" \u003e}}% of them, nobody was in the driver's seat, and in {{\u003c num \"av\" \"speed_zero_pct\" \u003e}}% the automated vehicle was standing still when the crash happened.\n\n{{\u003c data \"av-movement\" \u003e}}\n\nThis is one lesson of the data: in many crashes the automated vehicle was not moving, so another road user ran into it. Whether the robot car had stopped in an unexpected place, the reports do not say. They also do not say who caused a crash, and they contain no miles driven, so they cannot tell you whether robotaxis are safer than human drivers. The full analysis with all tables is on our page about [self-driving car crash data](/en/physical-ai/self-driving-car-crash-data/).\n\nSelf-driving cars are robots on wheels. The next part of Level 0 looks at robots that grasp and walk, and how they learn.","text":"A human driver uses two eyes, two ears and years of experience. A self-driving car replaces them with a ring of sensors and software that turns millions of measurements per second into three decisions: where to steer, when to accelerate and when to brake. In several US cities this already works without anyone behind the wheel, and every crash of these cars is reported to the federal road safety authority.\nThis is part 2 of Level 0 in our Physical AI learning path. You will learn what the levels of driving automation mean, which sensors a self-driving car uses and what each one sees, how the software gets from sensor data to steering and what public crash reports say about how these cars actually get into accidents.\nSix levels of automation \u0026ldquo;Self-driving\u0026rdquo; is used for very different systems. The standard SAE J3016 from SAE International, an association of automotive engineers, sorts them into six levels. The key question is who is responsible for watching the road.\nThe six SAE levels. The big step is between Level 2 and Level 3: who watches the road. At Levels 1 and 2, the car helps: it keeps its distance and its lane, but you drive and must watch the road at all times, even with your hands off the wheel. From Level 3, the system drives itself in defined conditions, and at Level 4 it needs no human at all inside its operational design domain (ODD): the area, roads, speeds, times and weather it was built and tested for. A robotaxi that serves one city is a Level 4 vehicle. Level 5, a car that drives anywhere a human could, does not exist yet.\nThe sensors and what they see No single sensor sees everything, so most self-driving vehicles combine several types. Each has strengths where another is weak.\nHow Self-Driving Cars See the Road: Cameras, Radar and Lidar Sensor What it measures Strengths Weaknesses Camera Color images, like an eye Reads signs, traffic lights and lane markings; cheap Needs light; dazzled by low sun; distance must be estimated Radar Distance and speed via radio waves Works in rain, fog and darkness; measures speed directly Coarse image; hard to tell what an object is Lidar Distance via laser pulses, as a 3D point cloud Precise 3D shape of everything around the car Expensive; weaker in heavy rain, snow and fog Ultrasonic Distance via sound, very close Parking and maneuvering Range of only a few meters GNSS and inertial sensors Position and motion of the car itself Knows where the car is and how it moves Satellite signal lost in tunnels and between tall buildings Overlapping sensor fields around the car. Where one sensor is weak, another covers the gap. Combining the readings is called sensor fusion. If the camera is dazzled by the sun, radar still measures the car ahead. If radar sees an object but cannot tell what it is, the camera can. Whether a car could drive on cameras alone, like a human, is one of the big open debates in the industry; most robotaxis that drive without a safety driver today combine cameras, radar and lidar.\nFrom sensor data to steering The software of a self-driving car has four classic stages. Perception finds and labels objects in the sensor data: cars, cyclists, pedestrians, cones. Prediction estimates where each of them will be in the next seconds. Planning chooses a path and a speed that are safe and comfortable. Control turns that plan into steering angle, throttle and brake pressure. Most stages use machine learning, especially perception, where neural networks detect objects in camera images and lidar point clouds.\nA newer approach replaces several stages with one large model that goes from camera images straight to driving commands, called end-to-end driving. It learns from many hours of recorded driving, much like a language model learns from text. It can handle situations that are hard to describe in rules, but it is harder to check why it made a decision.\nMany robotaxis also carry a detailed HD map of their area, with every lane, curb and traffic light. The car compares what it sees with the map to know exactly where it is, which is one reason why such services start in one city and expand street by street.\nWhen the car is unsure Even a Level 4 vehicle meets situations it cannot resolve on its own: a road blocked by a fallen tree, a police officer giving hand signals, a construction site that is not on the map. Two safety nets exist for such moments. First, the car can perform a minimal risk maneuver: it slows down and stops in a safe place, ideally at the side of the road. Second, most robotaxi operators have staff in a remote center who can answer questions from the car, for example by confirming which way to drive around an obstacle, while the car still steers itself. For passengers, this means that a robotaxi will sometimes stop and wait. That is safer than guessing, and it may be one reason why so many reported crashes involve a vehicle that was standing still.\nWhat the crash reports show In the US, every crash of a vehicle with an automated driving system has to be reported to the National Highway Traffic Safety Administration (NHTSA). We analyzed all 1,551 reports filed since June 2025. In 83.3% of them, nobody was in the driver\u0026rsquo;s seat, and in 50.9% the automated vehicle was standing still when the crash happened.\nWhat the automated vehicle was doing just before the crash MovementReportsShare Stopped (traffic, light)66042.6% Driving straight54335.0% Parked1187.6% Turning left684.4% Turning right422.7% Changing lanes372.4% Backing up261.7% Passing211.4% Source as for the first table on this page.\nThis is one lesson of the data: in many crashes the automated vehicle was not moving, so another road user ran into it. Whether the robot car had stopped in an unexpected place, the reports do not say. They also do not say who caused a crash, and they contain no miles driven, so they cannot tell you whether robotaxis are safer than human drivers. The full analysis with all tables is on our page about self-driving car crash data.\nSelf-driving cars are robots on wheels. The next part of Level 0 looks at robots that grasp and walk, and how they learn.\n","words":1058},"date":"2026-10-18","description":"How self-driving cars work: the SAE levels, what cameras, radar and lidar each see, how software turns sensor data into steering and what crash reports show.","image":"https://abxk.ai/images/cover/how-self-driving-cars-see.webp","json":"https://abxk.ai/en/physical-ai/learn/how-self-driving-cars-see/index.json","kind":"page","language":"en","learning_path":{"category":"physical-ai","level":0,"level_name":"Start here"},"modified":"2026-10-18","section":"physical-ai","seo_title":"How Self-Driving Cars See: Cameras, Radar and Lidar","tags":["autonomous-driving","self-driving-cars","lidar","sensors","beginner"],"title":"How Self-Driving Cars See the Road: Cameras, Radar and Lidar","translations":[{"json":"https://abxk.ai/de/physical-ai/learn/how-self-driving-cars-see/index.json","language":"de","url":"https://abxk.ai/de/physical-ai/learn/how-self-driving-cars-see/"}],"url":"https://abxk.ai/en/physical-ai/learn/how-self-driving-cars-see/"}