By Austin Little
Self-driving and driver-assist cars "see" with a mix of cameras, radar, lidar, and maps, then make guesses about what everything around them will do next. Here's how each piece works, where it struggles, and what that means for you behind the wheel.
AI disclosure. This page was drafted with AI assistance and edited for Apiary. We don't invent quotes, stats, people, or events. If something looks off, tell Austin — that's the point of a living hive.
When you drive, you're doing something amazing without thinking about it. Your eyes take in the road, the cars, the kid on the scooter, the brake lights two cars ahead, and the faded lane lines. Your brain figures out what each thing is, guesses what it'll do next, and decides what you should do — all in a fraction of a second, over and over, for the whole drive.
A self-driving car has to do all of that with machines. It needs "eyes" (sensors), a "brain" (computers running software, much of it based on machine learning), and a way to know where it is in the world. Different companies make different choices about which sensors to use, and those choices are part of a real, ongoing debate in the industry.
This guide explains, in plain English, how these cars see: what cameras, radar, lidar, and other sensors actually do, how the car combines them, where each one struggles, and — most important for regular drivers — what the difference is between a car that truly drives itself and one that just helps you drive.
The short answer
- Cameras see color, lane lines, signs, traffic lights, and shapes — much like human eyes. They struggle in glare, darkness, heavy rain, and fog.
- Radar sends out radio waves and measures what bounces back. It's great at measuring distance and speed and works in bad weather, but it gives a blurry picture.
- Lidar sends out laser pulses and builds a detailed 3D map of the surroundings. It's precise, but historically expensive, and it can be affected by heavy rain, snow, or fog.
- Ultrasonic sensors handle very short distances, like parking.
- GPS, motion sensors, and detailed maps help the car know exactly where it is.
- Sensor fusion — combining all of this — lets the car build one best-guess picture of the world.
- Most cars you can buy today are not self-driving. Features like adaptive cruise control and lane-keeping assist still require you to watch the road and be ready to take over.
First, a vocabulary check: "self-driving" means different things
Before we talk about sensors, it helps to know that "self-driving" is used loosely.
- Level 0: No automation. The car may warn you (like a beep for a car in your blind spot), but you do all the driving.
- Level 1: The car can help with one thing at a time — steering or speed — such as adaptive cruise control.
- Level 2: The car can help with steering and speed at the same time, like lane-centering plus adaptive cruise. You are still driving. You must watch the road and be ready to take over at any moment.
- Level 3: Under specific conditions, the car drives itself and you can take your attention off the road, but you must be ready to take over when the car asks.
- Level 4: The car drives itself without a human needing to take over, but only within a limited area or set of conditions (for example, a robotaxi service in certain parts of certain cities).
- Level 5: The car can drive itself anywhere, in any conditions a human could. This doesn't exist as a consumer product.
Most cars on the road today with "self-driving-like" features are Level 2. The sensors we're about to discuss show up across all of these levels, but how much the car is trusted to act on them varies a lot.
The cameras: the car's eyes
What they do
Cameras are the most familiar sensor. Most modern cars have at least one, often mounted behind the windshield near the rearview mirror, and many have several around the car. Some driverless vehicles use many cameras facing in every direction.
Cameras capture images, many times per second. Software then analyzes those images to find things like:
- Lane lines and road edges.
- Other vehicles, pedestrians, cyclists, and animals.
- Traffic lights and their colors.
- Stop signs, speed limit signs, and construction signs.
- Brake lights and turn signals on other cars.
Cameras are the only common sensor that sees color and reads text well. Radar can't tell you whether a light is red or green. Lidar can't read a speed limit sign the way a camera can. That makes cameras essential.
How the car understands what it sees
A camera image is just a grid of colored dots. The hard part is figuring out what those dots mean. Modern systems use machine learning — software trained on huge numbers of labeled examples — to recognize objects. Show a system enough examples of pedestrians in all kinds of clothes, poses, and lighting, and it learns to spot pedestrians in new images.
Cameras don't directly measure distance the way radar and lidar do. But software can estimate distance using two or more cameras (like your two eyes giving you depth perception), or by learning from experience how big things look at different distances, or by tracking how objects move between frames.
Where cameras struggle
- Darkness: Headlights only illuminate so much.
- Glare: Driving into a low sun, or headlights at night, can wash out the image — just as it does for human eyes.
- Bad weather: Heavy rain, snow, and fog reduce visibility.
- Dirt: A camera covered in mud, bugs, snow, or ice can't see.
- Unusual situations: Machine learning works best on things similar to what it was trained on. Odd-looking objects or rare situations can confuse it.
Radar: seeing through the weather
What it does
Radar has been used in cars for years, mainly for adaptive cruise control and automatic emergency braking. A radar unit sends out radio waves. When the waves hit an object, some bounce back. By measuring how long the echo takes to return, the car knows how far away the object is.
Radar has a special trick: it can measure an object's speed directly, using something called the Doppler effect. It's the same effect that makes an ambulance siren sound higher-pitched as it approaches and lower as it moves away. Radio waves bouncing off a car moving toward you come back slightly "squished," and the radar can use that to calculate how fast the car is moving relative to you.
Why it's useful
- Works in bad weather and darkness: Radio waves pass through rain, fog, snow, and dust much better than light does.
- Measures distance and speed well: Great for keeping a safe following distance on the highway.
- Relatively inexpensive and durable.
Where radar struggles
- Low detail: Traditional radar produces a fuzzy picture. It can tell something is there, how far, and how fast, but it's not great at telling what it is or exactly what shape it has.
- Confusing reflections: Radio waves can bounce off guardrails, tunnels, and metal surfaces, creating false echoes.
Lidar: the 3D laser map
What it does
Lidar stands for "light detection and ranging." It works like radar but uses pulses of laser light instead of radio waves. A lidar unit fires many laser pulses every second in different directions and measures how long each takes to bounce back. Each return is a single point in 3D space. Put millions of these points together and you get a point cloud — a detailed 3D outline of everything around the car: other vehicles, curbs, poles, pedestrians, trees, and the road surface.
You may have seen early self-driving test cars with a spinning cylinder on the roof. That was a lidar unit. Many newer lidar units are smaller and built into the car's body, though some robotaxis still carry visible sensor domes.
Why it's useful
- Precise distance and shape: Lidar gives accurate 3D measurements, which helps the car understand exactly where things are and how big they are.
- Works in the dark: Lidar brings its own light, so night isn't a problem in the way it is for cameras.
- Good at detecting objects even when they're unusual: Because lidar measures shape directly, it can detect that something is in the road even if the software doesn't know what it is.
Where lidar struggles
- Weather: Heavy rain, snow, and fog can scatter laser light and reduce performance, though modern systems use software to filter some of this out.
- No color: Lidar can't read a traffic light's color. Some can pick up the reflective paint of lane lines or signs, but cameras still do the heavy lifting for reading.
- Cost: Lidar has historically been expensive, which is one reason some carmakers avoid it.
- Dirt and damage: Like cameras, lidar units need clean windows to see through.
The big debate: lidar or no lidar?
Here's where the industry splits. Tesla, on the other hand, has pursued a camera-based approach for its driver-assistance systems, arguing that since humans drive with eyes and a brain, cameras and powerful software should be enough.
Both sides have real arguments. Multiple sensor types add cost and complexity, but they give redundancy. Camera-only systems are cheaper and scale more easily, but depend heavily on software being good enough to handle the hard cases. This article isn't going to settle that debate. Just know it exists, and that "which sensors?" is a design choice with tradeoffs, not a solved question.
The other senses
Ultrasonic sensors
These are the small round dimples you see on many car bumpers. They send out high-pitched sound pulses (too high for humans to hear) and listen for echoes, like a bat. They work at very short range and are mainly used for parking assist — the beeping that gets faster as you back toward a wall.
High-definition maps
Maps help a lot, but they can go out of date. Construction zones, new lane markings, and detours can surprise a car whose map hasn't been updated, which is why the car still has to trust its live sensors.
Sensor fusion: putting it all together
No single sensor is perfect. Cameras see color but struggle in glare. Radar sees through fog but gives a fuzzy picture. Lidar maps in 3D but can't read signs. The solution is sensor fusion: combining all the data into one best-guess picture of the world.
Imagine you're walking through a foggy parking lot at night. You can barely see, but you hear an engine and feel the vibration of a car nearby. Each sense alone is uncertain, but together they tell you a car is coming from your left. Sensor fusion does that for the car.
For example:
- The camera sees a shape that looks like a cyclist.
- The lidar confirms there's an object about the size of a cyclist at a specific distance.
- The radar says it's moving at about 12 miles per hour toward the intersection.
Combined, the car is much more confident: a cyclist, here, moving this fast. If one sensor is confused — say, the camera is blinded by the sun — the others can still provide useful information.
From seeing to driving: perceive, predict, plan
Seeing is only the first step. Self-driving systems typically work through a loop that repeats many times per second:
1. Perception: what's out there?
The car identifies and tracks objects: cars, trucks, pedestrians, cyclists, cones, debris. It also reads the road: lanes, signs, signals.
2. Prediction: what will everything do next?
This is one of the hardest parts. The car has to guess whether the pedestrian on the curb is about to step into the street, whether the car in the next lane will merge, whether the cyclist will swerve. Humans do this intuitively. Machines learn patterns from huge amounts of driving data.
3. Planning: what should I do?
Based on its predictions, the car decides on a path and speed: slow down, change lanes, wait, go.
4. Control: do it
The car sends commands to steering, brakes, and throttle.
Then it does it all again, constantly updating as the world changes. When people talk about self-driving systems being "trained," they often mean improving the software for perception and prediction using data from real-world and simulated driving.
Where self-driving systems still struggle
Even the best systems face tough situations:
- Bad weather: Snow can cover lane lines and coat sensors. Heavy rain and fog reduce visibility for cameras and lidar.
- Construction zones: Cones, temporary lanes, and flaggers waving people through don't match the map.
- Unusual events: A mattress in the road, a person in a costume, an emergency vehicle coming the wrong way, a police officer directing traffic by hand.
- Human unpredictability: Jaywalkers, aggressive drivers, a ball bouncing into the street followed by a child.
- Dirty or damaged sensors: Mud, ice, bugs, or a cracked windshield in front of a camera.
Companies working on driverless cars test extensively, both on roads and in simulation, to handle these "edge cases." But they're the reason fully driverless cars generally operate in limited areas and conditions today.
What this means for you as a driver
Here's the most important section for everyday drivers. Most people don't ride in robotaxis. They drive cars with driver-assistance features, and those features use many of the same sensors described above.
Know what your car's system is — and isn't
If your car has adaptive cruise control, lane-keeping assist, or a "hands-free" highway mode, it's almost certainly a driver-assistance system, not a self-driving car. Marketing names can be confusing. The owner's manual tells you what the system can and can't do. Read the section on its limitations — it's usually very specific about weather, road types, and situations it can't handle.
Stay alert
With Level 2 systems, you are the driver. Keep your eyes on the road and be ready to take over immediately. These systems can fail to see stopped vehicles, misread lanes, get confused in construction zones, or disengage with little warning. The U.S.
Never watch videos, read, nap, or let someone else "drive" from the passenger seat while using driver assistance. Driver-monitoring systems that watch your eyes or check for hands on the wheel exist for a reason.
Keep the sensors clean
Your car's sensors are its eyes. Before driving, especially in winter or after a dusty road trip:
- Clear snow and ice from the front grille, bumpers, and windshield area near the rearview mirror.
- Wipe off mud and bugs from camera lenses and sensor areas (check your manual for where they are).
- If your car shows a "sensor blocked" or "driver assistance unavailable" warning, take it seriously.
Be careful after repairs
A minor fender-bender or a windshield replacement can knock sensors out of alignment. Ask the repair shop whether your car's driver-assistance sensors need calibration, and make sure it's done.
Check for recalls and updates
Driver-assistance software gets updated, sometimes over the air and sometimes at the dealer.
When to worry and get help
Stop relying on the system and have the car checked by a dealer or qualified shop if:
- It brakes suddenly for no reason ("phantom braking").
- It drifts out of the lane or steers unexpectedly.
- It fails to slow for vehicles ahead that it used to detect.
- Warning lights for driver assistance or sensors stay on.
- It behaves differently after an accident or a repair.
If you experience a safety problem, you can also report it to NHTSA. And if you're ever in a crash while using a driver-assistance system, tell the police and your insurer that the system was active.
Sharing the road with robotaxis
If you live in a city with driverless vehicles, treat them like any other car: don't assume they'll see you, make eye contact where possible (with the car's direction of travel, since there's no driver to look at), and use crosswalks. Many robotaxi services provide ways to report problems with their vehicles.
Frequently asked questions
How do self-driving cars see at night?
Lidar and radar bring their own "light" (laser pulses and radio waves), so they work in the dark. Cameras rely on headlights and street lights, and some systems use cameras sensitive to low light.
Can self-driving cars see in fog and snow?
Radar handles fog and snow relatively well. Cameras and lidar can struggle. Snow covering lane lines and sensors is still a major challenge, which is one reason many driverless services operate in places and conditions with milder weather.
What's the difference between lidar and radar?
Radar uses radio waves and is great for distance and speed in bad weather but gives a fuzzy picture. Lidar uses laser light and produces detailed 3D maps but can be affected by heavy weather.
Why don't all self-driving cars use lidar?
Some companies argue cameras and software are enough and cheaper. Others argue lidar's precision and redundancy are worth the cost. It's an ongoing debate.
Is my car self-driving?
If you can buy it today, almost certainly not fully. Most systems are driver-assistance (Level 2), which require you to watch the road at all times. Check your owner's manual.
Do I need to recalibrate sensors after a windshield replacement?
Often, yes, if your car has a camera mounted behind the windshield. Ask the repair shop or dealer.
The takeaway
Self-driving and driver-assist cars see the road through a team of sensors: cameras for color and detail, radar for distance and speed in bad weather, lidar for precise 3D shapes, ultrasonic sensors for parking, and GPS, motion sensors, and maps to know where they are. Software fuses all of it into one picture, predicts what everyone around will do, and plans the car's next move — many times a second.
It's impressive engineering, and it still has limits. Most cars on the road today aren't self-driving; they're helping you drive. Keep your eyes on the road, keep the sensors clean, get them calibrated after repairs, and take warnings seriously. The machine is learning to see. You're still the one responsible for what it misses.