End-to-End Learning for Self-Driving: From Perception to Planning 2026

How neural networks are replacing hand-engineered robotics pipelines with unified models that see, think, and act — and the companies leading the race.

The Paradigm Shift in Autonomous Driving

For over a decade, autonomous driving systems followed a modular architecture: perception detects objects, prediction forecasts their motion, and planning computes a safe trajectory. Each module was separately engineered, tuned, and validated. In 2025-2026, a fundamentally different approach has gained dominance: end-to-end (E2E) learning, where a single neural network maps raw sensor inputs directly to driving actions.

The results are striking. Tesla’s Full Self-Driving (FSD) v13, released in late 2025, was built on an end-to-end architecture and demonstrated dramatically improved handling of complex urban scenes compared to the rule-based v12. Waymo’s 6th-generation driver, deployed in 2026, combines E2E neural components with structured safety layers. The industry consensus is that end-to-end learning is the path to truly scalable autonomous driving.

How End-to-End Driving Works

An end-to-end autonomous driving system takes sensor data as input and outputs driving commands:

• Inputs: Camera images (6-12 cameras at 3680×2160 @ 36fps), lidar point clouds (in some systems), radar returns, GNSS/IMU data, and vehicle state (speed, steering angle, acceleration). Some systems add HD map features and V2X messages.
• Architecture: A transformer-based backbone processes all sensor inputs:
  • Individual camera images are processed by a shared vision encoder (typically a Vision Transformer or EfficientNet variant)
  • Cross-view attention fuses information across cameras into a unified bird’s-eye-view (BEV) representation
  • A temporal transformer (past 2-4 seconds of encoded features) captures motion dynamics
  • Outputs include: predicted trajectory (sequence of future waypoints), object detections, lane graphs, and drivable area segmentation
• Output: A planned trajectory (typically 8 seconds at 0.5s intervals) that the vehicle’s low-level controller tracks. Some systems output direct control signals (steering, throttle, braking) for simpler scenarios.
• Training: The neural network is trained on massive datasets of human driving behavior (10M+ clips for Tesla, 20B+ miles for Waymo). Loss functions balance trajectory accuracy, safety constraints, comfort metrics (jerk, lateral acceleration), and traffic rule compliance.

World Models: The Next Frontier

The most exciting development in 2026 is the integration of „world models“ — neural networks that learn an internal representation of how the physical world evolves over time. World models enable the system to simulate possible futures before committing to a driving action.

• DrivingGPT / GAIA-1 (Wayve): Generative world models that simulate realistic future video frames given current observations and hypothetical actions. The system can „imagine“ 10 possible futures for the next 5 seconds and evaluate their safety before choosing an action. Wayve’s GAIA-1 has been validated on UK roads and licensed to multiple OEMs.
• GAIA-2 (2026): An evolution that incorporates language conditioning — the model can be prompted with natural language instructions („Take the highway exit ahead“ or „Avoid the construction zone“) that condition future predictions and planning. This enables a new level of driver-vehicle interaction.
• UniSim (Waymo + Google DeepMind): A neural simulation environment where the autonomous driving policy trains against simulated versions of itself and other realistic agents. This closed-loop training accelerates rare-scenario learning by 100x compared to real-world data collection.
• Tesla’s Occupancy Networks: Tesla’s E2E system uses 3D occupancy grids (voxel representations of occupied and free space) instead of bounding-box object detection. This better handles unusual objects and partial occlusions — a critical advantage for construction zones and emergency vehicles.

Sensor Fusion: Camera-Only vs. Multi-Modal

One of the most contentious debates in autonomous driving is sensor modality:

• Tesla’s Vision-Only Approach: Tesla removed radar in 2021 and ultrasonic sensors in 2022, betting entirely on cameras. FSD v13 demonstrates that camera-only can handle most driving scenarios at superhuman performance levels. The argument is that roads were designed for human vision, and adding other sensors adds complexity and cost. Tesla’s advantage is scale — billions of miles of camera data from its fleet.
• Waymo’s Multi-Modal Approach: Waymo’s 6th-generation system includes 29 cameras, 6 lidars, 6 radars, and external microphones. Lidar provides precise 3D geometry independent of lighting conditions, critical for pedestrian detection at night and in direct sunlight. Waymo’s argument is that redundant sensing modalities provide defense-in-depth for safety-critical decisions.
• Emerging Middle Ground: Several companies (Mobileye, Zoox, Cruise) use cameras as primary sensors with solid-state lidar as a safety backstop. The cost of lidar has dropped below $500 per unit in 2026, making multi-modal systems more economically viable for consumer vehicles.
• 4D Radar: A new generation of radar sensors provides elevation data in addition to range and velocity, creating „4D“ point clouds. Companies like Continental and Arbe are deploying 4D radar as a cost-effective complement to camera-only systems for adverse weather scenarios.

Safety Validation and Regulation

End-to-end systems pose new challenges for safety validation:

• Scenario-Based Testing: Rather than validating individual components, regulators now require end-to-end testing against standardized scenario libraries. The EU’s ALKS (Automated Lane Keeping Systems) regulation specifies 2,000+ test scenarios. China’s GB/T 40429-2025 autonomous driving standard includes 5,000+ scenarios.
• Safety Envelopes: Most production E2E systems include a „safety envelope“ — a parallel traditional system that monitors the neural network’s planned trajectory and intervenes if it violates safety constraints (exceeds speed limit, would cause collision, crosses lane markings). This defense-in-depth approach is required by most regulatory frameworks.
• ISO 21448 (SOTIF): Safety of the Intended Functionality standard addresses the unique challenges of ML-based systems. Manufacturers must demonstrate that all foreseeable misuse scenarios and performance limitations have been identified and mitigated.
• Data-Driven Evidence: In 2026, safety case submissions to regulators include statistical evidence: billions of autonomous miles with demonstrated lower crash rates than human drivers. Waymo has published data showing 85% fewer injury-causing crashes per million miles than human-driven vehicles in equivalent conditions.

The Companies to Watch

• Tesla (FSD v13-v14): Leading in scale (5M+ vehicles collecting data), deploying progressively more end-to-end neural networks. Robotaxi launch planned for Austin and San Francisco in 2026.
• Waymo: Operating commercial robotaxi service in 10+ cities. 6th-generation system integrates E2E components with proven safety architecture. Partnerships with Uber, Stellantis, and Zeekr.
• Cruise (relaunched 2026): After the 2023 San Francisco incident and temporary shutdown, Cruise has relaunched with enhanced safety systems. Deploying in Dubai, Phoenix, and Houston.
• Wayve: UK-based startup pioneering E2E + world models for passenger vehicles. Licensed its GAIA world model to multiple OEMs. First European OEM deployment expected in late 2026 with a major German manufacturer.
• BMW / Mercedes: Both have launched Level 3 (eyes-off) highway driving systems in 2026. BMW’s Highway Assistant and Mercedes‘ DRIVE PILOT represent the leading OEM approach — camera-lidar fusion with validated safety architectures.

Conclusion

End-to-end learning represents a paradigm shift in autonomous driving — from engineering rules to learning behaviors. World models add the ability to reason about future scenarios before acting. The technology has reached commercial viability, with robotaxis operating in 10+ cities and Level 3 systems available in production vehicles. The remaining challenges are scaling to all road types and weather conditions, reducing costs for consumer vehicles, and building public trust through transparent safety validation. The next 12 months will see the most significant expansion of commercial autonomous driving to date.