By combining FIR with other sensing technologies, automakers could finally get to the finish line of developing a Level-5 autonomous vehicle.

The race for the fully autonomous vehicle (AV) continues. But the current state of most OEMs’ sensing technologies prohibits further advancement of AVs. The most commonly used sensing solutions—radar, cameras, and lidar—cannot adequately and reliably provide autonomous vehicles with complete detection and coverage of their surroundings.

Radar, for example, can sufficiently detect long-range objects, but cannot clearly identify them. Cameras are much better able to identify objects but can only do so at close range. Many automakers, therefore, are combining their radar and camera efforts to provide more complete detection and coverage of vehicles’ surroundings: A radar sensor detects an object far down the road; as it approaches, a camera provides a clearer picture of it.

Lidar (light detection and ranging) sensors are another perception solution for most autonomous vehicles, but there are situations when they don’t work. Like radar, lidar works by sending out signals and use the reflection of those signals to measure the distance to an object. Lidar sensors can provide a wider field-of-view than radar, but a solution using eye-safe wavelength remains cost-prohibitive for mass market applications. Non-eye safe wavelength (near IR), which is cheaper to produce, is required by regulation to use lower emission power, resulting in a shorter detection range. Safe autonomous driving will only be enabled when automakers find a solution to fill the gaps left by other sensors. Far infrared (FIR) technology may be the solution.

Seeking a Sensor to Operate in Dangerous Conditions

FIR technology will become the primary sensor in autonomous vehicles’ sensor suites, as it is the only solution that can fill the gaps left by other sensors.

Unlike radar, cameras, and lidar, FIR (see Figure 1) can effectively detect and classify all objects in a vehicle’s surroundings, even when faced with the most dangerous driving conditions, including: inclement weather conditions, dynamic lighting conditions, pedestrians, animals, and nighttime. Delivering effective perception in these circumstances is crucial for the development of autonomous vehicles, as these are among the most common causes for vehicular accidents.

Unlike radar and lidar sensors that both transmit and receive signals, an FIR camera passively collects signals by detecting the thermal energy that radiates from objects. By scanning this infrared spectrum that’s just above visible light, FIR cameras access a different band of electromagnetic spectrum than other sensing technologies do. Thus, the FIR camera is able to generate a new layer of information, enabling it to detect objects that may not otherwise be perceptible to radar, cameras, or lidar. In addition to reading an object’s thermal signature, FIR cameras also capture objects’ emissivity—the rate at which an object emits heat. Since every object has a different emissivity, an FIR camera can immediately detect and classify any object—living or non-living—in its path.

Passivity is another advantage that FIR has over other sensing solutions. For instance, lidar and/or radar solutions are both active, energy-emitting modalities. As such, the lidar and/or radar installed and functioning on one vehicle may interfere with, and upset that of another passing vehicle. Conversely, because FIR is a passive technology, it can work to detect and cover a vehicle’s surroundings without ever upsetting the sensors of other vehicles.

FIR further trumps other sensing modalities, as it delivers an invariant image for lighting conditions: Its image perception is not compromised, neither by the color of an object, nor by its background or surroundings.

Both by accessing a new layer of information and by producing invariant images, FIR delivers intensely detailed images to the autonomous vehicle. This rich imagery, in turn, improves any feature of the AV that relies on machine perception, such as: on-road object detection, classification, intention recognition, tracking, distance estimation, semantic segmentation, and SLAM (simultaneous localization and mapping).

FIR can complete all of these tasks, even in the face of inclement weather conditions, because it uses long waves. Long-wave IR (LWIR, equivalent to FIR, wavelength 8-12 μm) detects emissive energy in the electromagnetic spectrum, rather than detecting reflective energy in the visible spectrum. For this reason, no light or other energy-emitting source is required for it to function. Compared to visible light and short-wave IR (SWIR) and mid-wave IR (MWIR <6 μm) infrared wavelength, long-wave IR also has superior transmittance in bad weather. Consequently, a thermal camera can produce richer images of a vehicle’s surroundings than other sensing solutions can, no matter if performing in fog, haze, or the like. FIR also outperforms other sensors when faced with dynamic lighting conditions, such as direct sunlight or oncoming headlights.

FIR Will Be the Leading Sensor in a CMOS Fusion

Proponents of FIR do not suggest that the technology replace all other sensors as the sole means of perception; instead, they believe FIR can be fused with a CMOS solution to deliver the complete sensing capabilities needed to achieve full autonomy.

For instance, the two sensing solutions can work synergistically to help autonomous vehicles best see and understand street signs. To effectively respond to a traffic sign, a vehicle must first identify and separate the sign from the background and then read it. Because FIR functions by assessing an object’s thermal signature and emissivity, it cannot see colors in great detail (i.e. it can struggle to read the sign). CMOS, on the other hand, cannot effectively detect thermally homogenic regions and separate them from the background—something FIR accomplishes seamlessly.

FIR Is a Mature and Proven Technology

While FIR has been used for decades in other industries, such as military and aviation, it has been neglected by the automotive market—even though autonomous vehicles need an accurate, reliable sensing solution. Experts postulate that FIR technology has remained out of the mass market auto industry as legacy companies have, until recently, reserved the technology exclusively for luxury brands.

FIR technology is now in the midst of a revolution. To achieve mass production and overcome installation challenges, the technology is now being produced at a lower price and in an optimal size that permits aesthetically-pleasing installation, almost anywhere on a vehicle.