More and more manufacturers are boasting that their phones are equipped with time-of-flight (ToF) cameras. But what are ToF cameras, how do they work and why would you want to have one on your smartphone?
ToF cameras have extra depth resolution
At a very basic level, ToF cameras are just ordinary HD cameras with increased depth resolution. In fact, phones like the 2019 Huawei P30 Pro display a resolution four times deeper than conventional cameras. Although "deep resolution" may seem confusing in the jargon of photography, it is actually easy to understand. A camera with a high depth resolution can differentiate near (foreground) and far (bottom) objects.
In photography, this idea is called foreground and background depth of field. That's what creates realism or concentration. Nearby objects are sharp, with sharp outlines, while distant objects are slightly fuzzy.
With a ToF camera, photographers have more options to control their depth of field. Of course, you do not have to be a photographer to benefit from the deeper resolution of the ToF camera. Automatic camera settings, when paired with a ToF camera, must produce stunning images with a well-defined foreground and background. A ToF camera can also be used to produce amazing video with advanced image stabilization or to increase the accuracy of your Instagram filters.
In addition, the ToF camera of a phone can be used to facial recognition, gestures and video games AR. Remember how Pokemon Go uses your camera to place a Pokémon in your environment? ToF cameras can make this feature much more compelling.
This technology is not new
Curiously, ToF cameras have been around since the late 70's. It just takes a little while for the technology to become economical and practical enough for mainstream applications.
In the past, ToF cameras were used for topographic mapping, automation of industrial machinery and automatic doors. But technology has made its way into many homes (including probably yours) in 2014. The Xbox One's Kinect uses a ToF camera to identify faces and accurately track hand gestures.
Since April 2019, only a few phones have integrated ToF cameras, such as LG G8 ThinQ, the View of honor 20, the Huawei P30 Pro, and the Oppo RX17 Pro. These phones are for photographers and geeks, but they set the standard for future phones, including Samsung and Apples Releases 2019 and 2020.
OK, the ToF cameras are super cool and exist since the release of Star Wars. But how do they work? Why do they have such a deep resolution?
ToF cameras use lidar to measure depth
Do you know how bats and dolphins use sound to "see" their environment? Well, we stole this trick, locked it up in submarines and named it sonar. ToF cameras use a similar technique to measure depth, but it's called lidar, and it relies on pulses of infrared light instead of sound.
When you take a picture with the ToF camera of a phone, it emits an invisible infrared light that is reflected on nearby objects. Some of this light is spreading into the abyss, but most of them find their way back into the phone's ToF camera.
Like a bat sonar wave, a lidar wave comes back in bits. The infrared light reflected by the near objects returns to the ToF camera, while the light reflected by the distant objects takes a little longer. Your phone measures the return time of each ray of light, crunches a few numbers, and creates a detailed 3D depth map.
It's a bit like those pinboard toys as you press your face in. These toys have a "3D effect" because your nose, located near the table, creates a footprint larger than your eyes and your cheeks, which are relatively far from the table.
Of course, the reflection of the infrared light only provides your phone with depth information. That's why ToF cameras combine IR functionality with a conventional high-resolution camera. Your phone uses software to mix a conventional photo with a 3D infrared depth card. You then get an image with the foreground and the background well defined.
Future applications for ToF cameras
In the coming years, ToF cameras will revolutionize the way we take pictures and videos with our phones. But let's think about the situation as a whole. We are talking about a technology capable of creating a depth map of an environment at the speed of light. His applications go well beyond the limits of photography.
In fact, your first experience with a phone's ToF camera will probably not just be photography. the Samsung Galaxy S10 5G will include ToF cameras, both in front and on the back. Apple is famous for its iPhone projects, but there are rumors the 2019 iPhones could also include ToF cameras. Yes, this update will make your selfies great, but it will also increase the accuracy of facial identity and other security measures. If you turn on facial recognition when setting up your phone, it will quietly capture a 3D map of your face before you have the chance to take selfies.
Gesture controls are another exciting (but probably dorky) application of ToF cameras. Yes, Kinect's gesture system is worth a look, but in the next few years, you'll likely use similar technology to control your phone, computer, smart assistant, or smartwatch. At the moment, companies like Moving are working on ToF applications for desktop computers, and manufacturers like Google use radar and lidar systems to create smartwatches and gesture-controlled, hands-free smart assistants.
In addition to gesture controls, ToF cameras are also ideal for VR and AR applications. Basically, Instagram and Pokemon Go filters will look much more "real" if used with a ToF camera. But for more complex applications, such as virtual reality, ToF cameras could be used to record player gestures and footsteps. They can also be used to quickly map physical environments and convert them to virtual reality environments, which could accelerate the development of professional games or open the virtual reality market to independent game developers.
But ToF cameras also have practical goals. In the past, they were used for machine automation in factories. It will not change. As ToF cameras improve (and at a lower cost), they will end up in more and more industrial and consumer AI products. More smart cars already uses a form of lidar technology to "see" its environment (it's worth noting that Tesla cars do not use lidar).
Although the ToF camera on your new mobile phone is limited to photographic, AR, and facial recognition applications, now is the time to become familiar with the technology. Finally, ToF cameras will be everywhere.