New tech will let smartwatches track your finger in mid-air using sound waves

Scientists, including those of Indian-origin, have developed a new sonar technology that allows you to interact with mobile devices by writing or gesturing on a tabletop, a sheet of paper or even in mid-air. An app, FingerIO tracks fine-grained finger movements by turning smartphones or smartwatches into an active sonar system using the device’s own microphones and speakers. Since sound waves travel through fabric and do not require a line of sight, users can even interact with a phone inside a front pocket or a smartwatch hidden under a sweater sleeve. Researchers at University of Washington (UW) showed that FingerIO can accurately track two-dimensional finger movements to within 8mm, which is sufficiently accurate to interact with today’s mobile devices. “You can’t type very easily onto a smartwatch display, so we wanted to transform a desk or any area around a device into an input surface,” said lead author Rajalakshmi Nandakumar, doctoral student at UW. “I don’t need to instrument my fingers with any other sensors – I just use my finger to write something on a desk or any other surface and the device can track it with high resolution,” Nandakumar said. Using FingerIO, one could use the flick of a finger to turn up the volume, press a button, or scroll through menus on a smartphone without touching it, or even write a search command or text in the air rather than typing on a tiny screen. FingerIO turns a smartwatch or smartphone into a sonar system using the device’s own speaker to emit an inaudible sound wave. That signal bounces off the finger, and those “echoes” are recorded by the device’s microphones and used to calculate the finger’s location in space. Using sound waves to track finger motion offers several advantages over cameras and other technologies like radar that require both custom sensor hardware and greater computing power, said assistant professor Shyam Gollakota. “Acoustic signals are great – because sound waves travel much slower than the radio waves used in radar, you don’t need as much processing bandwidth so everything is simpler,” said Gollakota. The researchers created a FingerIO prototype app for smartphone and a smartwatch customised with two microphones, which are needed to track finger motion in two dimensions. The researchers asked testers to draw shapes such as stars, squiggles or figure 8s on a touchpad next to a smartphone or smartwatch running FingerIO. Then they compared the touchpad tracings to the shapes created by FingerIO’s tracking. The average difference between the drawings and the FingerIO tracings was 0.8 centimetres for the smartphone and 1.2 centimetres for the smartwatch. “Given that your finger is already a centimetre thick, that’s sufficient to accurately interact with the devices,” said graduate student Vikram Iyer. PTI