The virtual and augmented reality advances gradually in both visual fidelity and headset comfort. Researchers are continuing to work on input solutions that will feel more natural than controller keeping. FingerTrak, a wristband-based solution that uses thermal cameras to monitor hand movements in 3D, was revealed by a group of researchers. It abstracting 20 finger joint positions on the wearer’s wrist.
FingerTrack is developed by Cornell University's Skyfi Lab with the help of Madison researchers at the University of Wisconsin. It uses a deep neural network to integrate input from three or four miniature thermal cameras mount on the wrist, finger and joint positions calculate using silhouettes created by cameras, and regression networks. Although the results are not perfect, they can use for any type of VR and AR inputs. Other potential applications for fingerprinting include human-robot interaction and control, sign language translation, including early detection of devastating diseases such as Parkinson's and Alzheimer's.
Interestingly, the researchers claim that wrist contours alone are appropriate to predict the entire hand position accurately. It enables the entire sensing device to mount on the wrist, rather than needing gloves, rings, or other previously published techniques. A demonstration video shows the hand motion tracking of FingerTrak which translates into bionic hand movements, as well as enabling a computer to determine when a user is writing, drinking coffee, and communicating with a smartphone.
It is uncertain if the device would be able to detect different movements easily, such as what a person is writing. The researchers note that the average angular error rate of FingerTrak during testing range from 6.46 to 8.06 degrees depending on the context it measure against. It might be possible to complement rather than substitute higher precision finger tracking solutions in full, at least for certain applications.
Wrist-mounted cameras could enhance the inside-out tracking cameras used in VR headsets like Oculus Quest. This control the user's finger and hand positions from the eyes. FingerTrak is already fairly small in prototype form, and with more development, it could easily go smaller. It relies mainly on very low-resolution (32- by 24-pixel) thermal cameras that are smaller than the thinnest Apple Watch. The prototype also relies on a tethered Raspberry Pi board. Although, the hardware could be operate alternately by a smaller solution.
FingerTrak will introduce at the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing in mid-September.