Hand-and-finger-awareness for mobile touch Interaction using deep learning [article]

Huy Viet Le, Universität Stuttgart, Universität Stuttgart
2019
Mobile devices such as smartphones and tablets have replaced desktop computers for a wide range of everyday tasks. Virtually every smartphone incorporates a touchscreen which enables an intuitive interaction through a combination of input and output in a single interface. Due to the success of touch input, a wide range of applications became available for mobile devices which were previously exclusive to desktop computers. This transition increased the mobility of computing devices and enables
more » ... sers to access important applications even while on the move. Despite the success of touchscreens, traditional input devices such as keyboard and mouse are still superior due to their rich input capabilities. For instance, computer mice offer multiple buttons for different functions at the same cursor position while hardware keyboards provide modifier keys which augment the functionality of every other key. In contrast, touch input is limited to the two-dimensional location of touches sensed on the display. The limited input capabilities slow down the interaction and pose a number of challenges which affect the usability. Among others, shortcuts can merely be provided which affects experienced users and contradicts Shneiderman's golden rules for interface design. Moreover, the use of mostly one finger for input slows down the interaction while further challenges such as the fat-finger problem and limited reachability add additional inconveniences. Although the input capabilities are sufficient for simple applications, more complex everyday tasks which require intensive input, such as text editing, are still not widely adopted yet. Novel touch-based interaction techniques are needed to extend the touch input capabilities and enable multiple fingers and even parts of the hand to perform input similar to traditional input devices. This thesis examines how individual fingers and other parts of the hand can be recognized and used for touch input. We refer to this concept as hand-and-finger-awareness for mobile touch interaction [...]
doi:10.18419/opus-10555 fatcat:qd4blarcdrdq3p7ljcfht4lofq