Touch-Based Interaction

About

The problem The use of touch-based devices has increased dramatically in recent years. In many application scenarios and environments, touch-based interaction offers an intuitive and practical alternative to the mouse and keyboard of the desktop PC and the physical keyboard of a traditional mobile phone. However, from a purely technical perspective, most mainstream touch-based interfaces restrict the user’s interaction to a small number of degrees of freedom with pressing, moving and releasing fingers at different positions on a two-dimensional surface.   The solution The novel development here employs a device using an Inertial Measurement Unit (IMU) in wearable technology to increase the potential complexity of a user’s interactions with a touch screen beyond simple press, drag and release actions. A wide variety of other gestures relating to the user’s hand and arm movement before, during and after a touch event can be captured and employed.   Data from the IMU is streamed to a host touch screen device. The data is analysed and gestures mapped to a range of actions on the host device. The implementation could be extended to use more complex activity recognition approaches to detect a wider range of gestures, track a user’s interactions over a time period longer than a single touch gesture, detect bi-manual gestures and allow users to define their own gestures. It is envisaged that the increased expressiveness offered by the invention could offer a range of new interaction possibilities in application contexts including musical performance, gaming and expert interaction with creative software packages like Adobe Illustrator.   The opportunity The technology has the potential to be adopted in a very wide range of settings and usage applications. For example, research in educational settings suggests that cognitive, emotional and general engagement was higher in lessons which were based on iPads. Touch-based gaming is well established as an application for touchscreen mobile devices, but could potentially be significantly enhanced by the addition of a wrist-worn sensing device. Industry projections suggest that the global market for touch screen modules in mobile devices will reach 1.3 billion units in 2018. The invention would act as an enabler for improvements in the capabilities offered by applications that are accessed through touch screen devices. The approach can be implemented by combining a wrist-worn IMU with many commercially available touch screens.   The invention is available for co-development and/or license.   Intellectual property The technology is protected through a UK patent application which has been filed and is currently being examined.