Interaction paradigms

Future mobile devices will include a number of sensors, significantly increased wireless communication and computational power. For example, already existing mobiles by the industry partner include accelerometers and video cameras. In this scenario, other sensors technologies, including specific acoustic and vibration sensors can be integrated to be able (i) to sense the surrounding space where the mobile is placed, (ii) to recognize the objects touched by the mobile, (iii) to perform a qualitative (including expressive and emotional) and quantitative analysis of user’s actions. In the first case, for example, the mobile, once placed on top of a table, a chair or another everyday object, gives the sense of touch to such object: therefore, the table itself (not augmented with any technology, so remaining a normal, simple everyday object) becomes an extension of the interface of the mobile. In the second case, the mobile can be used to touch objects, e.g., in the home, returning information on what the touched object is and related properties, to get audio/vocal information on the object, to enable a service or application in the home, to support elderly or disabled users. In the third case, the system recognizes the actions and provides an interpretation according to the context, including the environment state, the user’s state (e.g., emotional, including possible physiological sensors), the application state, and content semantics. Cross- and multi-modal integration of tangible acoustic interfaces with other different sensors, perception, and communication will be one of the key technologies to address such issues. These novel interface paradigms, that will emerge and will be exploited in this project, will be part of the end-to-end research platform, and thus will integrate context-aware technologies with natural multimodal technologies, to enable the user to become participative, highly interactive, and socially engaged.