ICARE Approach

ICARE Model

Systems support multiple interaction techniques such as the synergistic use of several modalities (speech, gesture, gaze, localization of a user, etc.). The flexibility they offer results in an increased complexity that current software development tools do not address appropriately. ICARE (Interaction CARE -Complementarity Assignment, Redundancy and Equivalence-) is a component-based approach which allows the easy and rapid development of multimodal interfaces.  ICARE is concentrated on input (i.e., from the user to the system) although our model holds for output as well. Nevertheless we did not test our approach for output so far.

The ICARE conceptual model includes two kinds of ICARE components:

1. Elementary components: Such components are building blocks useful for defining a modality. Two types of ICARE elementary components are defined: Device components and Interaction Language components. We reuse the definition of a modality as the coupling of a physical device d with an interaction language L: <d, L>.  A physical device is an artifact of the system that acquires (input device) information. Examples of devices include the mouse, microphone, GPS and magnetometer. An interaction language defines a set of well-formed expressions (i.e., a conventional assembly of symbols) that convey meaning. The generation of a symbol, or a set of symbols, results from actions on physical devices. Examples of interaction languages include pseudo-natural language, direct manipulation and localization. An interaction modality such as speech input is then described as the couple <microphone, pseudo natural language NL>, where NL is defined by a specific grammar. Similarly graphic input is described in terms of <mouse, direct manipulation>.

2. Composition components: Such components describe combined usages of modalities and therefore enable us to define new composed modalities. The ICARE composition components are defined based on the four CARE properties: the Complementarity, Assignment, Redundancy, and Equivalence that may occur between the modalities available in a multimodal user interface. Composition components can connect from two to n components and are not dependent on a particular modality.

* Elementary Components

Elementary components are dedicated to interaction modalities. Based on the definition of an interaction modality, we identify two types of elementary ICARE components, namely Device and Interaction Language components.

An ICARE Device component represents a supplementary layer of the physical device driver. For example, the mouse Device component abstracts the data provided by the mouse driver such as button pressed/released and movement. Likewise a microphone Device component abstracts the captured signal into a recognized utterance while another microphone Device component abstracts the captured signal into a level of noise. All ICARE Device components also enrich the raw data from the device driver by adding information that include the working state of the device, the time-stamp as well as a confidence factor of the produced data, and a description of the device in terms of human manipulation (passive/active modalities, human actions involved and physical location of these actions). An ICARE Device component is then linked to a listener component, an ICARE Interaction Language component in order to form an interaction modality.

An ICARE Interaction Language component corresponds to the logical level of an interaction modality. For example an Interaction Language component abstracts the data from a mouse Device component into commands such as the selection of a menu option. Similarly another Interaction Language component (NL component), abstracts a set of characters from a microphone Device component (recognized utterance) or from a keyboard Device component into a command. A third example, shown in Figure 1, corresponds to a passive modality: the 3D Location component that abstracts data from a localization sensor (e.g., GPS) Device component into a user’s location expressed in a given coordinate system. These three examples of Interaction Language components underline the fact that such components may need to rely on an external description of the well formed expressions to be obtained. Indeed in order to abstract data from the mouse into commands, a description of the graphical interface is required. Likewise NL recognition implies a description of the pseudo Natural Language to be recognized (NL grammar). Finally the 3D Location component may require a description of the environment of the user in order to produce an event such as <the user is entering a particular room>. As Device components are dependent on the underlying physical devices, Interaction Language components are dependent on a class of Device components that can produce the required inputs. For example an NL component requires a set of characters as inputs that can for example be produced by a microphone Device component or a keyboard Device component. Finally, as ICARE Device components, ICARE Interaction Language components also enrich the data by adding generic information that include the time-stamp as well as a confidence factor of the produced data.

Device and Interaction Language components constitute the building blocks for defining modalities. The designer can then combine these components in order to specify a new composed modality, in other words, a combined usage of several modalities.
 

* Composition Components

The CARE properties characterize the different usages of multiple modalities. Based on the CARE properties, we define three composition components: the Complementarity one, the Redundancy one, the Equivalence one and the Redundancy/Equivalence one. Assignment and Equivalence are not modeled as components in our ICARE model. Indeed, as shown in Figure 1, an assignment is represented by a single link between two components. An ICARE component "Magnetometer"  linked to a single component "3d orientation" implies that "Magnetometer" is assigned to "3d orientation". As for Assignment, Equivalence is not modeled as a component. As shown in Figure 1, when several components (2 to n components) are linked to the same component, they are equivalent. In our previous multimodal systems developed using ICARE components, we explicitly used an Equivalence component that had no functional role (no treatment except defining a new time-stamp to the data) but constituted an aid while manually assembling components. Using our platform under construction that will allow the user to graphically assemble ICARE components, such Equivalence component has no more utility.

In Figure 1, we present an example of ICARE specification that includes Complementarity components. Let us consider the Complementarity-1 component. In order to compute the location and orientation of the user that is required by the application, two passive modalities are used in a complementary way. The Complementarity-1 component of Figure 1 defines a customizable temporal window for merging data received by the two Interaction Language components (respectively orientation in radians and location as latitude/longitude in WGS84 normalization and altitude in meters).

As ICARE elementary components, ICARE composition components enrich the data by adding generic information that includes the time-stamp and a confidence factor of the produced combined data. In addition composition components include parameters that the designer can fix for customizing the composition mechanism.

ICARE Platform

The ICARE platform enables the designer to graphically manipulate and assemble ICARE software components in order to specify the multimodal interaction dedicated to a given task of the interactive system under development. From this specification, the code is automatically generated. To fully understand the scope of our ICARE platform we show in Figure 2 where the automatically generated code is located within the complete code of the interactive system structured along the ARCH software architectural model.

The originality of the ICARE platform relies on the fact that it is dedicated to designers and not developers. Indeed the user of the ICARE platform selects the modalities and specifies the combination of modalities in terms of the CARE ergonomic properties, all by graphically assembling software components without knowing the details of the code of the components. From this high level specification, the code of the input multimodal UI is then generated.

Figure 3 presents a sketch of the user interface of the ICARE platform: it contains a palette of components, an editing zone for assembling the selected components and a customization panel for setting the parameters of the components. Although the complete ICARE platform is not yet available, we have already designed and developed several components including modality components as well as combination components in order to validate our approach. By manually assembling these components, we have developed several multimodal systems. The following section describes these ICARE components that will in the near future be graphically manipulated in the ICARE platform.

Figure 3. Sketch of the graphical ICARE platform.

Multimodal Systems Developed Using ICARE

MID (Multimodal IDentification system) MID is a prototype. One user can identify himself with three modalities equivalent or redundant. The password can be given by the voice, the keyboard or the mouse.

MEMO AR MEMO AR, stand for Augmented Reality, allows users to annotate physical locations with digital notes which have a physical location. The notes can be read, removed or moved. Users move in real world and see numerical notes in  Head-Mounted Display.

FACET FACET is a larger system. It is a simulator of Rafale (a French military plane). One prototype has been realized with ICARE Components for few tasks. The complete system is on-going work.

RAZZLE Puzzle in augmented reality. On-going work.

References

ICARE papers

October 2004

J. Bouchet, L. Nigay, T. Ganille ICARE Software Components for Rapidly Developing Multimodal Interfaces, Proceedings of ICMI 2004, Penn State University - State College, PA, USA, October 2004, to appear.

August 2004

J. Bouchet, Ingénierie ergonomique de l'Interaction Multimodale : Approche à composants, Annexe des actes d'IHM 2004, Namur, Belgique, Août 2004, à paraître.

June 2004

J. Bouchet, L. Nigay, D. Balzagette ICARE : Approche à composants pour l'interaction multimodale, Actes des Premières Journées Francophones : Mobilité et Ubiquité 2004, Nice Sophia-Antipolis, France, Juin 2004, pp 36-43.

April 2004

J. Bouchet, L. Nigay ICARE: A Component-Based Approach for the Design and Development of Multimodal Interfaces, Extended Abstracts of CHI'04, Vienna, Austria, April 2004, pp 1325-1328.

Underlying concepts

1997

Nigay, L., Coutaz, J. Multifeature Systems: The CARE Properties and Their Impact on Software Design. Intelligence and Multimodality in Multimedia Interfaces, AAAI Press,1997, 16 p.

1996

Nigay, L., Coutaz, J. Espaces conceptuels pour l'interaction multimédia et multimodale. TSI, spécial Multimédia et Collecticiel, AFCET & Hermes Publ., Vol 15(9), 1996, pp. 1195-1225.

June 1995

Coutaz, J., Nigay, L., Salber, D., Blandford, A., May, J., Young, R. Four Easy Pieces for Assessing the Usability of Multimodal Interaction: The CARE properties. Proceedings of INTERACT'95, Lillehammer, Norway, June 1995, pp. 115-120.

May 1995

Nigay, L., Coutaz, J. A Generic Platform for Adressing the Multimodal Challenge. Proceedings of CHI'95, New York, USA, May 1995, pp. 98-105.

April 1993

Nigay, L., Coutaz, J. A design space for multimodal interfaces: concurrent processing and data fusion. Proceedings of INTERCHI'93, Amsterdam, April 1993, pp.172-178.

Contacts: laurence.nigay@imag.fr - jullien.bouchet@imag.fr