NINE: Multimodal conversational system for empathetic communication between people and machines

NINE aimed to give conversational systems awareness of their environment by processing acoustic and visual signals and transforming them into natural language clauses. On this basis, a multimodal communication channel was established, personalized to the user's profile and the context surrounding them, seeking to create an empathetic relationship between person and machine.

To this end, the project was organized around the following objectives:

• Feature extraction: speech (emotion and user profile), user appearance and gestures, both on a touch interface and as an enrichment of verbal communication.
• Determination of the user's personality and intent, by building feature analyzers, determining personalities, interpreting the meaning of body gestures and interpreting speech.
• Semantic reasoning of the response adapted to the context, combining vocal content, affective tone and gestural attitude.
• Control of embodied conversational agents (ECAs), using a modular diction platform and 3D animation.
• Real-time delivery of the virtual conversational agents through their various access and distribution channels.