VisualSpeech

This exploratory project proposes to research natural and multimodal interaction mechanisms for providing bio-feedback in speech therapy through the use of serious (computer) games. This project proposes a serious game toolset that has two main goals.

The first goal is to keep the patients engaged in the exercises in order to have more fruitful sessions. Building on our previous work, the game will use visual stimuli, and a reward system that will adapt to each particular patient and it will provide analysis [URL] of the facial exercises as part of speech therapy.

The second goal aims at providing the therapist with a toolset to plan the course of the ongoing therapy session. This will leverage on the team’s previous experience in language tutoring systems. The toolset will access multimodal information extracted from the current and previous sessions. More specifically, the toolset will provide a powerful set of inspection tools to examine the therapy recordings and assist the therapist with audio-visual recordings and annotations of the session.

Click on the figures to watch/try the demos:

visualspeech

3dspeech