Visual speech - A Serious Game for Speech Therapy with Visual Bio-feedback

Sofia Cavaco  Eskenazi Maxine  "Leveraging on the multidisciplinary nature of this project, we aim at improving human assistance technologies for people with disabilities such as speech disorders and facial paralysis due to strokes, brain injury caused by accidents, neurological or degenerative diseases, cancer of the neck or throat, hearing impairment, and mental health problems. The major outcome of the project is aligned with our previous work towards the use of digital games to make a real change in the life of groups with special needs, enhancing their better integration in society. We will work towards the construction of an effective serious game for speech therapy.
This project requires the development of new methodologies and tools for producing, applying and using gamification settings with targeted techniques for therapy with visual bio-feedback. Thus, this project will also contribute to the research of principled, natural and multimodal HCI methods to increase the effectiveness of digital games for professionals and researchers, intermediaries and social actors dealing with people with disabilities." Sofia Cavaco and Maxine Eskenazi 
Portuguese PI
Sofia Cavaco (FCT/UNL)

Maxine Eskenazi (LTI)


Research teams: CITI/FCT/UNL; CMU
Organizations: Hospital Garcia de Orta
Funding Reference: FCT CMUP-EPB/TIC/0075/2013 
Duration: 12 months 
Keywords: Serious game, speech therapy, natural interaction, facial expression analysis, automatic intelligent tutors

This exploratory project proposes to research natural and multimodal interaction mechanisms for providing bio-feedback in speech therapy through the use of serious (computer) games. This project proposes a serious game toolset that has two main goals: the first goal is to keep the patients engaged in the exercises in order to have more fruitful sessions. The game will use visual stimuli, and a reward system that will adapt to each particular patient and it will provide analysis of the facial exercises as part of speech therapy. The second goal aims at providing the therapist with a toolset to plan the course of the ongoing therapy session. This will leverage on the team’s previous experience in language tutoring systems. The toolset will access multimodal information extracted from the current and previous sessions. More specifically, the toolset will provide instantaneous bio-feedback, replacing other tools such as a mirror, and assist the therapist during the session with audio-visual recordings and annotations of the session.
The novelty of this toolset is the combination of visual bio-feedback and the gamification of the speech therapy. While the success of this approach has been proved in non-therapy areas, in this proposal we tackle the speech therapy area by including the analysis of mouth and lips positioning, and facial muscles movement. The toolset will also provide an application for post-session analysis where a patient’s audio-visual recordings can be examined through browsing and searching, and a future session can be planned. In this exploratory project, the proposed serious game will have the therapist judge the correctness of the speech pronunciation. The game is proposed as a key instrument to keep the patient engaged and reactive to the instant bio-feedback provided by the game.


The Phase II of the Carnegie Mellon Portugal Program emphasizes advanced education and research that can lead to significant entrepreneurial impact. The Early Bird Projects are designed to assist small teams of researchers from Portuguese institutions, Carnegie Mellon University and industry partners, to jumpstart high-impact potential activities of strategic relevance for the Program.

Research Opportunities more