WithYou: Automated Adaptive Speech Tutoring With Context-Dependent Speech Recognition

Learning to speak in foreign languages is hard. Speech shadowing has been rising as a proven way to practice speaking, which asks a learner to listen and repeat a native speech template as simultaneously as possible. However, shadowing can be hard to do because learners can frequently fail to follow the speech, and unintentionally interrupt a practice session. Worse, as a technical way to evaluate shadowing performance in real-time has not been established, no automated solutions are available to help. In this paper, we propose a technical framework with context-dependent speech recognition to evaluate shadowing in real-time. We propose a shadowing tutor system called WithYou, which can automatically adjust the playback and the difficulty of a speech template when learners fail, so shadowing becomes smooth and tailored. Results from a user study show that WithYou provides greater speech improvements (14%) than the conventional method (2.7%) with a lower cognitive load.

 

teaserV2.png

Figure 1. a) A learner (blue) tries to shadow a template speech (orange) by repeating what he listened to from an earphone as simultaneously as possible, but he fails. b) The template speech detects this failure and automatically rewinds the speech after adding some pauses. This is controlled by the computer in remote without any manual operations. c) The practice session resumes with lower difficulty, and the learner follows well this time.

 

rollback2020.png

Figure 2. WithYou can detect the fallback from a user’s speech, and automatically change the playback of the speech template to recover a broken practice session.
RA_single_trial.png
Figure 3. Averaged performance transition of participants as the comparison of the two methods tested. It is calculated by taking the difference between the performance of a training session to the performance of the pre-training test, and then averaging the results for each time of the training of all the participants for both methods.
RA_Overview.png
Figure 4. Pre-Post repeat accuracy improvement comparison of all participants in the experiment.
Please look at the paper for details 🙂

References

Xinlei Zhang, Takashi Miyaki, and Jun Rekimoto. WithYou: Automated Adaptive Speech Tutoring WithContext-Dependent Speech Recognition. in Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1-12. [CHI 2020 Honorable Mentions Award – Top 5%]