WithYou: An Interactive Shadowing Coach with Speech Recognition


Speech shadowing is a proven way of practicing speaking skills when learning foreign languages. Though effective, however, it is very difficult doing because the sound playback is irrelevant to the learner’s speech. When a learner fails to catch up, he/she will have to reset the playback or jumps directed to where is being played, resulting in a waste of time and learner’s resilience in doing it. In this research, we propose a shadowing support system called WithYou. By comparing on-the-fly speech recognition results and the words currently being played, WithYou can detect shadowing failure while a student is practicing, and then rewind the playback position to the corresponding time point when the failure occurred. Moreover, based on the recognition tailored for shadowing practice, WithYou is capable of recognizing pauses in learner’s speech during trial-and-error. In addition, it supports repetitive practice with a certain extent of error tolerance so that the feedback it provides is neither too frequent nor too late.




  • Xinlei Zhang, Takashi Miyaki, Jun Rekimoto: WithYou: An Interactive Shadowing Coach with Speech Recognition. UIST ’16 Demo
  • 張鑫磊, 味八木崇,暦本純一: WithYou: 音声認識を用いたインタラティブシャドーイングコーチ, インタラクション2016 インタラクティブ発表(プレミアム発表)