Team: Kazuki Kawamura (Student Researcher), Jun Rekimoto
This project aims to build a system to evaluate how well a user can speak a target language and a process that can be applied to improve the user's speaking skills in the target language. This project originally started from the experience that "even if a person in Tokyo imitates the Kyō-kotoba (Kyoto dialect), a person in Kyoto would immediately recognize the difference." We built a system to replicate this experience using AI to evaluate the speaking skill of the Kyō-kotoba uttered by the user using deep learning techniques. In addition, we have taken this technology a step further and built a system that enables learners to learn speaking interactively by visually presenting the difference and distance between the learner's speech and the speech of a native speaker. We are considering other applications as skill evaluation and skill acquisition assistance by AI in the future.
Publications:
- 1. Visualization of Speech Differences for Dialect Speech Training (CHI 2021 workshop)
- 2. A Language Acquisition Support System that Presents Differences and Distances from Model Speech (UIST 2022 Poster)
- 3. DDSupport: モデルとなる発音との差異・距離を提示する言語学習支援システム (INTERACTION 2022)
Media Mentions:
- 英語ネイティブとの発音の違いを深層学習で可視化 ソニーCSLが開発（ITmedia NEWS）