Multilingual Byte2Speech Models for Scalable Low-resource Speech Synthesis [article]

Mutian He, Jingzhou Yang, Lei He, Frank K. Soong
2021 arXiv   pre-print
To scale neural speech synthesis to various real-world languages, we present a multilingual end-to-end framework that maps byte inputs to spectrograms, thus allowing arbitrary input scripts. Besides strong results on 40+ languages, the framework demonstrates capabilities to adapt to new languages under extreme low-resource and even few-shot scenarios of merely 40s transcribed recording, without the need of per-language resources like lexicon, extra corpus, auxiliary models, or linguistic
more » ... se, thus ensuring scalability. While it retains satisfactory intelligibility and naturalness matching rich-resource models. Exhaustive comparative and ablation studies are performed to reveal the potential of the framework for low-resource languages. Furthermore, we propose a novel method to extract language-specific sub-networks in a multilingual model for a better understanding of its mechanism.
arXiv:2103.03541v2 fatcat:z7xtjh723rey3gyrzhrrepj3ea