Demo for Adversarial-Multi-Task Learning-SVS

Abstract

        Recently, deep learning-based generative models have been introduced to generate singing voices. One approach is to predict the parametric vocoder features consisting of explicit speech parameters. This approach has the advantage that the meaning of each feature is explicitly distinguished. Another approach is to predict mel-spectrograms for a neural vocoder. However, parametric vocoders have limitations of voice quality and the mel-spectrogram features are difficult to model because the timbre and pitch information are entangled. In this study, we propose a singing voice synthesis model with multi-task learning to use both approaches -- acoustic features for a parametric vocoder and mel-spectrograms for a neural vocoder. By using the parametric vocoder features as auxiliary features, the proposed model can efficiently disentangle and control the timbre and pitch components of the mel-spectrogram. Moreover, a generative adversarial network framework is applied to improve the quality of singing voices in a multi-singer model. Experimental results demonstrate that our proposed model can generate more natural singing voices than the single-task models, while performing better than the conventional parametric vocoder-based model.

Audio Samples using Parallel WaveGAN (48k)
Audio Samples using WORLD vocoder (48k)

Sentence: 사라져가는 저 별빛 사이로 감출 순 없나요 커져버린 마음 (Pronunciation): salajyeoganeun jeo byeolbich sailo gamchul sun eobsnayo keojyeobeolin ma-eum
N-Singer	Single-Task	Multi-Task (Proposed)	Copy-Synthesis (Ground Truth)

Sentence: 남들 눈엔 힘 빠지는 한숨으로 보일지 몰라도 나는 알고 있죠 (Pronunciation): namdeul nun-en him ppajineun hansum-eulo boilji mollado naneun algo issjyo
N-Singer	Single-Task	Multi-Task (Proposed)	Copy-Synthesis (Ground Truth)

Sentence: 사라져가는 저 별빛 사이로 감출 순 없나요 커져버린 마음 (Pronunciation): salajyeoganeun jeo byeolbich sailo gamchul sun eobsnayo keojyeobeolin ma-eum
N-Singer	Single-Task	Multi-Task (Proposed)	Copy-Synthesis (Ground Truth)

Sentence: 이제 다른 생각은 마요 깊이 숨을 쉬어봐요 그대로 내뱉어요 (Pronunciation): ije daleun saeng-gag-eun mayo gip-i sum-eul swieobwayo geudaelo naebaet-eoyo
N-Singer	Single-Task	Multi-Task (Proposed)	Copy-Synthesis (Ground Truth)

Sentence: 내가 안아줄게요 정말 수고했어요 (Pronunciation): naega an-ajulgeyo jeongmal sugohaess-eoyo
N-Singer	Single-Task	Multi-Task (Proposed)	Copy-Synthesis (Ground Truth)

Adversarial Multi-Task Learning for Disentangling Timbre and Pitch

in Singing Voice Synthesis

Tae-Woo Kim, Min-Su Kang, Gyeong-Hoon Lee

{ktw0114, mskang, ghlee3401}@ncsoft.com

Abstract

Contents

1. Audio Samples using Parallel WaveGAN (48k)

2. Audio Samples using WORLD vocoder

Sentence: 사라져가는 저 별빛 사이로 감출 순 없나요 커져버린 마음 (Pronunciation): salajyeoganeun jeo byeolbich sailo gamchul sun eobsnayo keojyeobeolin ma-eum
XiaoiceSing	Multi-Task (Proposed)	Copy-Synthesis (Ground Truth)