This is section for LJ Multi-task demo (Zero-shot). Note that LJSpeech has not been used in training our model.
Text-to-speech (TTS) systems that scale up the amount of training data have achieved significant improvements in zero-shot speech synthesis. However, these systems have certain limitations: they require a large amount of training data, which increases costs, and often overlook prosody similarity. To address these issues, we propose MultiVerse, a zero-shot multi-task TTS system capable of performing TTS and speech style transfer in zero-shot and cross-lingual conditions, while requiring much less training data than traditional data-driven approaches. To ensure zero-shot performance even with limited data, we leverage source-filter theory-based disentanglement, utilizing the prompt for modeling filter-related and source-related representations. Additionally, to further enhance prosody similarity, we adopt a prosody modeling approach combining prompt-based autoregressive and non-autoregressive methods. Evaluations demonstrate the remarkable zero-shot multi-task TTS performance of MultiVerse and show that MultiVerse not only achieves zero-shot TTS performance comparable to data-driven TTS systems with much less data, but also significantly outperforms other zero-shot TTS systems trained with the same small amount of data. In particular, our innovative prosody modeling technique enables Multiverse to generate speech with a high degree of prosody similarity to the given prompts.
This is section for LJ Multi-task demo (Zero-shot). Note that LJSpeech has not been used in training our model.
Pitch control (+6, from Zero-shot (EN)) |
Prompt | Duration control (0.5x, from Cross-lingual (KO)) |
---|---|---|
This paper introduces, MultiVerse, a novel deep learning model for multi task speech synthesis. |
Printing, in the only sense with which we are at present concerned, differs from most if not from all the arts and crafts represented in the Exhibition. |
이 논문에서는 멀티태스크 음성 합성을 위한 새로운 딥러닝 모델인 멀티버스를 소개합니다. |
Zero-shot (EN) | Code-switching (EN & KO) | Cross-lingual (KO) |
---|---|---|
This paper introduces, MultiVerse, a novel deep learning model for multi task speech synthesis. |
이 paper에서는 multi task speech synthesis를 위한 새로운 deep learning model인 MultiVerse를 소개합니다. |
이 논문에서는 멀티태스크 음성 합성을 위한 새로운 딥러닝 모델인 멀티버스를 소개합니다. |
Style transfer | Style prompt | Cross-lingual style transfer |
---|---|---|
His palms are sweaty, knees weak, arms are heavy. There's vomit on his sweater already, mom's spaghetti. He's nervous, but on the surface, he looks calm and ready. To drop bombs but he keeps on forgetting. |
His palms are sweaty, knees weak, arms are heavy. There's vomit on his sweater already, mom's spaghetti. He's nervous, but on the surface, he looks calm and ready. To drop bombs but he keeps on forgetting. |
산속의 바람이 부드럽게 부리, 푸른 나무가 살짝 스치리, 알록달록 꽃들이 화려하게 피리, 시냇물이 조용히 출렁이리. |
This is section for zero-shot demo for celebrity.
Celeb | Text | Prompt | MultiVerse |
---|---|---|---|
Iron man |
Text-to-speech (TTS) systems that scale up the amount of training data have achieved significant improvements in zero-shot speech synthesis. |
||
Spider-man |
However, these systems have certain limitations: they require a large amount of training data, which increases costs, and often overlook prosody similarity. |
||
Wanda maximoff |
To address these issues, we propose MultiVerse, a zero-shot multi-task TTS system capable of performing TTS and speech style transfer in zero-shot and cross-lingual conditions, while requiring much less training data than traditional data-driven approaches. |
||
Black widow |
To ensure zero-shot performance even with limited data, we leverage source-filter theory-based disentanglement, utilizing the prompt for modeling filter-related and source-related representations. |
||
Olaf (Frozen) |
Additionally, to further enhance prosody similarity, we adopt a prosody modeling approach combining prompt-based autoregressive and non-autoregressive methods. |
||
Jocker |
Evaluations demonstrate the remarkable zero-shot multi-task TTS performance of MultiVerse and show that MultiVerse not only achieves zero-shot TTS performance comparable to data-driven TTS systems with much less data. |
||
Captain america |
It also significantly outperforms other zero-shot TTS systems trained with the same small amount of data. |
||
Homer Simpson |
In particular, our innovative prosody modeling technique enables Multiverse to generate speech with a high degree of prosody similarity to the given prompts. |
This is section for zero-shot scenario (Intra-lingual / EN). Please click the below toggle button and enjoy the samples!
Neutral
Text | Prompt | Ground Truth | GANSpeech+ | YourTTS | MultiVerse |
---|---|---|---|---|---|
Fitzgerald was still trying to find out how the germ had been transmitted. |
|||||
In all our history you are the first people from outside our borders who have ever stepped a foot in our land. |
|||||
The native American has been generally despised by his white conquerors for his poverty and simplicity. |
|||||
The guard chanced to be Lumpy Bates, and he administered, what to him, was a gentle kick, to hurry the boy along. |
Expressive
Text | Prompt | Ground Truth | GANSpeech+ | YourTTS | MultiVerse |
---|---|---|---|---|---|
Anti aging creams can bring on breakouts. |
|||||
Could you start researching boarding schools? |
|||||
But some other folks are making silly sounds! |
|||||
Dishes available at Kitchen Pantry Parlor. |
This is section for zero-shot scenario (Intra-lingual / KO). Please click the below toggle button and enjoy the samples!
Neutral
Text | Prompt | Ground Truth | GANSpeech+ | YourTTS | MultiVerse |
---|---|---|---|---|---|
오늘 바람이 센 편이니 조심하세요. |
|||||
진짜 큰 문제는 시작도 안 됐다. |
|||||
내일 동작구의 풍향은 북풍으로 관측됩니다. |
|||||
도서관에 비치된 월간지를 검색할게요. |
Expressive
Text | Prompt | Ground Truth | GANSpeech+ | YourTTS | MultiVerse |
---|---|---|---|---|---|
마음이 편한 곳으로 가더니 만족한 표정을 지었다. |
|||||
누나 오빠라고 부를수 없는 처지가 괴로웠다. |
|||||
다음은 나의 주위에 대한 실망이다. |
|||||
요즘 놀이터에는 방방이가 있대요! |
This is section for zero-shot scenario (Cross-lingual / KO -> EN). Please click the below toggle button and enjoy the samples!
Neutral
Text | Prompt | GANSpeech+ | YourTTS | MultiVerse |
---|---|---|---|---|
They have too often shown their patriotism by murdering their generals, underrating their enemies and slighting their friends. |
||||
I think you ought to write and tell him, and perhaps invite him here for the holidays. |
||||
And again, it would be hard if I were not allowed to investigate the pedigree of the woman I hope to win for my wife. |
||||
I will lie down in my clothes, and keep watch, ready to listen, or to follow her if need be. |
Expressive
Text | Prompt | GANSpeech+ | YourTTS | MultiVerse |
---|---|---|---|---|
few women of my age that haven't seen a good deal of family troubles and family secrets. |
||||
He looked from the miniature to the crest and back to the miniature again, then at Alice. |
||||
Do you think we will learn anything from him? We will learn all we wish to know as to where he obtained his material. |
||||
You have come to the wrong place, she flung in. |
This is section for zero-shot scenario (Cross-lingual / EN -> KO). Please click the below toggle button and enjoy the samples!
Neutral
Text | Prompt | GANSpeech+ | YourTTS | MultiVerse |
---|---|---|---|---|
팔레스타인 인들은 예루살렘으로 계속 진군하겠습니다. |
||||
이미 소매치기 수법과 조직을 꿰고 있던 서씨에게는 소매치기가 낯설지 않았습니다. |
||||
연씨와 언론의 이 같은 숨바꼭질은 일일이 처음은 아닙니다. |
||||
이렇게 논란이 확산되자 백화점 관계자들은 내일 긴급 회의를 열어 경품자제를 협의하기로 했습니다. |
Expressive
Text | Prompt | GANSpeech+ | YourTTS | MultiVerse |
---|---|---|---|---|
옹달샘 주위는 한겨울에도 푸르름을 자랑해 주위와 대조를 이룹니다. |
||||
인구뿐만 아니라 지역 대표성도 고려해야 한다는 주장이었습니다. |
||||
비를 무릅쓰고 벼를 일으켜 세우지만 속수무책입니다. |
||||
유리창은 산산조각이 났고 타이어까지도 펑크가 났습니다. |
This is section for Comparison with Data-Driven Models. Please click the below toggle button and enjoy the samples!
Note that the data of the language used as the prompt for cross-lingual scenarios are not included in the our training data.
(The numbers in parentheses next to the model name indicate the amount of the training data in hours.)
VALL-E (Intra-lingual / EN)
Text | Prompt | Ground Truth | VALL-E | MultiVerse |
---|---|---|---|---|
The army found the people in poverty and left them in comparative wealth. |
||||
We’ve made a couple of albums. |
||||
We have to reduce the number of plastic bags. |
||||
So what is the campaign about? |
VALL-E X (Cross-lingual / ZH -> EN)
Text | Prompt | VALL-E X | MultiVerse |
---|---|---|---|
There could be little art in this last and final round of fencing. |
|||
Look a little closer while our guide lets the light of his lamp fall upon the black wall at your side. |
|||
It was youth and poverty and proximity and everything was young and kindly. |
|||
He honours whatever he recognizes in himself, such morality equals self glorification. |
NaturalSpeech2 (Intra-lingual / EN)
Text | Prompt | Ground Truth | NaturalSpeech2 (44K) | MultiVerse |
---|---|---|---|---|
Indeed, there were only one or two strangers who could be admitted among the sisters without producing the same result. |
||||
The air and the earth are curiously mated and intermingled as if the one were the breath of the other. |
||||
His death in this conjuncture was a public misfortune. |
||||
It is this that is of interest to theory of knowledge. |
||||
For a few miles, she followed the line hitherto presumably occupied by the coast of Algeria, but no land appeared to the south. |
||||
It is an absolute nonsense. |
||||
We have a long way to go this week. |
||||
We will turn the corner. |
||||
It will also require a lengthy series of clinical trials. |
Mega-TTS (Intra-lingual / EN)
Text | Prompt | Ground Truth | Mega-TTS (20K) | MultiVerse |
---|---|---|---|---|
Instead of shoes, the old man wore boots with turnover tops, and his blue coat had wide cuffs of gold braid. |
||||
The army found the people in poverty and left them in comparative wealth. |
||||
He was in deep converse with the clerk and entered the hall holding him by the arm. |
||||
So what is the campaign about? |
||||
Nothing is yet confirmed. |
||||
Her husband was very concerned that it might be fatal. |
||||
We've made a couple of albums. |
Mega-TTS (Cross-lingual / ZH -> EN)
Text | Prompt | Mega-TTS (20K) | MultiVerse |
---|---|---|---|
He honours whatever he recognizes in himself, such morality equals self-glorification. |
|||
There could be little art in this last and final round of fencing. |
|||
It's the first time Hilda has been to our house and Tom introduces her around. |
|||
It was youth and poverty and proximity and everything was young and kindly. |
Voicebox (Intra-lingual / EN)
Text | Prompt | Voicebox (60K) | MultiVerse |
---|---|---|---|
The army found the people in poverty and left them in comparative wealth. |
|||
He was in deep converse with the clerk and entered the hall holding him by the arm. |
|||
Number ten fresh nelly is waiting on you good night husband. |
|||
When feline magicians enchant the city and crafty canine illusionists work to restore balance, don't miss the uproarious clash in magic and mischief. the paws of mystery. |
|||
peter piper picked a peck of pickled peppers. |
|||
Voicebox is the swiss army knife of text to speech acing multiple languages, changing voice styles, and dishing out custom samples. |
|||
In a land where cat pirates sail the high seas and dog buccaneers chase their tails, embark on a swashbuckling comedy adventure in furry buccaneers. the quest for the golden bone. |
Voicebox (Cross-lingual / ES, FR, DE, PL, PT -> EN)
Text | Prompt | Voicebox (50K) | MultiVerse |
---|---|---|---|
(ES -> EN) In gentle phrase in high flown classical spirit to the priest they send her she gives thanks but needs not any shriving any ghostly or other aid from him. |
|||
(FR -> EN) It is more than probable he might learn of him the grounds upon which he accustomed himself to so spare a diet fifteen to what purpose said solon should i trouble him. |
|||
(DE -> EN) Herodotus describing a war which had been going on for some years between the lydians and the medes gives the following account of the circumstances which led to its premature termination. |
|||
(PL -> EN) When indeed we consider how busily some insects have been shown to be engaged in conveying anther dust from flower to flower. |
|||
(PT -> EN) When they imported the negation of such or such a particular mode of agency for instance not to strike such a person or such a thing or in such a direction ten. |
Mega-TTS2 (Intra-lingual / EN)
Text | Prompt | Ground Truth | Mega-TTS2 (60K) | MultiVerse |
---|---|---|---|---|
It is true that the horses are here, but the Hurons are gone; let us, then, hunt for the path by which they parted. |
||||
If I must say it, mother, I want to go away, and get out of this dead level. |
||||
The first mill plant was placed in the woollen factory of james Harrison at Newburgh, New York, about september fifteenth eighteen eighty one. |
||||
There their sad condition evoked for a time general commiseration. |
||||
Now in this of Burne Jones, the landscape is clearly full of light everywhere, color or glass light: that is, the outline is prepared for modification of color only. |
||||
When they entered the stage box on the left the first act was well under way, the scene being the interior of a cabin in the south of Ireland. |
Mega-TTS2 (Style transfer)
Text | Timbre prompt | Style prompt | Mega-TTS2 (60K) | MultiVerse |
---|---|---|---|---|
You have the right to abandon me, and I have the capital to make you regret. |
||||
My heart aches with sadness and tears stream down my face. |
||||
She bounced into the classroom, full of the joys of spring. |
This section is for style transfer samples. Please click the below toggle button and enjoy the samples!!
Pair
Text | Timbre prompt | Style prompt | Daft-exprt | MultiVerse |
---|---|---|---|---|
내가 만든 간식 좀 먹어볼래? 방금 구운 쿠키야! |
||||
누가 현혹됐다는 거야! 너야말로 때와 장소를 가려! |
||||
전사한 전우들을 위하여! 우리는 승리할 것이다! |
Unpair
Text | Timbre prompt | Style prompt | Daft-exprt | MultiVerse |
---|---|---|---|---|
오늘 청주 날씨는 오전에 잠깐 비가 내리고, 오후엔 화창할 예정이라고 하네요. |
||||
캘리포니아의 따뜻한 햇살을 상상해보면, 추위를 어느 정도 견딜 수 있더라고요. |
||||
그런 막막함에 부딪힐 때마다 저도 모르게 기도를 하게 되더라구요. |
This section is for controllability samples. Please click the below toggle button and enjoy the samples!
Text
KT 위즈 외국인투수 윌리엄 쿠에바스가 KBO 8월 월간 최우수선수로 선정됐다.
(KT wijeu oegugintusu willieom kuebaseuga KBO palwol wolgan choeususeonsuro seonjeongdwaessda.)
Pitch
-6 | -4 | -2 | 0 | +2 | +4 | +6 |
---|---|---|---|---|---|---|
Duration
0.5x | 0 | 2x |
---|---|---|
This section is for appendix samples. Please click the below toggle button and enjoy the samples!
Analysis on content and style representations
Content represnetation | Style representation | MultiVerse |
---|---|---|
Sample-adaptive kernel selection
MultiVerse w/o sample-adaptive kernel selection | MultiVerse |
---|---|
Comparison with Vector Quantization based Prosody Modeling
(a) Ground Truth | (b) MultiVerse | (c) MultiVerse(VQ) |
---|---|---|
This is section for Audio Samples for Rebuttal. Please click the below toggle button and enjoy the samples!
Wide range emotion (KO -> KO)
Text | Prompt | MultiVerse |
---|---|---|
우린 비닐봉지 사용을 줄여야 해 (Angry) |
||
우린 비닐봉지 사용을 줄여야 해 (Baseball) |
||
우린 비닐봉지 사용을 줄여야 해 (Fear) |
||
우린 비닐봉지 사용을 줄여야 해 (Happy) |
||
우린 비닐봉지 사용을 줄여야 해 (Sad) |
Wide range emotion (KO -> EN)
Text | Prompt | MultiVerse |
---|---|---|
We have to reduce the number of plastic bags. (Angry) |
||
We have to reduce the number of plastic bags. (Baseball) |
||
We have to reduce the number of plastic bags. (Fear) |
||
We have to reduce the number of plastic bags. (Happy) |
||
We have to reduce the number of plastic bags. (Sad) |