MultiVerse: Efficient and Expressive Zero-Shot Multi-Task Text-to-Speech

Abstract

Text-to-speech (TTS) systems that scale up the amount of training data have achieved significant improvements in zero-shot speech synthesis. However, these systems have certain limitations: they require a large amount of training data, which increases costs, and often overlook prosody similarity. To address these issues, we propose MultiVerse, a zero-shot multi-task TTS system capable of performing TTS and speech style transfer in zero-shot and cross-lingual conditions, while requiring much less training data than traditional data-driven approaches. To ensure zero-shot performance even with limited data, we leverage source-filter theory-based disentanglement, utilizing the prompt for modeling filter-related and source-related representations. Additionally, to further enhance prosody similarity, we adopt a prosody modeling approach combining prompt-based autoregressive and non-autoregressive methods. Evaluations demonstrate the remarkable zero-shot multi-task TTS performance of MultiVerse and show that MultiVerse not only achieves zero-shot TTS performance comparable to data-driven TTS systems with much less data, but also significantly outperforms other zero-shot TTS systems trained with the same small amount of data. In particular, our innovative prosody modeling technique enables Multiverse to generate speech with a high degree of prosody similarity to the given prompts.

Model Overview

Interpolate start reference image.

Overall structure of MultiVerse. The acoustic model and the auto-regressive prosody predictor are on the left and right side of the figure, respectively. During training, overall modules are trained together, except the pre-trained speaker encoder. Multi-task TTS can be accomplished by varying input conditions.

LJ Multi-task demo (Zero-shot)

This is section for LJ Multi-task demo (Zero-shot). Note that LJSpeech has not been used in training our model.

Pitch control
(+6, from Zero-shot (EN))
Prompt Duration control
(0.5x, from Cross-lingual (KO))

This paper introduces, MultiVerse, a novel deep learning model for multi task speech synthesis.

Printing, in the only sense with which we are at present concerned, differs from most if not from all the arts and crafts represented in the Exhibition.

이 논문에서는 멀티태스크 음성 합성을 위한 새로운 딥러닝 모델인 멀티버스를 소개합니다.

Zero-shot (EN) Code-switching (EN & KO) Cross-lingual (KO)

This paper introduces, MultiVerse, a novel deep learning model for multi task speech synthesis.

이 paper에서는 multi task speech synthesis를 위한 새로운 deep learning model인 MultiVerse를 소개합니다.

이 논문에서는 멀티태스크 음성 합성을 위한 새로운 딥러닝 모델인 멀티버스를 소개합니다.

Style transfer Style prompt Cross-lingual style transfer

His palms are sweaty, knees weak, arms are heavy. There's vomit on his sweater already, mom's spaghetti. He's nervous, but on the surface, he looks calm and ready. To drop bombs but he keeps on forgetting.

His palms are sweaty, knees weak, arms are heavy. There's vomit on his sweater already, mom's spaghetti. He's nervous, but on the surface, he looks calm and ready. To drop bombs but he keeps on forgetting.

산속의 바람이 부드럽게 부리, 푸른 나무가 살짝 스치리, 알록달록 꽃들이 화려하게 피리, 시냇물이 조용히 출렁이리.

Zero-shot demo for celebrity

This is section for zero-shot demo for celebrity.

Celeb Text Prompt MultiVerse

Iron man

Text-to-speech (TTS) systems that scale up the amount of training data have achieved significant improvements in zero-shot speech synthesis.

Spider-man

However, these systems have certain limitations: they require a large amount of training data, which increases costs, and often overlook prosody similarity.

Wanda maximoff

To address these issues, we propose MultiVerse, a zero-shot multi-task TTS system capable of performing TTS and speech style transfer in zero-shot and cross-lingual conditions, while requiring much less training data than traditional data-driven approaches.

Black widow

To ensure zero-shot performance even with limited data, we leverage source-filter theory-based disentanglement, utilizing the prompt for modeling filter-related and source-related representations.

Olaf (Frozen)

Additionally, to further enhance prosody similarity, we adopt a prosody modeling approach combining prompt-based autoregressive and non-autoregressive methods.

Jocker

Evaluations demonstrate the remarkable zero-shot multi-task TTS performance of MultiVerse and show that MultiVerse not only achieves zero-shot TTS performance comparable to data-driven TTS systems with much less data.

Captain america

It also significantly outperforms other zero-shot TTS systems trained with the same small amount of data.

Homer Simpson

In particular, our innovative prosody modeling technique enables Multiverse to generate speech with a high degree of prosody similarity to the given prompts.

Zero-shot scenario (Intra-lingual / EN)

This is section for zero-shot scenario (Intra-lingual / EN). Please click the below toggle button and enjoy the samples!

Toggle Button 1

Neutral

Text Prompt Ground Truth GANSpeech+ YourTTS MultiVerse

Fitzgerald was still trying to find out how the germ had been transmitted.

In all our history you are the first people from outside our borders who have ever stepped a foot in our land.

The native American has been generally despised by his white conquerors for his poverty and simplicity.

The guard chanced to be Lumpy Bates, and he administered, what to him, was a gentle kick, to hurry the boy along.

Expressive

Text Prompt Ground Truth GANSpeech+ YourTTS MultiVerse

Anti aging creams can bring on breakouts.

Could you start researching boarding schools?

But some other folks are making silly sounds!

Dishes available at Kitchen Pantry Parlor.


Zero-shot scenario (Intra-lingual / KO)

This is section for zero-shot scenario (Intra-lingual / KO). Please click the below toggle button and enjoy the samples!

Toggle Button 2

Neutral

Text Prompt Ground Truth GANSpeech+ YourTTS MultiVerse

오늘 바람이 센 편이니 조심하세요.
(oneul barami sen pyeonini josimhaseyo.)

진짜 큰 문제는 시작도 안 됐다.
(jinjja keun munjeneun sijakdo an dwaessda.)

내일 동작구의 풍향은 북풍으로 관측됩니다.
(naeil dongjakguui punghyangeun bukpungeuro gwancheukdoepnida.)

도서관에 비치된 월간지를 검색할게요.
(doseogwane bichidoen wolganjireul geomsaekhalgeyo.)

Expressive

Text Prompt Ground Truth GANSpeech+ YourTTS MultiVerse

마음이 편한 곳으로 가더니 만족한 표정을 지었다.
(maeumi pyeonhan goseuro gadeoni manjokhan pyojeongeul jieossda.)

누나 오빠라고 부를수 없는 처지가 괴로웠다.
(nuna opparago bureulsu eopsneun cheojiga goerowossda.)

다음은 나의 주위에 대한 실망이다.
(daeumeun naui juwie daehan silmangida.)

요즘 놀이터에는 방방이가 있대요!
(yojeum noriteoeneun bangbangiga issdaeyo!)


Zero-shot scenario (Cross-lingual / KO -> EN)

This is section for zero-shot scenario (Cross-lingual / KO -> EN). Please click the below toggle button and enjoy the samples!

Toggle Button 3

Neutral

Text Prompt GANSpeech+ YourTTS MultiVerse

They have too often shown their patriotism by murdering their generals, underrating their enemies and slighting their friends.

I think you ought to write and tell him, and perhaps invite him here for the holidays.

And again, it would be hard if I were not allowed to investigate the pedigree of the woman I hope to win for my wife.

I will lie down in my clothes, and keep watch, ready to listen, or to follow her if need be.

Expressive

Text Prompt GANSpeech+ YourTTS MultiVerse

few women of my age that haven't seen a good deal of family troubles and family secrets.

He looked from the miniature to the crest and back to the miniature again, then at Alice.

Do you think we will learn anything from him? We will learn all we wish to know as to where he obtained his material.

You have come to the wrong place, she flung in.


Zero-shot scenario (Cross-lingual / EN -> KO)

This is section for zero-shot scenario (Cross-lingual / EN -> KO). Please click the below toggle button and enjoy the samples!

Toggle Button 4

Neutral

Text Prompt GANSpeech+ YourTTS MultiVerse

팔레스타인 인들은 예루살렘으로 계속 진군하겠습니다.
(palleseutain indeureun yerusallemeuro gyesok jingunhagessseupnida.)

이미 소매치기 수법과 조직을 꿰고 있던 서씨에게는 소매치기가 낯설지 않았습니다.
(imi somaechigi subeopgwa jojigeul kkwego issdeon seossiegeneun somaechigiga naccseolji anhassseupnida.)

연씨와 언론의 이 같은 숨바꼭질은 일일이 처음은 아닙니다.
(yeonssiwa eonronui i gateun sumbakkokjireun iriri cheoeumeun anipnida.)

이렇게 논란이 확산되자 백화점 관계자들은 내일 긴급 회의를 열어 경품자제를 협의하기로 했습니다.
(ireohge nonrani hwaksandoeja baekhwajeom gwangyejadeureun naeil gingeup hoeuireul yeoreo gyeongpumjajereul hyeobuihagiro haessseupnida.)

Expressive

Text Prompt GANSpeech+ YourTTS MultiVerse

옹달샘 주위는 한겨울에도 푸르름을 자랑해 주위와 대조를 이룹니다.
(ongdalsaem juwineun hangyeouredo pureureumeul jaranghae juwiwa daejoreul irupnida.)

인구뿐만 아니라 지역 대표성도 고려해야 한다는 주장이었습니다.
(inguppunman anira jiyeok daepyoseongdo goryeohaeya handaneun jujangieossseupnida.)

비를 무릅쓰고 벼를 일으켜 세우지만 속수무책입니다.
(bireul mureupsseugo byeoreul ireukyeo seujiman soksumuchaegipnida.)

유리창은 산산조각이 났고 타이어까지도 펑크가 났습니다.
(yurichangeun sansanjogagi nassgo taieokkajido peongkeuga nassseupnida.)


Comparison with Data-Driven Models

This is section for Comparison with Data-Driven Models. Please click the below toggle button and enjoy the samples!
Note that the data of the language used as the prompt for cross-lingual scenarios are not included in the our training data.
(The numbers in parentheses next to the model name indicate the amount of the training data in hours.)

Toggle Button 5

VALL-E (Intra-lingual / EN)

Text Prompt Ground Truth VALL-E MultiVerse

The army found the people in poverty and left them in comparative wealth.

We’ve made a couple of albums.

We have to reduce the number of plastic bags.

So what is the campaign about?

VALL-E X (Cross-lingual / ZH -> EN)

Text Prompt VALL-E X MultiVerse

There could be little art in this last and final round of fencing.

Look a little closer while our guide lets the light of his lamp fall upon the black wall at your side.

It was youth and poverty and proximity and everything was young and kindly.

He honours whatever he recognizes in himself, such morality equals self glorification.

NaturalSpeech2 (Intra-lingual / EN)

Text Prompt Ground Truth NaturalSpeech2 (44K) MultiVerse

Indeed, there were only one or two strangers who could be admitted among the sisters without producing the same result.

The air and the earth are curiously mated and intermingled as if the one were the breath of the other.

His death in this conjuncture was a public misfortune.

It is this that is of interest to theory of knowledge.

For a few miles, she followed the line hitherto presumably occupied by the coast of Algeria, but no land appeared to the south.

It is an absolute nonsense.

We have a long way to go this week.

We will turn the corner.

It will also require a lengthy series of clinical trials.

Mega-TTS (Intra-lingual / EN)

Text Prompt Ground Truth Mega-TTS (20K) MultiVerse

Instead of shoes, the old man wore boots with turnover tops, and his blue coat had wide cuffs of gold braid.

The army found the people in poverty and left them in comparative wealth.

He was in deep converse with the clerk and entered the hall holding him by the arm.

So what is the campaign about?

Nothing is yet confirmed.

Her husband was very concerned that it might be fatal.

We've made a couple of albums.

Mega-TTS (Cross-lingual / ZH -> EN)

Text Prompt Mega-TTS (20K) MultiVerse

He honours whatever he recognizes in himself, such morality equals self-glorification.

There could be little art in this last and final round of fencing.

It's the first time Hilda has been to our house and Tom introduces her around.

It was youth and poverty and proximity and everything was young and kindly.

Voicebox (Intra-lingual / EN)

Text Prompt Voicebox (60K) MultiVerse

The army found the people in poverty and left them in comparative wealth.

He was in deep converse with the clerk and entered the hall holding him by the arm.

Number ten fresh nelly is waiting on you good night husband.

When feline magicians enchant the city and crafty canine illusionists work to restore balance, don't miss the uproarious clash in magic and mischief. the paws of mystery.

peter piper picked a peck of pickled peppers.

Voicebox is the swiss army knife of text to speech acing multiple languages, changing voice styles, and dishing out custom samples.

In a land where cat pirates sail the high seas and dog buccaneers chase their tails, embark on a swashbuckling comedy adventure in furry buccaneers. the quest for the golden bone.

Voicebox (Cross-lingual / ES, FR, DE, PL, PT -> EN)

Text Prompt Voicebox (50K) MultiVerse

(ES -> EN) In gentle phrase in high flown classical spirit to the priest they send her she gives thanks but needs not any shriving any ghostly or other aid from him.

(FR -> EN) It is more than probable he might learn of him the grounds upon which he accustomed himself to so spare a diet fifteen to what purpose said solon should i trouble him.

(DE -> EN) Herodotus describing a war which had been going on for some years between the lydians and the medes gives the following account of the circumstances which led to its premature termination.

(PL -> EN) When indeed we consider how busily some insects have been shown to be engaged in conveying anther dust from flower to flower.

(PT -> EN) When they imported the negation of such or such a particular mode of agency for instance not to strike such a person or such a thing or in such a direction ten.

Mega-TTS2 (Intra-lingual / EN)

Text Prompt Ground Truth Mega-TTS2 (60K) MultiVerse

It is true that the horses are here, but the Hurons are gone; let us, then, hunt for the path by which they parted.

If I must say it, mother, I want to go away, and get out of this dead level.

The first mill plant was placed in the woollen factory of james Harrison at Newburgh, New York, about september fifteenth eighteen eighty one.

There their sad condition evoked for a time general commiseration.

Now in this of Burne Jones, the landscape is clearly full of light everywhere, color or glass light: that is, the outline is prepared for modification of color only.

When they entered the stage box on the left the first act was well under way, the scene being the interior of a cabin in the south of Ireland.

Mega-TTS2 (Style transfer)

Text Timbre prompt Style prompt Mega-TTS2 (60K) MultiVerse

You have the right to abandon me, and I have the capital to make you regret.

My heart aches with sadness and tears stream down my face.

She bounced into the classroom, full of the joys of spring.


Style transfer

This section is for style transfer samples. Please click the below toggle button and enjoy the samples!!

Toggle Button 6

Pair

Text Timbre prompt Style prompt Daft-exprt MultiVerse

내가 만든 간식 좀 먹어볼래? 방금 구운 쿠키야!
(naega mandeun gansik jom meogeobollae? banggeum guun kukiya!)

누가 현혹됐다는 거야! 너야말로 때와 장소를 가려!
(nuga hyeonhokdwaessdaneun geoya! neoyamallo ttaewa jangsoreul garyeo!)

전사한 전우들을 위하여! 우리는 승리할 것이다!
(jeonsahan jeonudeureul wihayeo! urineun seungrihal geosida!)

Unpair

Text Timbre prompt Style prompt Daft-exprt MultiVerse

오늘 청주 날씨는 오전에 잠깐 비가 내리고, 오후엔 화창할 예정이라고 하네요.
(oneul cheongju nalssineun ojeone jamkkan biga naerigo, ohuen hwachanghal yejeongirago haneyo.)

캘리포니아의 따뜻한 햇살을 상상해보면, 추위를 어느 정도 견딜 수 있더라고요.
(kaelliponiaui ttatteushan haessareul sangsanghaebomyeon, chuwireul eoneu jeongdo gyeondil su issdeoragoyo.)

그런 막막함에 부딪힐 때마다 저도 모르게 기도를 하게 되더라구요.
(geureon makmakhame budijhil ttaemada jeodo moreuge gidoreul hage doedeoraguyo.)


Controllability

This section is for controllability samples. Please click the below toggle button and enjoy the samples!

Toggle Button 7

Text

KT 위즈 외국인투수 윌리엄 쿠에바스가 KBO 8월 월간 최우수선수로 선정됐다.
(KT wijeu oegugintusu willieom kuebaseuga KBO palwol wolgan choeususeonsuro seonjeongdwaessda.)

Pitch

pitch-contour
-6 -4 -2 0 +2 +4 +6

Duration

0.5x 0 2x

Appendix

This section is for appendix samples. Please click the below toggle button and enjoy the samples!

Toggle Button 8

Analysis on content and style representations

Content represnetation Style representation MultiVerse
filter image source image combined image

Sample-adaptive kernel selection

adaptive image.
MultiVerse w/o sample-adaptive kernel selection MultiVerse

Comparison with Vector Quantization based Prosody Modeling

vq
(a) Ground Truth (b) MultiVerse (c) MultiVerse(VQ)

Audio Samples for Rebuttal

This is section for Audio Samples for Rebuttal. Please click the below toggle button and enjoy the samples!

Toggle Button 9

Wide range emotion (KO -> KO)

Text Prompt MultiVerse

우린 비닐봉지 사용을 줄여야 해 (Angry)

우린 비닐봉지 사용을 줄여야 해 (Baseball)

우린 비닐봉지 사용을 줄여야 해 (Fear)

우린 비닐봉지 사용을 줄여야 해 (Happy)

우린 비닐봉지 사용을 줄여야 해 (Sad)

Wide range emotion (KO -> EN)

Text Prompt MultiVerse

We have to reduce the number of plastic bags. (Angry)

We have to reduce the number of plastic bags. (Baseball)

We have to reduce the number of plastic bags. (Fear)

We have to reduce the number of plastic bags. (Happy)

We have to reduce the number of plastic bags. (Sad)