Audio samples from "High Fidelity Text-to-Speech Via Discrete Tokens Using Token Transducer and Group Masked Language Model"

Zero-Shot Adaptive TTS

The models are trained on LibriTTS (train-clean-100, train-clean-360, train-other-500).
Audio samples are sampled from the test-clean subset, so that the speakers are unseen speakers during training.

Text

A circle of a few hundred feet in circumference was drawn, and each of the party took a segment for his portion.

Ground Truth

Reference (Prompt)

VITS

VALLE-X

Kim et al.

Proposed

He was of sturdy, athletic build and dressed neatly in a suit that was of coarse material but well brushed and cared for.

Then the curtain rises, and it is apparent that we are assisting at an At Home of considerable splendour.

I like trees because they seem more resigned to the way they have to live than other things do..

Text

It has no beauty whatsoever, no specialty of picturesqueness; and all its lines are cramped and poor.

Ground Truth

Reference (Prompt)

VITS

VALLE-X

Kim et al.

Proposed

Whatever you meditate, he probably anticipates it-you know best-and you will find him prepared..

I then remembered that the Professor, before starting, had estimated the length of this underground sea at thirty leagues.

"But it is not with a view to distinction that you should cultivate this talent, if you consult your own happiness.

Controllability

To verify the paralinguistic controllability of each stage, we investigated the samples in the case of mismatched $p_{s}$ $\textbf{p}_{s}$ and $p_{a}$ $\textbf{p}_{a}$ .
- $p_{s}$ $\textbf{p}_{s}$ : speech prompt used for semantic token generation (Interpreting).
- $p_{a}$ $\textbf{p}_{a}$ : speech prompt used for acoustic token generation (Speaking).

$p_{s}$ $\textbf{p}_{s}$ controllability <Interpreting>

Here, we fix the $p_{a}$ $\textbf{p}_{a}$ and control the paralinguistic information using $p_{s}$ $\textbf{p}_{s}$ .
Thus, the generated samples share the same acoustic details (such as speaker and details) of $p_{a}$ $\textbf{p}_{a}$ ,
and different prosody attributes (such as temporal dynamics and speech rate) of $p_{s}$ $\textbf{p}_{s}$ are reflected in the generated speech.

Text

It seemed to him that the newspaper managers didn't want genius, but mere plodding and grubbing.

$p_{a}$ $\textbf{p}_{a}$ (fixed)

$p_{s}$ $\textbf{p}_{s}$ #1

Kim et al. #1

Proposed #1

$p_{s}$ $\textbf{p}_{s}$ #2

Kim et al. #2

Proposed #2

$p_{s}$ $\textbf{p}_{s}$ #3

Kim et al. #3

Proposed #3

$p_{s}$ $\textbf{p}_{s}$ #4

Kim et al. #4

Proposed #4

Text

What one useful thing could I do for a living, for the support of mother and the children

$p_{a}$ $\textbf{p}_{a}$ (fixed)

$p_{s}$ $\textbf{p}_{s}$ #1

Kim et al. #1

Proposed #1

$p_{s}$ $\textbf{p}_{s}$ #2

Kim et al. #2

Proposed #2

$p_{s}$ $\textbf{p}_{s}$ #3

Kim et al. #3

Proposed #3

$p_{s}$ $\textbf{p}_{s}$ #4

Kim et al. #4

Proposed #4

Text

But I thought I would make the offer, seeing that youth commonly loves life.

$p_{a}$ $\textbf{p}_{a}$ (fixed)

$p_{s}$ $\textbf{p}_{s}$ #1

Kim et al. #1

Proposed #1

$p_{s}$ $\textbf{p}_{s}$ #2

Kim et al. #2

Proposed #2

$p_{s}$ $\textbf{p}_{s}$ #3

Kim et al. #3

Proposed #3

$p_{s}$ $\textbf{p}_{s}$ #4

Kim et al. #4

Proposed #4

$p_{a}$ $\textbf{p}_{a}$ controllability <Speaking>

Here, we fix the generated semantic tokens using $p_{s}$ $\textbf{p}_{s}$ and control the paralinguistic information using $p_{a}$ $\textbf{p}_{a}$ .
Thus, the generated samples share the same prosody attributes (such as temporal dynamics and speech rate) of $p_{s}$ $\textbf{p}_{s}$ ,
and different acoustic details of $p_{a}$ $\textbf{p}_{a}$ are reflected in the generated speech.