International Audio Laboratories Erlangen

Target Speaker Adaptation in Text-to-Speech Synthesis: A Comparison of Efficient Fine-Tuning and Zero-Shot Methods

Kishor Kayyar Lakshminarayana, Frank Zalkow, Christian Dittmar, Nicola Pia, Emanuël A.P. Habets

Published in EURASIP Journal on Audio, Speech, and Music Processing here.

Abstract

Neural text-to-speech (TTS) systems can synthesize highly natural speech. A key capability is speaker adaptation, which enables speech synthesis that matches a target speaker’s voice characteristics, such as timbre, pitch, and prosody. Traditional neural approaches require extensive speaker-specific data and full retraining, making them resource-intensive. Recent target-speaker adaptation techniques fall into two broad categories: (i) fine-tuning-based few-shot methods, which adapt models using small amounts of target-speaker data, and (ii) zero-shot methods, which generalize to unseen speakers without additional training, using a single reference sample during inference. This work compares these two adaptation approaches, along with a training-from-scratch baseline, across different metrics. We show that fine-tuning a non-autoregressive architecture, such as ForwardTacotron[1], achieves speaker-adaptation quality comparable to zero-shot methods, with lower computational complexity but a lower mean opinion score. As a second contribution, we show that fine-tuning can be made more efficient in terms of data and while largely retaining quality.

Comparison of Neural TTS Methods

We compare different state-of-the-art TTS methods across different parameters below. The encoder here refers to the text-to-token model and decoder refers to the token-to-speech model. The real time factor (RTF) was measured on an NVIDIA RTX2080 GPU with 12GB memory as the ratio of generated speech duration to synthesis time. Training times of zero-shot models are from the original authors using A100 (80GB) GPUs. AdapterMix and ForwardTacotron times are for RTX2080 GPUs (12GB). Using A100s here would slightly reduce training times. HierSpeech++ was trained using multiple GPUs (e.g. eight A6000 GPUs for LibriTTS). The minimum estimated training time tabulated here assumes 4 A100 GPUs for 4 days.

Audio Examples

Here, we share a few samples synthesized using the models listed above for two target speakers, along with the corresponding text. Adapter-Mix was fine-tuned using twenty minutes of target speaker data. ForwardTacotron versions utilized all available data for the target speaker, which was 7 hours for 'Ian' and 23 hours for '92'.

Efficient Fine-tuning with ForwardTacotron

We employ noise augmentation during fine-tuning to address low-resource data scenarios, similar to our prior work [8]. Additionally, we freeze selected parameters during fine-tuning, thereby avoiding the computation of their gradients and updates, which reduces computational cost and improves parameter efficiency. Some audio samples synthesized using models trained with these approaches are provided below.

¹Parameter efficient version with only 13.34M trainable parameters whereas the other models had 23.03M trainable parameters.

References

1. Christian Schäfer, Ollie McCarthy, and contributors
   ForwardTacotron
   https://github.com/as-ideas/ForwardTacotron, 2020. Bibtex

   @misc{Schaefer20_ForwardTacotron_Github,
   author = {Christian Schäfer and Ollie McCarthy and contributors},
   howpublished = {\url{https://github.com/as-ideas/ForwardTacotron}},
   journal = {GitHub repository},
   publisher = {GitHub},
   title = {{ForwardTacotron}},
   year = {2020}
   }

2. Lee, Sang-Hoon, Choi, Ha-Yeong, Kim, Seung-Bin, and Lee, Seong-Whan
   HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis
   IEEE Transactions on Neural Networks and Learning Systems, 36(10): 18422–18436, 2025. Bibtex DOI

   @article{lee2025_itnnls_hierspeech++,
   author={Lee, Sang-Hoon and Choi, Ha-Yeong and Kim, Seung-Bin and Lee, Seong-Whan},
   journal={IEEE Transactions on Neural Networks and Learning Systems},
   title={HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis},
   year={2025},
   volume={36},
   number={10},
   pages={18422-18436},
   keywords={Speech synthesis;Adaptation models;Acoustics;Semantics;Diffusion models;Synthesizers;Cloning;Artificial intelligence;Robustness;Transformers;Voice conversion (VC);zero-shot text-to-speech (TTS)},
   doi={10.1109/TNNLS.2025.3584944}
   }

3. Wang, Yuancheng, Zhan, Haoyue, Liu, Liwei, Zeng, Ruihong, Guo, Haotian, Zheng, Jiachen, Zhang, Qiang, Zhang, Xueyao, Zhang, Shunsi, and Wu, Zhizheng
   MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer
   arXiv preprint arXiv:2409.00750, 2024. Bibtex

   @article{wang2024_arxiv_maskgct,
   title={MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer},
   author={Wang, Yuancheng and Zhan, Haoyue and Liu, Liwei and Zeng, Ruihong and Guo, Haotian and Zheng, Jiachen and Zhang, Qiang and Zhang, Xueyao and Zhang, Shunsi and Wu, Zhizheng},
   journal={arXiv preprint arXiv:2409.00750},
   year={2024}
   }

4. Guo, Hao-Han, Hu, Yao, Shen, Fei-Yu, Tang, Xu, Wu, Yi-Chen, Xie, Feng-Long, and Xie, Kun
   Fireredtts-1S: An upgraded streamable foundation text-to-speech system
   arXiv preprint arXiv:2503.20499, 2025. Bibtex

   @article{guo2025_arxiv_fireredtts,
   title={{Fireredtts-1S}: An upgraded streamable foundation text-to-speech system},
   author={Guo, Hao-Han and Hu, Yao and Shen, Fei-Yu and Tang, Xu and Wu, Yi-Chen and Xie, Feng-Long and Xie, Kun},
   journal={arXiv preprint arXiv:2503.20499},
   year={2025}
   }

5. Chen, Yushen, Niu, Zhikang, Ma, Ziyang, Deng, Keqi, Wang, Chunhui, Zhao, Jian, Yu, Kai, and Chen, Xie
   F5-TTS: A fairytaler that fakes fluent and faithful speech with flow matching
   arXiv preprint arXiv:2410.06885, 2024. Bibtex

   @article{chen2024_arxiv_f5tts,
   title={{F5-TTS}: A fairytaler that fakes fluent and faithful speech with flow matching},
   author={Chen, Yushen and Niu, Zhikang and Ma, Ziyang and Deng, Keqi and Wang, Chunhui and Zhao, Jian and Yu, Kai and Chen, Xie},
   journal={arXiv preprint arXiv:2410.06885},
   year={2024}
   }

6. Ambuj Mehrish, Abhinav Ramesh Kashyap, Li Yingting, Navonil Majumder, and Soujanya Poria
   AdapterMix: Exploring the efficacy of mixture of adapters for low-resource TTS adaptation
   In Proc. Interspeech: 4284–4288, 2023. Bibtex DOI

   @inproceedings{mehrish23_interspeech_adaptermix,
   author={Ambuj Mehrish and Abhinav {Ramesh Kashyap} and Li Yingting and Navonil Majumder and Soujanya Poria},
   title={{AdapterMix: Exploring the efficacy of mixture of adapters for low-resource TTS adaptation}},
   year=2023,
   booktitle={Proc. Interspeech},
   pages={4284--4288},
   doi={10.21437/Interspeech.2023-1568}
   }

7. Frank Zalkow, Paolo Sani, Michael Fast, Judith Bauer, Mohammad Joshaghani, Kishor Kayyar, Emanuël A. P. Habets, and Christian Dittmar
   The AudioLabs System for the Blizzard Challenge 2023
   In Proceedings of the Blizzard Challenge Workshop: 63–68, 2023. Bibtex

   @inproceedings{ZalkowEtAl23_AudioLabs_Blizzard,
   address = {Grenoble, France},
   author = {Frank Zalkow and Paolo Sani and Michael Fast and Judith Bauer and Mohammad Joshaghani and Kishor Kayyar and Emanu{\"e}l A. P. Habets and Christian Dittmar},
   booktitle = {Proceedings of the Blizzard Challenge Workshop},
   pages = {63--68},
   title = {The {AudioLabs} System for the {B}lizzard {C}hallenge 2023},
   year = {2023}
   }

8. Kishor Kayyar, Frank Zalkow, Christian Dittmar, Nicola Pia, and Emanuël A. P. Habets
   Low-Resource Text-to-Speech Synthesis Using Noise-Augmented Training of ForwardTacotron
   In Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP): 1–5, 2025. Bibtex DOI

   @inproceedings{kayyar25_icassp_lowres,
   author={Kishor Kayyar and Frank Zalkow and Christian Dittmar and Nicola Pia and Emanu{\"e}l A. P. Habets},
   booktitle={Proc. {IEEE} Intl. Conf. on Acoustics, Speech and Signal Processing
   (ICASSP)},
   title={Low-Resource Text-to-Speech Synthesis Using Noise-Augmented Training of ForwardTacotron},
   year={2025},
   volume={},
   number={},
   pages={1-5},
   doi={10.1109/ICASSP49660.2025.10890686}
   }