Speech and Singing Voice Generation
Speech synthesis, singing voice synthesis, voice conversion, singing technique control, and zero-shot vocal modeling.
01 / Profile
About
I study generative AI for voice, music, and human motion, with an emphasis on controllable systems that support expressive creation.
I work where sound, intelligence, and human expression meet.
I am a Ph.D. student at the School of Computing, National University of Singapore, advised by Prof. Ye Wang in the Sound and Music Computing Lab.
My research covers speech and singing voice synthesis, neural audio codecs, music generation, talking head generation, co-speech gesture generation, and affective multimodal learning.
I am especially interested in models and interfaces that make generated media easier to guide, evaluate, and use in creative or communicative settings.
Before and alongside my research, I have spent many years studying piano and singing. This musical background shapes the questions I care about: how generated sound can remain controllable, expressive, and useful for real creative workflows.
For selected performances and music projects, check my music page.
02 / Research directions
Research Focus
Three threads connect most of my work: expressive voice generation, music and audio intelligence, and multimodal human signal modeling.
Music and Audio Intelligence
Neural audio codecs, controllable audio generation, piano transcription/rendering, music style transfer, and creative ML systems.
Audio-Driven Human Generation
Talking head generation, co-speech gesture generation, multimodal emotion analysis, and human-centered generation.
03 / Updates
Recent News
A compact timeline for papers, internships, teaching, and milestones.
- One paper has been accepted by Interspeech 2026.
- I am joining Tencent as a Research Intern in Speech Synthesis in Singapore.
- One paper has been accepted by ACL 2026 Main Conference.
- One journal paper has been accepted by IEEE TASLP 2026.
- One paper has been accepted by ICLR 2026.
- Two papers have been accepted by ICASSP 2026.
- One paper has been accepted by Interspeech 2025.
- I have been selected for the SoC Teaching Fellowship Scheme.
- One paper has been accepted by ICASSP 2025.
- One journal paper has been accepted by IEEE/ACM TASLP 2024.
- I started my Ph.D. journey at NUS in the SMC Lab.
- I received my Master's degree from the National University of Singapore (NUS).
- I joined THUNLP Lab as a Research Intern.
- I joined the SMC Lab as a Student Researcher.
- I received my Bachelor's degree from Beijing University of Posts and Telecommunications (BUPT).
04 / Research output
Publications
Research across voice, music, and human-centered multimodal systems. Browse by year, filter by field, or search the collection.
* Equal contribution · † Corresponding author
2026
Does Expert Supervision Improve Text-to-Speech? An Empirical Study of Preference Optimization
arXiv preprint, 2026.
Paper
Overview
An empirical study of expert-guided TTS preference optimization across two student models, real and synthetic speech, and direct and bridge supervision. It shows why demonstration quality and target likelihood alone do not reliably predict improvements in generated speech.
Cite
@misc{vu2026expertsupervision,
title = {Does Expert Supervision Improve Text-to-Speech? An Empirical Study of Preference Optimization},
author = {Minh Duc Vu and Junchuan Zhao and Ye Wang},
year = {2026},
eprint = {2610.25292},
archivePrefix = {arXiv},
primaryClass = {cs.SD},
doi = {10.48550/arXiv.2610.25292},
url = {https://arxiv.org/abs/2610.25292}
}X-MDD: LLM-Based Mispronunciation Detection and Diagnosis for Unseen Languages with Canonical References
arXiv preprint, 2026.
Paper
Overview
A multilingual pronunciation diagnosis model that combines canonical speech and phone references with in-context examples and GRPO. Trained on Mandarin and English, it evaluates transfer to Arabic without Arabic MDD training.
Cite
@misc{wang2026xmdd,
title = {X-MDD: LLM-Based Mispronunciation Detection and Diagnosis for Unseen Languages with Canonical References},
author = {Xintong Wang and Junchuan Zhao and Ye Wang},
year = {2026},
eprint = {2610.23079},
archivePrefix = {arXiv},
primaryClass = {cs.SD},
doi = {10.48550/arXiv.2610.23079},
url = {https://arxiv.org/abs/2610.23079}
}Edit Who Speaks, Control How They Speak: Global Timbre Editing and Local Instruction Control for TTS
arXiv preprint, 2026.
Paper
Demo
Overview
EDICT combines reference-based timbre editing with segment-specific expressive instructions, using an edited acoustic reference and KV-cache reconstruction to maintain voice consistency and acoustic continuity.
Cite
@misc{zhao2026editwhospeaks,
title = {Edit Who Speaks, Control How They Speak: Global Timbre Editing and Local Instruction Control for {TTS}},
author = {Junchuan Zhao and Chenglin Xu and Wei Zeng and Haoyang Li and Yiwen Guo and Ye Wang},
year = {2026},
eprint = {2610.11437},
archivePrefix = {arXiv},
primaryClass = {cs.SD},
doi = {10.48550/arXiv.2610.11437},
url = {https://arxiv.org/abs/2610.11437}
}Preference Optimization for Non-Verbal Vocalization Synthesis
arXiv preprint, 2026.
Paper
Overview
A study of preference signals, pair construction, and DPO objectives for non-verbal vocalization synthesis, introducing a non-verbal-aware character error rate to balance vocalization realization and lexical accuracy.
Cite
@misc{li2026preferenceoptimization,
title = {Preference Optimization for Non-Verbal Vocalization Synthesis},
author = {Haoyang Li and Chenglin Xu and Junchuan Zhao and Yuang Cao and Liumeng Xue and Yiwen Guo and Eng Siong Chng},
year = {2026},
eprint = {2608.24163},
archivePrefix = {arXiv},
primaryClass = {eess.AS},
doi = {10.48550/arXiv.2608.24163},
url = {https://arxiv.org/abs/2608.24163}
}AnyBand: Unified Multi-Bandwidth Speech Extension via Frequency-Aware In-Context Spectral Infilling
arXiv preprint, 2026.
Paper
Demo
Overview
A unified frequency-aware framework for speech bandwidth extension across multiple input cutoffs, formulated as in-context spectral infilling.
Cite
@misc{zhao2026anyband,
title = {{AnyBand}: Unified Multi-Bandwidth Speech Extension via Frequency-Aware In-Context Spectral Infilling},
author = {Junchuan Zhao and Minh Duc Vu and Bowen Zhang and Ye Wang},
year = {2026},
eprint = {2608.00572},
archivePrefix = {arXiv},
primaryClass = {cs.SD},
doi = {10.48550/arXiv.2608.00572},
url = {https://arxiv.org/abs/2608.00572}
}Joycent: Multi-Accent TTS via Disentangled Accent Modeling and Layer-Specific Conditioning
arXiv preprint, 2026.
Paper
Demo
Overview
A diffusion-based TTS framework for controlling accent while preserving speaker identity. It learns speaker-disentangled accent representations and applies accent and speaker conditioning at different encoder layers, including cross-accent synthesis with separate reference prompts.
Cite
@misc{wang2026joycent,
title = {Joycent: Multi-Accent TTS via Disentangled Accent Modeling and Layer-Specific Conditioning},
author = {Xintong Wang and Junchuan Zhao and Ye Wang},
year = {2026},
eprint = {2606.16417},
archivePrefix = {arXiv},
primaryClass = {cs.SD},
doi = {10.48550/arXiv.2606.16417},
url = {https://arxiv.org/abs/2606.16417}
}Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems
arXiv preprint, 2026.
Paper
Demo
Cite
@misc{chen2026remixtimbre,
title = {Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems},
author = {Leduo Chen and Junchuan Zhao and Shengchen Li},
year = {2026},
eprint = {2605.09259},
archivePrefix = {arXiv},
primaryClass = {cs.SD},
doi = {10.48550/arXiv.2605.09259},
url = {https://arxiv.org/abs/2605.09259}
}PersonaGest: Personalized Co-Speech Gesture Generation with Semantic-Guided Hierarchical Motion Representation
arXiv preprint, 2026.
Paper
Demo
Cite
@misc{zhao2026personagest,
title = {{PersonaGest}: Personalized Co-Speech Gesture Generation with Semantic-Guided Hierarchical Motion Representation},
author = {Junchuan Zhao and Qifan Liang and Ye Wang},
year = {2026},
eprint = {2605.07252},
archivePrefix = {arXiv},
primaryClass = {cs.GR},
doi = {10.48550/arXiv.2605.07252},
url = {https://arxiv.org/abs/2605.07252}
}MSpoofTTS: Multi-Resolution Spoof-Guided Inference for Discrete Speech Synthesis
Interspeech, 2026.
Paper
Demo
Code
Cite
@inproceedings{zhao26g_interspeech,
title = {{MSpoofTTS: Multi-Resolution Spoof-Guided Inference for Discrete Speech Synthesis}},
author = {Junchuan Zhao and Minh Duc Vu and Ye Wang},
year = {2026},
booktitle = {{Interspeech 2026}},
pages = {7441--7446},
doi = {10.21437/Interspeech.2026-2159},
issn = {2958-1796},
}CodecFlow: Efficient Bandwidth Extension via Conditional Flow Matching in Neural Codec Latent Space
arXiv preprint, 2026.
Paper
Demo
Cite
@misc{zhang2026codecflow,
title = {{CodecFlow}: Efficient Bandwidth Extension via Conditional Flow Matching in Neural Codec Latent Space},
author = {Bowen Zhang and Junchuan Zhao and Ian McLoughlin and Ye Wang and A S Madhukumar},
year = {2026},
eprint = {2603.02022},
archivePrefix = {arXiv},
primaryClass = {cs.SD},
doi = {10.48550/arXiv.2603.02022},
url = {https://arxiv.org/abs/2603.02022}
}TED-TTS: Training-Free Intra-Utterance Emotion and Duration Control for Text-to-Speech Synthesis
ACL Main, 2026.
Paper
Demo
Code
Data
Cite
@inproceedings{liang-etal-2026-ted,
title = {{TED}-{TTS}: Training-Free Intra-Utterance Emotion and Duration Control for Text-to-Speech Synthesis},
author = {Liang, Qifan and Liu, Yuansen and Wei, Ruixin and Lu, Nan and Zhao, Junchuan and Wang, Ye},
booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
year = {2026},
address = {San Diego, California, United States},
publisher = {Association for Computational Linguistics},
pages = {23485--23508},
doi = {10.18653/v1/2026.acl-long.1077},
url = {https://aclanthology.org/2026.acl-long.1077/}
}KSDiff: Keyframe-Augmented Speech-Aware Dual-Path Diffusion for Facial Animation
ICASSP, 2026.
Paper
Demo
Cite
@inproceedings{lyu2026ksdiff,
title = {{KSDIFF}: Keyframe-Augmented Speech-Aware Dual-Path Diffusion for Facial Animation},
author = {Lyu, Tianle and Zhao, Junchuan and Wang, Ye},
booktitle = {ICASSP 2026 - 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
year = {2026},
pages = {13132--13136},
doi = {10.1109/icassp55912.2026.11465063},
url = {https://doi.org/10.1109/icassp55912.2026.11465063}
}InconVAD: A Two-Stage Dual-Tower Framework for Multimodal Emotion Inconsistency Detection
ICASSP, 2026.
Paper
Cite
@inproceedings{li2026inconvad,
title = {{InconVAD}: A Two-Stage Dual-Tower Framework for Multimodal Emotion Inconsistency Detection},
author = {Li, Zongyi and Zhao, Junchuan and Lee, Francis Bu Sung and Yee, Andrew Zi Han},
booktitle = {ICASSP 2026 - 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
year = {2026},
pages = {17962--17966},
doi = {10.1109/icassp55912.2026.11460821},
url = {https://doi.org/10.1109/icassp55912.2026.11460821}
}CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidance
IEEE TASLP, 2026.
Paper
Demo
Cite
@article{zhao2026comelsinger,
title = {{CoMelSinger}: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidance},
author = {Zhao, Junchuan and Zeng, Wei and Lyu, Tianle and Wang, Ye},
journal = {IEEE Transactions on Audio, Speech and Language Processing},
year = {2026},
volume = {34},
pages = {1374--1387},
doi = {10.1109/taslpro.2026.3664643},
url = {https://doi.org/10.1109/taslpro.2026.3664643}
}Bridging Piano Transcription and Rendering via Disentangled Score Content and Style
ICLR, 2026.
Paper
Demo
Cite
@inproceedings{ICLR2026_97005e2b,
author = {Zeng, Wei and Zhao, Junchuan and Wang, Ye},
booktitle = {International Conference on Learning Representations},
editor = {C. Vondrick and B. Hariharan and C. Raffel and L. Pinto and D. Yang and A. Faust},
pages = {93485--93512},
title = {Bridging Piano Transcription and Rendering via Disentangled Score Content and Style},
url = {https://proceedings.iclr.cc/paper_files/paper/2026/file/97005e2b425c93015a91c312ba5c5298-Paper-Conference.pdf},
volume = {2026},
year = {2026}
}2025
Prosody-Adaptable Audio Codecs for Zero-Shot Voice Conversion via In-Context Learning
Interspeech, 2025.
Paper
Cite
@inproceedings{zhao2025pace,
title = {Prosody-Adaptable Audio Codecs for Zero-Shot Voice Conversion via In-Context Learning},
author = {Zhao, Junchuan and Wang, Xintong and Wang, Ye},
booktitle = {Interspeech 2025},
year = {2025},
pages = {4893--4897},
doi = {10.21437/interspeech.2025-464},
url = {https://doi.org/10.21437/interspeech.2025-464}
}SPSinger: Multi-Singer Singing Voice Synthesis with Short Reference Prompt
ICASSP, 2025.
Paper
Demo
Cite
@inproceedings{zhao2025spsinger,
title = {{SPSinger}: Multi-Singer Singing Voice Synthesis with Short Reference Prompt},
author = {Zhao, Junchuan and Low, Chetwin and Wang, Ye},
booktitle = {ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
year = {2025},
pages = {1--5},
doi = {10.1109/icassp49660.2025.10888907},
url = {https://doi.org/10.1109/icassp49660.2025.10888907}
}2024
SinTechSVS: A Singing Technique Controllable Singing Voice Synthesis System
IEEE/ACM TASLP, 2024.
Paper
Demo
Data
Cite
@article{zhao2024sintechsvs,
title = {{SinTechSVS}: A Singing Technique Controllable Singing Voice Synthesis System},
author = {Zhao, Junchuan and Chetwin, Low Qi Hong and Wang, Ye},
journal = {IEEE/ACM Transactions on Audio, Speech, and Language Processing},
year = {2024},
volume = {32},
pages = {2641--2653},
doi = {10.1109/taslp.2024.3394769},
url = {https://doi.org/10.1109/taslp.2024.3394769}
}No publications match this search.
05 / Experience
Academic Record
Teaching, academic service, honours, and contact links collected in one place.
Teaching
- Teaching Assistant, CS3244 Machine Learning, Sem2 AY2025/2026.
- Teaching Assistant, CS4347/CS5647 Sound and Music Computing, Sem1 AY2025/2026.
- Teaching Assistant, CS4347/CS5647 Sound and Music Computing, Sem1 AY2024/2025.
Service
- Reviewer for ToMM, TASLP, INTERSPEECH, ICASSP, ISMIR, ACM MM, AAAI, NeurIPS, EMNLP, and SLT.
- Event chair and performer at NUS Sound and Music Computing Concert 2025.
- Featured performer at NUS SMC Concert and NUS Jazz Band events.
Honours
- SoC Teaching Fellowship Scheme, 2025-2026.
- NUS Research Scholarship, 2023-2027.
- Outstanding College Student in Beijing, 2022.
- Queen Mary University of London Undergraduate College Prize, 2022.
Contact
For research discussions, collaboration, code, CV, and professional profiles.