Speech and Singing Voice Generation
Speech synthesis, singing voice synthesis, voice conversion, singing technique control, and zero-shot vocal modeling.
01 / Profile
About
I study generative AI for voice, music, and human motion, with an emphasis on controllable systems that support expressive creation.
I work where sound, intelligence, and human expression meet.
I am a Ph.D. student at the School of Computing, National University of Singapore, advised by Prof. Ye Wang in the Sound and Music Computing Lab.
My research covers speech and singing voice synthesis, neural audio codecs, music generation, talking head generation, co-speech gesture generation, and affective multimodal learning.
I am especially interested in models and interfaces that make generated media easier to guide, evaluate, and use in creative or communicative settings.
Before and alongside my research, I have spent many years studying piano and singing. This musical background shapes the questions I care about: how generated sound can remain controllable, expressive, and useful for real creative workflows.
For selected performances and music projects, check my music page.
02 / Research directions
Research Focus
Three threads connect most of my work: expressive voice generation, music and audio intelligence, and multimodal human signal modeling.
Music and Audio Intelligence
Neural audio codecs, controllable audio generation, piano transcription/rendering, music style transfer, and creative ML systems.
Audio-Driven Human Generation
Talking head generation, co-speech gesture generation, multimodal emotion analysis, and human-centered generation.
03 / Updates
Recent News
A compact timeline for papers, internships, teaching, and milestones.
- One paper has been accepted by Interspeech 2026.
- I am joining Tencent as a Research Intern in Speech Synthesis in Singapore.
- One paper has been accepted by ACL 2026 Main Conference.
- One journal paper has been accepted by IEEE TASLP 2026.
- One paper has been accepted by ICLR 2026.
- Two papers have been accepted by ICASSP 2026.
- One paper has been accepted by Interspeech 2025.
- I have been selected for the SoC Teaching Fellowship Scheme.
- One paper has been accepted by ICASSP 2025.
- One journal paper has been accepted by IEEE/ACM TASLP 2024.
- I started my Ph.D. journey at NUS in the SMC Lab.
- I received my Master's degree from the National University of Singapore (NUS).
- I joined THUNLP Lab as a Research Intern.
- I joined the SMC Lab as a Student Researcher.
- I received my Bachelor's degree from Beijing University of Posts and Telecommunications (BUPT).
04 / Research output
Publications
Research across voice, music, and human-centered multimodal systems. Browse by year, filter by field, or search the collection.
* Equal contribution · † Corresponding author
2026
Edit Who Speaks, Control How They Speak: Global Timbre Editing and Local Instruction Control for TTS
arXiv preprint, 2026.
Paper
Demo
Overview
EDICT combines reference-based timbre editing with segment-specific expressive instructions, using an edited acoustic reference and KV-cache reconstruction to maintain voice consistency and acoustic continuity.
Cite
@misc{zhao2026editwhospeaks,
title = {Edit Who Speaks, Control How They Speak: Global Timbre Editing and Local Instruction Control for {TTS}},
author = {Junchuan Zhao and Chenglin Xu and Wei Zeng and Haoyang Li and Yiwen Guo and Ye Wang},
year = {2026},
eprint = {2610.11437},
archivePrefix = {arXiv},
primaryClass = {cs.SD},
doi = {10.48550/arXiv.2610.11437},
url = {https://arxiv.org/abs/2610.11437}
}Preference Optimization for Non-Verbal Vocalization Synthesis
arXiv preprint, 2026.
Paper
Overview
A study of preference signals, pair construction, and DPO objectives for non-verbal vocalization synthesis, introducing a non-verbal-aware character error rate to balance vocalization realization and lexical accuracy.
Cite
@misc{li2026preferenceoptimization,
title = {Preference Optimization for Non-Verbal Vocalization Synthesis},
author = {Haoyang Li and Chenglin Xu and Junchuan Zhao and Yuang Cao and Liumeng Xue and Yiwen Guo and Eng Siong Chng},
year = {2026},
eprint = {2608.24163},
archivePrefix = {arXiv},
primaryClass = {eess.AS},
doi = {10.48550/arXiv.2608.24163},
url = {https://arxiv.org/abs/2608.24163}
}AnyBand: Unified Multi-Bandwidth Speech Extension via Frequency-Aware In-Context Spectral Infilling
arXiv preprint, 2026.
Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems
arXiv preprint, 2026.
PersonaGest: Personalized Co-Speech Gesture Generation with Semantic-Guided Hierarchical Motion Representation
arXiv preprint, 2026.
MSpoofTTS: Multi-Resolution Spoof-Guided Inference for Discrete Speech Synthesis
Interspeech, 2026.
CodecFlow: Efficient Bandwidth Extension via Conditional Flow Matching in Neural Codec Latent Space
arXiv preprint, 2026.
TED-TTS: Training-Free Intra-Utterance Emotion and Duration Control for Text-to-Speech Synthesis
ACL Main, 2026.
KSDiff: Keyframe-Augmented Speech-Aware Dual-Path Diffusion for Facial Animation
ICASSP, 2026.
InconVAD: A Two-Stage Dual-Tower Framework for Multimodal Emotion Inconsistency Detection
ICASSP, 2026.
CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidance
IEEE TASLP, 2026.
Bridging Piano Transcription and Rendering via Disentangled Score Content and Style
ICLR, 2026.
2025
2024
SinTechSVS: A Singing Technique Controllable Singing Voice Synthesis System
IEEE/ACM TASLP, 2024.
No publications match this search.
05 / Experience
Academic Record
Teaching, academic service, honours, and contact links collected in one place.
Teaching
- Teaching Assistant, CS3244 Machine Learning, Sem2 AY2025/2026.
- Teaching Assistant, CS4347/CS5647 Sound and Music Computing, Sem1 AY2025/2026.
- Teaching Assistant, CS4347/CS5647 Sound and Music Computing, Sem1 AY2024/2025.
Service
- Reviewer for ToMM, TASLP, INTERSPEECH, ICASSP, ISMIR, ACM MM, AAAI, NeurIPS, EMNLP, and SLT.
- Event chair and performer at NUS Sound and Music Computing Concert 2025.
- Featured performer at NUS SMC Concert and NUS Jazz Band events.
Honours
- SoC Teaching Fellowship Scheme, 2025-2026.
- NUS Research Scholarship, 2023-2027.
- Outstanding College Student in Beijing, 2022.
- Queen Mary University of London Undergraduate College Prize, 2022.
Contact
For research discussions, collaboration, code, CV, and professional profiles.