Instance-Specific Test-Time Training for Speech Editing in the Wild
Taewoo Kim1, Uijong Lee1, Hayoung Park1, Choongsang Cho1, Nam In Park2, Young Han Lee1
1Korea Electronics Technology Institute, South Korea
2National Forensic Service, South Korea
Abstract
Speech editing systems aim to naturally modify speech content while preserving acoustic consistency and speaker identity. However, previous studies often struggle to adapt to unseen and diverse acoustic conditions, resulting in degraded editing performance in real-world scenarios. To address this, we propose an instance-specific test-time training method for speech editing in the wild. Our approach employs direct supervision from ground-truth acoustic features in unedited regions, and indirect supervision in edited regions via auxiliary losses based on duration constraints and phoneme prediction. This strategy mitigates the bandwidth discontinuity problem in speech editing, ensuring smooth acoustic transitions between unedited and edited regions. Additionally, it enables precise control over speech rate by adapting the model to target durations via mask length adjustment during test-time training. Experiments on in-the-wild benchmark datasets demonstrate that our method outperforms existing speech editing systems in both objective and subjective evaluations.
This page is for research demonstration purposes only.
Model Overview
Fig 1. Overview of Proposed Speech Editing System
Speech Editing Samples (Training on LibriTTS Clean, Testing on GigaSpeech)
* VoiceCraft: pretrained on GigaSpeech XL
Sample 1: Number seven coffee contains essential nutrients the body loves and needs. one cup of coffee contains nutrients like riboflavin potassium magnesium and others.
Ground Truth
FluentSpeech
VoiceCraft
Proposed
w/o TTT for SD
w/o TTT for DP
w/o TTT for Both
Sample 2: He says while having your hand in every pot might not be the most efficient way to run a business it did mean that the changs knew what their brand was all about.
Ground Truth
FluentSpeech
VoiceCraft
Proposed
w/o TTT for SD
w/o TTT for DP
w/o TTT for Both
Sample 3: Now you're using triangles and squares.
Ground Truth
FluentSpeech
VoiceCraft
Proposed
w/o TTT for SD
w/o TTT for DP
w/o TTT for Both
Sample 4: In part. One of the things that I've I've spoken about openly is the fact that I was in a ah you know a convent school.
Ground Truth
FluentSpeech
VoiceCraft
Proposed
w/o TTT for SD
w/o TTT for DP
w/o TTT for Both
Sample 5: Open the box. Place the sensors. plug it in. And your home is protected around the clock.
