Y. Wang, A. Politis, K. Drossos, and T. Virtanen, "Dynamic Clustering for Cross-Segment Permutation Alignment in Long Speech Separation," in Proceedings of the IEEE International Workshop on Machine Learning for Signal Processing (MLSP), Atlanta, U.S.A., 2026
Long speech separation typically employs a segment-separation-stitch paradigm where recordings are divided into short segments, processed independently, and stitched together. Its challenge lies in predicting cross-segment permutations. This paper proposes a training-free dynamic clustering approach for cross-segment permutation alignment using speaker embedding reference pools. The method predicts the permutation using the cosine similarity between current segment embeddings and the reference pools. The approach updates reference pools by retaining the most representative speaker embeddings based on their overall cosine similarity with existing references. As a plug-and-play post-processing module compatible with existing separation models, the proposed method demonstrates superior performance compared to existing methods on dense and sparse long speech scenarios, particularly in challenging sparse scenarios with extended utterance gaps, and further shows robustness to speaker count estimation errors in unknown speaker count scenarios.
This paper addresses a practical problem in long speech separation: when a long recording is split into shorter segments and each segment is separated independently, the same speaker may appear in different output channels from one segment to the next. This makes it difficult to reconstruct continuous, speaker-consistent audio, especially in meetings, interviews, or other recordings where speakers may remain silent for long periods. The proposed method solves this by maintaining a small, continuously updated reference pool of speaker embeddings for each output stream. For every new segment, the system compares its speaker embeddings with these reference pools and assigns the segment to the most likely speakers, while retaining only the most representative and reliable embeddings over time. The approach requires no additional training and can be added as a post-processing stage to existing speech-separation systems. Experiments on long recordings with both dense and sparse speech show that the method substantially improves cross-segment speaker consistency, reaching 98.5% permutation accuracy in dense scenarios and 96.8% in sparse ones, with separation quality close to the oracle upper bound. It is particularly effective when there are long gaps between utterances, where conventional overlap- or clustering-based methods degrade, and it remains robust even when the number of speakers is estimated incorrectly.
