Reference: Y. Wang, A. Politis, K. Drossos, and T. Virtanen, "Moving Speaker Separation Via Parallel Spectral-Spatial Processing," IEEE Transactions on Audio, Speech and Language Processing, 2026
Abstract: Multi-channel speech separation in dynamic environments is challenging as time-varying spatial and spectral features evolve at different temporal scales. Existing methods typically employ sequential architectures, forcing a single network stream to simultaneously model both feature types, creating an inherent modeling conflict. In this paper, we propose a dual-branch parallel spectral-spatial (PS2) architecture that separately processes spectral and spatial features through parallel streams. The spectral branch uses a bi-directional long short-term memory (BLSTM)-based frequency module, a Mamba-based temporal module, and a self-attention module to model spectral features. The spatial branch employs bi-directional gated recurrent unit (BGRU) networks to process spatial features that encode the evolving geometric relationships between sources and microphones. Features from both branches are integrated through a cross-attention fusion mechanism that adaptively weights their contributions. Experimental results demonstrate that the PS2 outperforms existing state-of-the-art (SOTA) methods by 1.6-2.2 dB in scale-invariant signal-to-distortion ratio (SI-SDR) for moving speaker scenarios, with robust separation quality under different reverberation times (RT60), noise levels, and source movement speeds. Even with fast source movements, the proposed model maintains SI-SDR improvements of over 13 dB. These improvements are consistently observed across multiple datasets, including WHAMR! and our generated WSJ0-Demand-6ch-Move dataset.
AI-Generated Summary: This paper addresses the problem of separating individual speakers from a recording when the speakers are moving, a considerably harder problem than conventional speech separation because the acoustic relationship between each speaker and the microphones continuously changes. The key observation is that the sound characteristics of speech and the spatial information describing where a speaker is located change at different rates, while most existing systems try to process both together. The proposed PS2 system therefore uses two parallel processing paths: one focuses on the spectral content of speech and the other on spatial information, and an attention mechanism combines them according to their importance over time. To evaluate the approach, the paper also introduces a simulated multi-microphone dataset containing moving speakers, background noise, and room reverberation. PS2 outperforms strong existing speech-separation systems by 1.6–2.2 dB SI-SDR in moving-speaker scenarios, while also performing strongly for stationary speakers. Importantly, the advantage remains across different levels of reverberation and noise and becomes particularly clear when speakers move quickly or are spatially close together, showing that explicitly treating spectral and spatial information separately provides a more robust way of separating speech in dynamic, realistic environments.
