Adapting Offline Models to a Streaming Context for Music Source Separation

Dylan Sechet, Marc Evrard, Matthieu Kowalski

[Code]

Abstract. Real-time music source separation must satisfy two constraints: a bound on algorithmic latency and a bound on computational cost. Offline separators are usually left out of real-time comparisons, or credited with a latency equal to their full input length. We show that this latency is set by where the output is read, not by the length of the separator's input. An unmodified offline model can therefore run in a streaming setting, without retraining. At each step, the input slides by one STFT hop, and one output hop is read out. The resulting latency can be as low as one STFT hop (23 ms), and the computational cost does not increase as latency shrinks. We identify a theoretical model-dependent latency boundary under which separation quality should drop steeply, and confirm experimentally on 3 architectures. At equal algorithmic latency, streamed off-the-shelf checkpoints of HT-Demucs and SCNet match the published results of dedicated real-time models in separation quality. Streamed models remain far less computationally efficient: only HT-Demucs runs faster than real time on a GPU.

Audio samples

30s-long excerpts from the MUSDB18-HQ test set.

So Easy (Cristina Vane)
DrumsBassOtherVocals
Reference
HT-Demucs 23 ms k = -3
46 ms k = -2
93 ms k = 0
186 ms k = 4
3060 ms k = 128
Offline
SCNet 23 ms k = -3
46 ms k = -2
93 ms k = 0
186 ms k = 4
3060 ms k = 128
Offline
Catching Up (Lyndsey Ollard)
DrumsBassOtherVocals
Reference
HT-Demucs 23 ms k = -3
46 ms k = -2
93 ms k = 0
186 ms k = 4
3060 ms k = 128
Offline
SCNet 23 ms k = -3
46 ms k = -2
93 ms k = 0
186 ms k = 4
3060 ms k = 128
Offline
Knockout (M.E.R.C. Music)
DrumsBassOtherVocals
Reference
HT-Demucs 23 ms k = -3
46 ms k = -2
93 ms k = 0
186 ms k = 4
3060 ms k = 128
Offline
SCNet 23 ms k = -3
46 ms k = -2
93 ms k = 0
186 ms k = 4
3060 ms k = 128
Offline
Too Bright (Mu)
DrumsBassOtherVocals
Reference
HT-Demucs 23 ms k = -3
46 ms k = -2
93 ms k = 0
186 ms k = 4
3060 ms k = 128
Offline
SCNet 23 ms k = -3
46 ms k = -2
93 ms k = 0
186 ms k = 4
3060 ms k = 128
Offline
Mallory (The Mountaineering Club)
DrumsBassOtherVocals
Reference
HT-Demucs 23 ms k = -3
46 ms k = -2
93 ms k = 0
186 ms k = 4
3060 ms k = 128
Offline
SCNet 23 ms k = -3
46 ms k = -2
93 ms k = 0
186 ms k = 4
3060 ms k = 128
Offline