by Ashesh DasOptimizing the audio pipeline: DeepFilterNet and room scaling to 100+ users
How we rebuilt the ModVoice audio stack. We replaced older filters with DeepFilterNet 3 running on WebAssembly, cached neural network assets, and changed signaling to support 100 users in a single room.
This last week we rewrote ModVoice’s audio stack. If you’ve ever used an in-browser voice service, you’re familiar with that familiar crackle of ambient room noise in the background, or that gate which cuts off the first few letters of each of your words. We wanted to see what it would take to create a close replica of a studio setup, in your browser.
Here is what we implemented, the compromises we made, and the workarounds that let it scale.
DeepFilterNet 3 in the Browser
Before, our background noise filtering came from RNNoise.RNNoise is light-weight and inexpensive, but not adept at dealing with fast, percussive noises, like your colleague clattering at a mechanical keyboard, or a distant ambulance wailing past. RNNoise also has a tendency to mute you as the room gets noisier.
Now, we utilize a deep-learning-based noise reduction tool called DeepFilterNet 3, which separates voice from any ambient sound. To utilize the power of this neural network inside a browser tab we compile the model’s C code into a WebAssembly module and run it in an AudioWorkletNode. Running audio processing within an AudioWorkletNode keeps expensive, demanding operations off of the main JavaScript thread and prevents your video feed from glitching as your browser renders new frame.
The only obvious problem was size. Both the WASM module and the model weights were around 10 MB. You would have been waiting a few seconds for the page to become responsive until you’d successfully click the join button to engage your mic.
We upgraded rooms to handle 100 users
Although the voice data in ModVoice is routed using WebRTC, the control signals telling people what state their peers are in live on a signaling server. Previously, the signaling server had to push out to every user in the room an updated representation of the entire room when anything changed. If someone mute their mic in a 100-user room, the signaling server sends a list of all 100 people to everyone else. All those messages eat up massive queuing buffers on the signaling server and client, slow the WebSocket connections down, and bog the UI.
We adjusted the signaling protocol so that it only sends delta updates - just what changed - as opposed to a whole snapshot. When a user mutes their mic or leaves, just a little, lean packet with the specific change information is sent to the client, who then updates its view. The 100-person rooms now transmit 90 percent less traffic on the WebSocket connection, and active speakers can be visually highlighted on the sidebar without dropped frames.