Google’s SoundStream codec concurrently suppresses noise and compresses audio

Google at this time time detailed SoundStream, an conclude-to-conclude “neural” audio codec that can present higher-quality audio while encoding diversified sound varieties, along side orderly speech, noisy and reverberant speech, tune, and environmental sounds. The company claims this is the indispensable AI-powered codec to work on speech and tune while being ready to urge in staunch time on a smartphone processor at the equivalent time.

Audio codecs compress audio to nick the need for excessive storage and bandwidth requirements. Ideally, the decoded audio will comprise to be perceptually indistinguishable from the normal and introduce minute latency. While most codecs leverage domain abilities and moderately engineered signal processing pipelines, there’s been ardour in changing handcrafted specs with AI that can be taught to encode on the cruise.

Earlier this yr, Google launched Lyra, a neural audio codec trained to compress low-bitrate speech. SoundStream extends this work with a gadget consisting of an encoder, decoder, and quantizer. The encoder converts audio correct into a coded signal that’s compressed the use of the quantizer and converted reduction to audio the use of the decoder. Once trained, the encoder and decoder can be urge on separate clients to transmit audio over the safe, and the decoder can operate at any bitrate.

Compressing audio

In former audio processing pipelines, compression and enhancement — i.e., the removing of background noise — are regularly carried out by diversified modules. But SoundStream is designed to manufacture compression and enhancement at the equivalent time. At 3kbps, SoundStream outperforms the usual Opus codec at 12kbps and approaches the usual of EVS at 9.6kbps while the use of 3.2-4 cases fewer bits, Google claims. Moreover, SoundStream performs better than the sizzling version of Lyra when put next at the equivalent bitrate.

Right here’s reference audio before processing with SoundStream:

And here’s the audio after processing:

Google cautions that SoundStream is serene within the experimental levels. On the other hand, the company plans to launch an updated version of Lyra that contains its parts to lift both higher audio quality and “reduced complexity.”

“Ambiance pleasant compression is indispensable every time one desires to transmit audio, whether when streaming a video or within the future of a convention name. SoundStream is a needed step toward bettering machine studying-pushed audio codecs. It outperforms inform-of-the-art codecs, equivalent to Opus and EVS, can enhance audio on ask, and requires deployment of most intriguing a single scalable mannequin, in arena of many,” Google compare scientist Neil Zeghidour and crew compare Marco Tagliasacchi wrote in a weblog post. “By integrating SoundStream with Lyra, developers can leverage the prevailing Lyra APIs and tools for his or her work, offering both flexibility and better sound quality.”