Introduction

Every digital experience we consume today – whether it be a Zoom call, streaming a show on Netflix, or using a Virtual Reality (VR) headset, one piece of technology is quietly doing the heavy lifting: the codec. Codecs short for coder–decoders are the invisible engines that shrink massive audio and video files into streams light enough to travel over the internet, yet smart enough to reconstruct them fast with acceptable quality. Without codecs, even a short voice message would eat hundreds of megabytes and still buffer endlessly to load. Codecs are the backbone of real-time communication, powering every single bit of data of multimedia that travels across the web.

Traditional codecs like Advanced Audio Coding (AAC), Opus are reliable and are fine-tuned over the years, but they are also hitting their limits. On the other hand, Neural codecs that are powered by Artificial Intelligence (AI), deliver a stunning quality, but they are computationally heavy to run and tough to deploy. That’s where hybrid codecs step in: they become a bridge between both worlds. They bring together the proven stability and efficiency of traditional codecs, and the smart, adaptive intelligence of AI.

Codec Development Over Time

Over the past few decades, audio codecs have evolved from simple waveform storage to advanced AI-driven systems. Each era represents a major leap toward efficiency and realistic representation of sound transmission and reception over the web.

Figure 1: Evolution of Audio Codecs over time

• Pulse Code Modulation (PCM) was one of the earliest digital audio formats, storing raw data samples without compression. It was most accurate but required a high bandwidth.
• MP3/AAC format brought in the era of digital music. It used a psychoacoustic model to remove inaudible sounds, reducing the file size but having good quality.
• Opus was introduced as a blend of two codecs, SILK (for speech) and CELT (for music). With its adaptability across bandwidths and low bitrates, Opus has become a dominant codec behind platforms like Zoom, Whatsapp, and Discord.

Since the 2020 pandemic, reliance on virtual meetings has increased significantly. Online communication increased significantly, and this puts strain on the network.

As communication sessions become larger and wide-spread, networks and servers bear the weight. Cutting even a fraction of the bitrate eases that burden and reduces costs. At the same time, internet connectivity continues to differ across users and regions. Next generation audio codecs face the challenge of maintaining audio quality irrespective of the connection speed.

The key challenges for next-generation codecs are:

• Bandwidth Efficiency: Fit more audio information into fewer bits, reducing the bandwidth required for transmission.
• Low Latency: Minimize the delay in audio transmission so that speech and calls feel instantaneous and natural in real-time communication.
• Perceptual Quality: Preserve natural-sounding voices, maintain the depth and richness of music, and keep videos clear and sharp.

Together, these requirements create a challenge for next-generation codec development: to balance high audio quality, low latency, and efficient bandwidth usage.

Understanding Hybrid Codecs and Machine Learning

Hybrid codecs are a new class of AI-powered audio codecs that use Machine Learning (ML) to achieve high-quality audio at extremely low bitrates. They use an audio compression system that extracts key features from an audio signal using traditional techniques and then combines these features with generative AI to synthesize natural-sounding audio. Instead of simply decompressing audio, it generates it from the received data. This enables more efficient compression, since the model needs only minimal information to recreate the full sound. Hybrid codecs are highly effective in low-bandwidth scenarios.

How they work

Hybrid codecs usually follow a layered design in operations. At each stage, they can leverage either traditional signal processing or use a neural network-based method. Corresponding stages are:

• Pre-processing:

Before the signal is compressed, audio is often cleaned through denoising, filtering, and level normalization. Traditional codecs use fixed algorithms, but in hybrid codecs, a neural front-end can use an ML model to perform this task. This model is trained using huge volume of data to take action intelligently on the received input. It can adapt its ways according to the input received, thus giving a more stable and cleaner signal as an input to the main encoder.

• Mid layer:

Traditional codecs use conventional techniques like Modified Discreet Cosine Transform (MDCT), ideal for general sound by analyzing and simplifying its frequency content or Linear Predictive Coding (LPC). LPC is optimized for speech by modelling the core mechanism of human speech production. However, these methods often lose high-frequency details and introduce quantization noise. A neural network captures this missing “residual” information and fills the gaps. For example, Google’s Sound Stream uses Residual Vector Quantization (RVQ) to encode the difference between the original and compressed audio using iterative neural code blocks. The mid layer is the core of hybrid codecs.

• Post-filtering:

At the decoder stage, when the compressed data is received, the traditional codec adds a post-filter to identify and remove unwanted content to make the output smoother. Hybrid codecs replace this with a neural enhancer that acts as an audio super-resolution filter. It is a deep-learning technique that uses decoded low-resolution signal and side information to reconstruct lost high-frequency components, extend bandwidth, and produce a high-fidelity signal. This generative post-processing is one of the key reasons why these hybrid codecs can achieve such high quality even at very low bitrates.

Figure 2: Hybrid Codec Architecture (Conceptual)

What Modules Can Be Substituted as Neural?

Almost every component in the codec pipeline can be substituted by an AI/ML candidate. When all components are replaced, it is called a “Neural” codec. The beauty of the “hybrid” approach is that we can selectively replace specific modules to strike the correct balance between maximum efficiency and audio quality.

The modules most frequently replaced across hybrid codecs include:

• Feature extraction:
  Traditional codecs rely on fixed psychoacoustic models to pick features like pitch and format. This step can be replaced with a neural network that is trained to learn the most perceptually relevant features for a given audio signal. It is usually a more compact and expressive representation..
• Quantization:
  Instead of using traditional scalar or vector quantization, we can use neural vector quantization techniques like RVQ (used in Google Lyra). These techniques are more effective in preserving the fine spectral details, thereby maintaining audio quality even at low bitrates while minimizing the quantization errors that are introduced.
• Packet Loss Concealment (PLC):
  When a packet is lost due to network conditions, traditional codecs often repeat a previous frame or add silence. A neural PLC like Microsoft Satin’s Packet Loss Resilience can intelligently generate a realistic and natural sounding fill for the missing audio. This makes the overall audio more stable for network instability and interruptions.
• Decoder/Vocoder:
  Among neural elements in hybrid codecs, this is the most used and is typically located on the decoder side. It is a lightweight generative model that uses compressed features to rebuild the complete audio waveform. Google’s Lyra, for instance, uses a lightweight neural vocoder to generate high-quality speech. This represents a shift from traditional reconstruction to data-driven synthesis, where the neural model functions as a “mini synthesizer” trained on real speech samples.

Comparison between Traditional, Hybrid, Neural Codecs: Audio Quality

Table 1: Codec comparison chartLinking the

Hybrid Codecs in Practice

The term “hybrid” is broad, encompassing models that either combine a traditional frontend with a neural back end, or integrate neural quantization within a conventional coding framework. Essentially, they are required to carefully create a balance between audio quality, latency, computational cost, power consumption, and operational readiness.

A few implementation challenges include:

• Computational Cost: Real-time neural decoding is computationally intensive, requiring architectural optimization to meet latency and power budgets.
• Power, Audio quality trade-off: AI improves audio quality at low bitrates, but balancing the increased power demands with battery life requires efficient power management and hardware acceleration.
• Interoperability: Hybrid codecs must maintain compatibility with established standards, supporting seamless fallback and reliable negotiation to prevent service interruptions.

Given the above challenges (Fig. 4), standardization bodies are now actively evaluating how to integrate these high-performance, yet complex, AI components into existing standards. Hybrid codecs serve as a “pragmatic bridge”, offering an immediate, deployable solution that delivers clear, high-quality audio over the most bandwidth-constrained networks today.

Real World Hybrid Audio Codecs

Rather than replacing decades of codec technology, hybrid codecs work alongside it. They use traditional models where the signal is predictable and stable and apply AI only where traditional models begin to struggle, especially at very low bitrates or under poor network conditions.

1. LPCNet (Mozilla)
   LPCNet is often considered the blueprint for hybrid neural vocoders.
   • Bitrate: ~1.6 kbps
   • Latency: ~20 ms frame size (configuration dependent)
   • Core idea: Uses LPC for predicting the next speech sample as a linear combination of previous samples, with Gated Recurrent Unit (GRU), for predicting residual difference between the actual speech and the LPC prediction (excitation signal).
   • Deployment: Reference design for ultra-low-bitrate speech, widely cited in embedded, research, and experimental low-bandwidth communication systems.
2. Lyra V2 (Google)
   Lyra V2 represents a shift toward generative speech compression.
   • Bitrate: ~3.2 kbps, 6 kbps, 9.2 kbps
   • Latency: ~20–40 ms
   • Core idea: Instead of transmitting waveform samples, Lyra extracts compact speech features—such as pitch and spectral envelope—using a traditional front-end. A neural generative decoder then reconstructs the waveform at the receiver.
   • Deployment: Integrated into Google Meet and Google Duo (now part of Meet). It automatically kicks in when your signal connection drops to critical levels.
3. SATIN (Microsoft)
   Microsoft’s SATIN employs a band-splitting strategy.
   • Bitrate: 6 kbps to 36 kbps
   • Core idea: Uses a hybrid AI-powered architecture that combines advanced speech modeling, psychoacoustic signal processing, and deep neural networks. It transmits a compact representation of speech and uses a neural decoder to reconstruct missing high-frequency details, delivering high-quality audio. It incorporates strong packet-loss resilience at low bitrates.
   • Deployment: Integrated into Microsoft Teams and Skype for real-time voice communications.

Future Direction & Conclusion

Hybrid codecs are more than just a transitional technology. By blending the reliability of traditional Digital Signal Processing (DSP) with the perceptual power of AI; they address today’s challenges scalable streaming, real-time calls, and immersive AR/VR while paving the way for a fully neural future. Soon, hybrid codecs will be part of everyday life: our video calls will sound clearer; music will stream at lower bitrates, and AR experiences will deliver richer 3D soundscapes all powered by a seamless combination of classic DSP and intelligent AI.

eInfochips has a deep expertise in audio codecs, guiding you through every step from selecting the right codec for your application to integrating and optimizing it for maximum efficiency, quality, and compatibility ensuring your audio solutions deliver the best possible performance across devices and networks.

Frequently Asked Questions

1. How is a hybrid codec different from a fully neural codec?

A hybrid codec selectively replaces specific codec components with AI models, while a fully neural codec replaces most or all stages of the encoding and decoding pipeline with neural networks. This allows hybrid codecs to strike a balance between audio quality, computational cost, latency, and deployment readiness.

2. Why are hybrid codecs gaining attention now?

The rise of video conferencing, remote collaboration, cloud gaming, and AR/VR applications has increased the demand for high-quality audio over bandwidth-constrained networks. Hybrid codecs help deliver better audio quality at lower bitrates without requiring the heavy computational resources of fully neural solutions.

3. How do hybrid codecs handle packet loss better than traditional codecs?

Traditional codecs often compensate for lost packets by repeating previous audio frames or inserting silence. Hybrid codecs can use AI-based Packet Loss Concealment (PLC) models to generate realistic replacement audio, resulting in a more natural listening experience during network interruptions.

4. Are hybrid codecs suitable for real-time communication?

Yes. Hybrid codecs are specifically designed to balance audio quality and computational complexity, making them suitable for real-time applications such as video conferencing, voice calls, online gaming, and collaborative platforms.

5. How do hybrid codecs help reduce bandwidth consumption? 

Rather than transmitting the complete audio waveform, hybrid codecs send compact audio features and use AI models to reconstruct missing details at the receiver. This significantly reduces bitrate requirements while maintaining perceptually high audio quality.