smol-audio: A Colab-Friendly Notebook Collection for Fine-Tuning Advanced Audio Models

smol-audio: A Colab-Friendly Notebook Collection for Fine-Tuning Whisper, Parakeet, Voxtral, Granite Speech, and Audio Flamingo 3

The Deep-unlearning team has released smol-audio, an Apache-2.0 licensed repository of self-contained Jupyter notebooks for audio AI tasks. Every recipe is designed to run in a standard 16 GB Google Colab runtime, removing the barrier of local GPU infrastructure.

Why This Matters

While audio AI has advanced with models like Whisper and Audio Flamingo 3, practical implementation knowledge remains fragmented across research blogs and private repositories. Engineers often face significant hurdles when adapting these models to specific domains, frequently starting from scratch due to a lack of transparent, reproducible workflows. smol-audio addresses this by exposing full training loops and data pipelines within the Hugging Face ecosystem, including transformers, datasets, peft, and accelerate. This transparency allows engineers to modify configurations without reverse-engineering a hidden framework, reducing the time from experimentation to production.

Key Insights

• LoRA (Low-Rank Adaptation) is integrated for NVIDIA’s Audio Flamingo 3 to reduce GPU memory requirements by an order of magnitude compared to full fine-tuning.
• Mistral’s Voxtral, built on Ministral 3B and Mistral Small 3.1 24B, requires prompt masking during ASR fine-tuning to avoid computing loss on prompt tokens.
• NVIDIA’s Parakeet utilizes a CTC (Connectionist Temporal Classification) architecture, which requires alignment between audio frames and output tokens rather than autoregressive decoding.
• Meta’s Perception Encoder Audiovisual (PE-AV) enables zero-shot video classification by learning a shared embedding space across audio, video, and text modalities.
• The repository uses a ‘flat repo’ design to ensure every step of the training loop and data pipeline is visible and modifiable by the user.

Practical Applications

• Accessibility and Indexing: Fine-tuning Audio Flamingo 3 for audio captioning to generate natural language descriptions of sound clips. Pitfall: Attempting full fine-tuning on large multimodal models without LoRA often leads to OOM errors on commodity hardware.
• Multilingual ASR Deployment: Adapting IBM’s Granite Speech for specific languages like Italian using the YODAS-Granary dataset. Pitfall: Neglecting architecture-specific handling, such as prompt masking in LLM-based speech models, results in degraded training dynamics.
• Voice Agent Synthesis: Using Nari Labs’ Dia-1.6B for generating natural multi-speaker dialogue exchanges. Pitfall: Over-simplifying multi-speaker TTS as single-speaker synthesis leads to loss of conversational nuance.

References:

• https://www.marktechpost.com/2026/04/29/smol-audio-a-colab-friendly-notebook-collection-for-fine-tuning-whisper-parakeet-voxtral-granite-speech-and-audio-flamingo-3/