Stable Audio Open 1.0

## Overview

Stable Audio Open 1.0 is an open-weight generative AI model developed by Stability AI for text-to-audio synthesis. Designed as a research foundation, the model enables researchers and artists to fine-tune, experiment, and advance text-guided audio generation. Stable Audio Open 1.0 serves as an accessible baseline for the development of new generative audio models, emphasizing transparency by releasing weights, training code, and detailed evaluation metrics. The model and its capabilities are documented in the research publication available on [arXiv:2407.14358](https://arxiv.org/abs/2407.14358).

Model Architecture

Stable Audio Open 1.0 is based on a latent diffusion architecture with integrated transformer components, similar to its predecessor Stable Audio 2.0, but notably adopts T5-based text conditioning instead of CLAP-based conditioning. The model has approximately 1.21 billion parameters and integrates three central modules: a variational autoencoder, a T5 text encoder, and a transformer diffusion model, as documented in the research publication.

The autoencoder compresses stereo waveforms sampled at 44.1 kHz into a latent representation with a size of 64, utilizing five convolutional blocks for downsampling and expansion. Reconstruction loss is calculated using perceptually weighted, multi-resolution STFT, balancing mid-side and left-right stereo presentation. The text encoder, leveraging the T5-base architecture, processes natural language prompts to condition audio output. The diffusion transformer (DiT) operates in the autoencoder’s latent space through stacked attention blocks and gated multilayer perceptrons with rotary positional embeddings, supporting variable-length and timing-conditioned generation. Conditioning on both timing and text is introduced via cross-attention mechanisms.

Technical enhancements such as block-wise attention and gradient checkpointing are implemented to manage computational and memory demands. These choices enable the model to efficiently handle variable-length generation—up to 47 seconds of stereo audio—by appending silence to shorter outputs, which can be post-processed by users (details available on the Hugging Face model page).

Training Data and Methodology

Training of Stable Audio Open 1.0 was exclusively conducted on audio licensed under Creative Commons, prioritizing scientific transparency and avoiding proprietary datasets, as detailed in the research paper. The core dataset comprises 486,492 audio recordings, totaling approximately 7,300 hours. Of these, 472,618 samples are sourced from the Freesound database, while 13,874 are drawn from the Free Music Archive (FMA). Comprehensive filtering was undertaken to eliminate copyrighted material using tools such as the PANNs music classifier and Audible Magic’s identification, alongside metadata cross-references with the Spotify tracks dataset, with manual review for flagged items.

Training text prompts were constructed from descriptive metadata. For Freesound audio, prompts utilized natural language descriptions, titles, and tags. For FMA, metadata such as year, genre, album, artist, and title contributed to prompt generation, with further random transformations for diversity.

The variational autoencoder was trained on 5-second snippets of high-fidelity audio, including 48kHz and 44.1kHz files. Training employed AdamW optimization, with distinct batch sizes for encoder and decoder, and learning rates adapted for each module. The DiT was trained on 1,024 latent tokens—a sequence corresponding to nearly 47 seconds of audio. Regularization strategies and exponential learning-rate scheduling were incorporated to improve training stability and performance, as outlined in arXiv:2407.14358.

Performance and Evaluation

Model performance was evaluated using multiple metrics—FD_openl3, KL_passt, and CLAP score—addressing both fidelity and prompt relevance. On the AudioCaps dataset for general sounds and field recordings, Stable Audio Open achieved a FD_openl3 of 78.24 (lower is better), KL_passt of 2.14 (lower is better), and CLAP score of 0.29 (higher is better). These results indicate strong performance for realistic sound and field recording synthesis, surpassing contemporary open models such as AudioLDM2 and AudioGen.

On the Song Describer dataset, which evaluates instrumental music generation, the model performed modestly compared to proprietary and previous Stable Audio models (FD_openl3: 96.51, KL_passt: 0.55, CLAP: 0.41) but slightly exceeded MusicGen, the leading open alternative at that time.

Autoencoder reconstruction was assessed using STFT distance, MEL distance, and SI-SDR. Results showed parity with Stable Audio 2.0 on general sounds and slightly lower music reconstruction quality, attributable to the model’s exclusive use of Creative Commons data. Memorization analyses found no evidence of unauthorized reproduction of training data.

In terms of efficiency, inference speed for diffusion on common hardware ranged from 8 to 20 steps per second, depending on GPU memory. Decoding from latent space to waveform is memory-intensive, but chunked decoding offers substantial optimization (see arXiv:2407.14358).

Use Cases and Model Limitations

The open availability and reproducibility of Stable Audio Open 1.0 make it applicable for AI research in audio synthesis, academic studies on generative models, and technical exploration by practitioners interested in fine-tuning or benchmarking generative audio capabilities (further details on the Hugging Face model page). The model demonstrates particular strength for high-quality, text-guided sound effect and field recording generation.

Notable limitations remain. Stable Audio Open 1.0 does not generate realistic vocals or intelligible speech, and performance is primarily tuned to English-language prompts due to the composition of the training metadata. Music generation capabilities are less robust than those of certain proprietary or non-CC-trained models, reflecting the dataset’s constraints. Audio quality and stylistic breadth may also be uneven for some musical genres or cultural forms. For complex prompts, especially those using conjunctions, careful prompt design may be required to achieve optimal outcomes.

Comparison to Other Models

Stable Audio Open 1.0 represents a research branch diverging from previous Stable Audio models in both dataset curation and the conditional text encoder. While Stable Audio 1.0 and 2.0 support longer audio generation and were trained using broader data that includes non-CC content, Stable Audio Open 1.0 focuses strictly on Creative Commons sources and T5-based conditioning for open research utility. In side-by-side evaluations, Stable Audio Open 1.0 generally outperformed open baselines such as AudioLDM2 and AudioGen for non-musical sound synthesis and achieved comparable or slightly superior results to MusicGen on instrumental music, albeit with lower performance than non-open Stable Audio models for music fidelity.

Licensing and Access

Stable Audio Open 1.0 is released under the Stability AI Community License. All training data are drawn from audio under CC0, CC BY, or CC Sampling+ licenses, with full attribution provided (Attributions). For further details on licensing and permitted uses, consult the commercial license terms and privacy policy.

Helpful Resources

• Stable Audio Open 1.0 – Hugging Face Model Card
• Stable Audio Open 1.0 – Research Paper (arXiv:2407.14358)
• Stable Audio Open Demo
• stabilityai/stable-audio-tools – Source Code
• Stability-AI/stable-audio-metrics – Benchmarking Tools
• Stability AI Attributions Page
• Hugging Face Documentation: diffusers
• T5-base Model Card