From words to sound

Tango: language conditioning for ten-second audio

Tango separates language understanding from sound synthesis. A frozen 780M-parameter FLAN-T5-Large encodes the caption; an 866M-parameter, Stable-Diffusion-style U-Net predicts a latent mel spectrogram through cross-attention. A pretrained AudioLDM variational autoencoder and HiFi-GAN vocoder turn that latent representation into a 10-second, 16 kHz waveform. Only the denoiser is trained for Tango: the VAE and vocoder were already trained on broader audio collections.

The denoiser was trained for 40 epochs on 45,438 AudioCaps clips, with 2,240 clips for validation, using relative sound pressure when mixing training examples so that one source did not mask the other. The reported AudioCaps test uses one randomly selected caption per clip, 200 sampling steps and classifier-free guidance of 3. Under that setting, Tango records FD 24.52, KL 1.37 and FAD 1.59, all lower than the directly comparable AudioLDM-L model trained on AudioCaps for its denoiser: 27.12, 1.86 and 2.08. The boundary is also visible in the paper’s examples: prompts that differ only in a vegetable or surface can produce nearly identical audio. AudioCaps is small and narrow, and the “AudioCaps-only” comparison applies to the denoiser rather than to the pretrained audio codec and vocoder.

Tango 2: train on omissions and wrong event order

Tango 2 starts from the 866M-parameter Tango-full-FT checkpoint and constructs the Audio-alpaca preference set. The pipeline clusters AudioCaps captions and selects 14,000 prompts, then creates candidate outputs in four ways: different denoising budgets, repeated generations at the same budget, GPT-4 concept perturbations and GPT-4 temporal perturbations. Two CLAP checkpoints jointly filter the candidates. The final 15,025 winner–loser pairs comprise 3,004 different-step pairs, 2,725 same-step pairs, 4,544 concept-perturbation pairs and 4,752 temporal-perturbation pairs. Training uses one supervised epoch on the winners followed by four diffusion-DPO epochs; the paper reports about 3.5 hours on two A100 GPUs.

Evaluation generates one AudioCaps sample per prompt with 200 steps and guidance 3. Against Tango-full-FT, Tango 2 raises CLAP alignment from 0.54 to 0.57 and Inception Score from 7.87 to 9.09, and lowers KL from 1.15 to 1.12. FAD moves in the other direction, from 2.51 to 2.69, so the result is not a uniform gain in every quality metric. In a separate study of 50 AudioCaps prompts, with at least four ratings per sample on a five-point scale, overall quality rises from 3.81 to 3.99 and prompt relevance from 3.77 to 4.07. Audio-alpaca encodes judgments from CLAP and synthetic prompt perturbations rather than human preferences, and the listening study covers only 50 prompts.

Mustango: expose the musical variables

Mustango extends latent diffusion with MuNet, a 1.4B-parameter denoiser that attends in sequence to FLAN-T5 text representations, beat positions and chord sequences. Music-specific embeddings distinguish beats from downbeats and encode chord root, quality and time. Prompts can state beats, chords, tempo and key; when explicit controls are absent, a DeBERTa-Large beat predictor and a FLAN-T5-Large chord predictor infer them from the text. The system still uses the AudioLDM VAE and produces 10-second clips.

MusicBench contains 52,768 training instances derived from the 5,479 available MusicCaps clips. Beat, chord, key and tempo extractors annotate the audio, while tempo, pitch and volume transformations expand 3,413 filtered clips before captions are rephrased. Because some pretrained comparisons may already have seen MusicCaps, the paper also constructs FMACaps from 1,000 Free Music Archive clips. All reported model comparisons use 10-second outputs, 200 diffusion steps and guidance 3. On the hard TestB split, Mustango trained from scratch reaches 41.75% exact-key accuracy and 68.46% chord-root/major-minor match in any order, compared with 34.25% and 32.21% for the corresponding Tango model trained on MusicBench. Its chord score reaches 75.83% on FMACaps.

The second listening study uses 17 general listeners and four music experts, all rating on seven-point scales. Experts give Mustango 6.10 for chord match versus 2.99 for MusicGen-M, but MusicGen-M scores higher for rhythmic consistency, 5.61 versus 4.14. Mustango’s control vocabulary is based on Western key and chord conventions, the model is limited to ten seconds, and its conditions come from imperfect music-information-retrieval systems. Those constraints matter when extending the results to non-Western music, long-form structure or audio outside MusicBench.

TangoFlux: replace long diffusion sampling with rectified flow

TangoFlux uses a 515M-parameter transformer with six multimodal diffusion-transformer blocks followed by 18 diffusion-transformer blocks. Frozen FLAN-T5 representations and a duration embedding condition rectified-flow prediction in the latent space of Stable Audio Open’s VAE. Training uses approximately 400,000 WavCaps clips followed by 45,000 AudioCaps clips. The WavCaps stage runs for 80 epochs and the AudioCaps stage for 65 on five A40 GPUs. Clips are padded or centre-cropped to 30 seconds, and their mono channels are duplicated for compatibility with the 44.1 kHz stereo codec; the output is therefore pseudo-stereo rather than native stereo.

CLAP-Ranked Preference Optimization regenerates its evidence as the model changes. At each of five iterations, the method samples 20,000 prompts from an AudioCaps prompt bank, generates five clips per prompt and uses CLAP similarity to select one winner and one loser. Eight epochs of flow-based preference optimisation follow, with an additional winner reconstruction term to limit drift. Reusing a fixed pair set instead makes CLAP performance peak by the second iteration and then decline in the paper’s ablation.

The main automatic evaluation uses an 886-example AudioCaps split, 50 Euler steps and guidance 4.5. Variable-length models are conditioned to ten seconds and the first ten seconds are scored. TangoFlux records FDopenl3 75.1, KLpasst 1.15, CLAP 0.480 and Inception Score 12.2; Tango 2, evaluated with its recommended 200 steps, records 108.4, 1.11, 0.447 and 9.0. TangoFlux does not improve the KL result. In the paper’s separate same-A40 timing table, a 30-second TangoFlux output takes 3.7 seconds and a 10-second Tango 2 output takes 22.8 seconds, so those latency values do not use the same output duration. On 50 GPT-4o-generated, human-vetted prompts containing three to six events, with at least four annotators per clip, TangoFlux’s mean ranks are 1.7 for quality and 1.1 for relevance; Tango 2 scores 2.4 and 1.9. The speed is hardware-specific, the listening set is small and partly synthetic, and CLAP remains a proxy reward even though a different CLAP checkpoint is used for evaluation.

JAM: place words inside full songs

JAM scales the rectified-flow design to songs. Its 530M-parameter backbone has 16 LLaMA-style transformer layers and a frozen 44.1 kHz stereo codec assembled from the Stable Audio Open encoder and DiffRhythm decoder. The model conditions on lyrics, either a text or reference-audio style prompt, and a target duration. Word start and end times are converted to a latent-aligned phoneme sequence; separate filler tokens mark pauses within a word and regions in which no word is sung. A global duration embedding sets the requested length, while a learned padding signal suppresses content after the end. Pretraining uses random 90-second crops, and full-song supervised tuning extends the maximum duration to 230 seconds.

Training data comprise roughly one million English songs, or 54,000 hours. HTDemucs separates vocals and accompaniment, Parakeet supplies transcripts and word timestamps, and MuQ-MuLan encodes ten reference-style excerpts per song. The paper reports 750,000 pretraining steps, 250,000 supervised steps and three 20,000-step preference rounds on eight H100 GPUs. Each preference round samples 13,000–20,000 lyric-and-style conditions, generates five songs per condition and uses the average of five SongEval criteria to select a winner and loser. A reconstruction-anchored DPO variant is also tested because unregularised preference tuning can improve aesthetic scores while moving generated audio away from the reference distribution.

JAME evaluates 250 English songs collected from New Music Friday between 1 May and 10 July 2025, after the stated training cut-offs, across five genre groups. On this set, JAM reports word and phoneme error rates of 0.151 and 0.101; DiffRhythm, the next-best system on those two measures, records 0.348 and 0.264. The human study is much smaller: eight musically trained annotators rate outputs for ten lyrics, two from each genre, on five-point scales. JAM scores 3.75 for quality, 3.48 for enjoyment, 3.73 for musicality, 3.66 for voice naturalness and 3.70 for structural clarity. It leads this study on enjoyment, musicality and structure, but not on quality or voice naturalness.

JAM currently assumes accurate word-level timing at inference. Replacing reference timings with GPT-4o predictions raises phoneme error rate from 0.101 to 0.320 and produces audibly rigid vocals; quarter-beat quantisation limits the increase to 0.144 but still changes vocal character. It also lacks direct phoneme-duration control. The 250-song automatic benchmark and ten-lyric listening test are bounded evidence, and several reported aesthetic measures come from learned evaluators rather than listeners.