MiniMax-Music3: port the text-to-music lane (Qwen3 global LLM + 8-codebook RVQ + flow-matching DiT + DAC Flow-VAE) — our first music-generating model

[localai-bot]

What

MiniMaxAI/MiniMax-Music3 —
lyrics + a structured music description in, up to five minutes of stereo song
out. Complete support: every component, both quantized and bf16/fp32 arms, and
the e2e pipeline through the shared seams.

It is absent from the pinned vLLM (no source files; the registry carries only
MiniMax M2/M3 text models), so this is the first row to exercise the fallback
oracle rule added in #647.

Component map (measured, not inferred)

Geometry read from config.json per component and from the safetensors headers
by HTTP range request:

Component	Class	Geometry	Params	dtype on disk
language_model	Qwen3ForCausalLM (transformers)	36L, hidden 4096, 32 heads / 8 KV, head_dim 128, ffn 12288, vocab 200000, rope_theta 1e6, max_pos 10240	~8.6B	BF16
condition_encoder	MiniMaxMusic3ConditionEncoder	4 tensors — layer_scale, layer_weight_logits, proj; a learned weighted mix over num_condition_layers: 8 LLM layers, not an encoder tower	0.025B	F32
rvq_depth_decoder	MiniMaxMusic3RVQDepthDecoder	4L, hidden 4096, 16 heads, ffn 6144, num_codebooks: 8, audio_vocab_size: 1024, max_pos 16	0.646B	BF16
transformer	MiniMaxMusic3Transformer1DModel	36L, 32 heads x head_dim 64 (hidden 2048), ff_inner 8192, in_channels 128, condition_dim 2048, fourier 256, rotary_dim 32	2.4B	F32
scheduler	FlowMatchEulerDiscreteScheduler	invert_sigmas: true, num_train_timesteps: 1, shift 1.0, exponential	—	—
vocoder	MiniMaxMusic3Vocoder	DAC-style, latent 128, upsampling_ratios [8,8,4,2], decoder hidden 1536 / in 1024, weight_g/weight_v weight-norm	0.054B	F32

The language_model is our existing Qwen3ForCausalLM architecture exactly,
retrained on a 200k music vocab — a config value, not an architecture change.

Two packagings, one set of weights

The repo ships both, which is why it is 57.4 GB:

• native arm — qwen_7B/qwen_7B/ (AbabForCausalLM, model_type: mixtral,
  auto_map to remote modeling_abab.py), flowmatching_vae.pth (DiT +
  condition projection), dav.pth (DAC Flow-VAE decoder). SGLang-Omni serves
  this one (sglang_omni/models/minimax_music3/checkpoint.py), pulling the RVQ
  decoder out of the Qwen shards as model.audio_decoder.*.
• diffusers arm — the six modular components above, safetensors only.

scripts/convert_minimax_music3_to_diffusers.py in the diffusers PR shows the
conversion is a pure re-layout: load the three native files, rename tensors,
save. Same weights. So the two oracles are comparable, and the diffusers arm is
the port target: ~28.5 GB resident, no pickle, no trust_remote_code, and
every component has a class to gate against.

Oracles (per #647)

• primary: diffusers — PR huggingface/diffusers#14456
  (Add MiniMax Music 3), OPEN, head c6da9936e4bda83107943a16eb8682e9a37d8527,
  branch huggingface:minimax-music3-integration. An unmerged PR branch can be
  rebased, so the pin is the exact head SHA.
• cross-check: SGLang-Omni — 748a0b437e4a8faad44d7bbfd5a0ae55d1fef830,
  for e2e and the speed axis. Owes its own oracle record.
• transformers for the Qwen3ForCausalLM half.

Open question, to be resolved from source and not guessed

Output sample rate. The model card and SGLang-Omni's README both say
32 kHz stereo. Every config says 44100 — vocoder/config.json
sampling_rate: 44100 with upsampling_ratios [8,8,4,2] (hop 512), and
condition_encoder output_sampling_rate: 44100. W0 resolves this against the
vocoder module and the pipeline's decoders.py before any waveform golden is
captured.

Phases

W0 spec + stand up the diffusers oracle (closes part of #647) · W1 modular
loader, weight-norm folding, dtype policy · W2 global LLM on our Qwen3 path ·
W3 condition mixer + RVQ depth decoder · W4 flow-matching DiT + scheduler ·
W5 vocoder decode + WAV · W6 e2e through the shared seams, include/vllm.h,
SGLang-Omni cross-check and speed · W7 quantized arms, unimplemented arms
refused by name.

Gate shape follows MiniMax-H3: the diffusion half has no logits and so gets no
token-exact gate — per-stage tensor parity against the oracle at fixed seed and
reduced dimensions — while the LLM half can be token-exact.