You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ace-step: gallery entry for acestep-v15-xl-base + expose audio-conditioned task types (cover/repaint/lego) in /v1/sound-generation and Studio Sound UI #12721
The ace-step backend supports only text-to-music via the /v1/sound-generation endpoint and the Studio "Sound" page. The upstream ACE-Step 1.5 model family — and the Python backend implementation in backend/python/ace-step/backend.py — already implements the full task matrix (text2music, cover, repaint, extract, lego, complete) and accepts audio-conditioned inputs, but none of it is reachable from the HTTP API or
the React UI.
Describe the solution
Four related improvements:
1. Gallery entry for acestep-v15-xl-base (4B DiT)
ace-step-turbo is already in the model gallery; please add the XL base variant (ACE-Step/acestep-v15-xl-base), the highest-quality DiT of the family (~18.8 GB bf16, MIT license). Manual setup works today but requires hand-placing weights under models/<name>/checkpoints/<name>/.
2. LM variant selection in the gallery entry
The backend supports lm_model_path with acestep-5Hz-lm-0.6B, acestep-5Hz-lm-1.7B and acestep-5Hz-lm-4B. The gallery entry (or two variants of it) should let the user pick which LM is downloaded and
configured, since the 4B planner meaningfully improves CoT metadata/caption quality.
3. Audio Reference inputs on /v1/sound-generation + Studio Sound UI
The backend already maps request payload keys into GenerationParams, including task_type, reference_audio_path, src_audio, audio_cover_strength and repainting_start/repainting_end. What is
missing is the plumbing above it:
a way to upload/pass a source audio file in /v1/sound-generation (e.g. multipart form or a file-path field);
UI fields on the Studio Sound page for:
reference_audio — global timbre/style reference (cover, style transfer);
src_audio — source audio for non-text2music tasks;
audio_codes — semantic codes for cover-mode reuse/extensions;
repainting_start / repainting_end — interval control for repaint and lego tasks.
4. task_type selector with task-specific controls
Expose task_type in the API and add a selector in Studio Sound with the ACE-Step 1.5 task types: text2music, cover, repaint, lego, extract, complete — enabling the relevant inputs per task (lyrics and audio refs for cover, intervals for repaint/lego, etc.).
Additional context
Backend capability already exists: backend/python/ace-step/backend.py forwards model options into the generation payload, and GenerationParams accepts the fields above — this feature is endpoint + UI plumbing, not new inference work.
The audio.cpp backend already has a source_audio input path for editing routes (build_sound_generation_request), so this would align the Python ace-step backend with a pattern the codebase partially supports.
Environment
LocalAI v4.11.0 (binary), cpu-ace-step backend
ACE-Step 1.5 XL base + acestep-5Hz-lm-4B working via manual YAML
Activity