Skip to content

ace-step: gallery entry for acestep-v15-xl-base + expose audio-conditioned task types (cover/repaint/lego) in /v1/sound-generation and Studio Sound UI #12721

Open

Description

@Pinucci0

Is your feature request related to a problem?

The ace-step backend supports only text-to-music via the /v1/sound-generation endpoint and the Studio "Sound" page. The upstream ACE-Step 1.5 model family — and the Python backend implementation in
backend/python/ace-step/backend.py — already implements the full task matrix (text2music, cover, repaint, extract, lego, complete) and accepts audio-conditioned inputs, but none of it is reachable from the HTTP API or
the React UI.

Describe the solution

Four related improvements:

1. Gallery entry for acestep-v15-xl-base (4B DiT)

ace-step-turbo is already in the model gallery; please add the XL base variant (ACE-Step/acestep-v15-xl-base), the highest-quality DiT of the family (~18.8 GB bf16, MIT license). Manual setup works today but requires hand-placing weights under models/<name>/checkpoints/<name>/.

2. LM variant selection in the gallery entry

The backend supports lm_model_path with acestep-5Hz-lm-0.6B, acestep-5Hz-lm-1.7B and acestep-5Hz-lm-4B. The gallery entry (or two variants of it) should let the user pick which LM is downloaded and
configured, since the 4B planner meaningfully improves CoT metadata/caption quality.

3. Audio Reference inputs on /v1/sound-generation + Studio Sound UI

The backend already maps request payload keys into GenerationParams, including task_type, reference_audio_path, src_audio, audio_cover_strength and repainting_start/repainting_end. What is
missing is the plumbing above it:

  • a way to upload/pass a source audio file in /v1/sound-generation (e.g. multipart form or a file-path field);
  • UI fields on the Studio Sound page for:
    • reference_audio — global timbre/style reference (cover, style transfer);
    • src_audio — source audio for non-text2music tasks;
    • audio_codes — semantic codes for cover-mode reuse/extensions;
    • repainting_start / repainting_end — interval control for repaint and lego tasks.

4. task_type selector with task-specific controls

Expose task_type in the API and add a selector in Studio Sound with the ACE-Step 1.5 task types: text2music, cover, repaint, lego, extract, complete — enabling the relevant inputs per task (lyrics and audio refs for cover, intervals for repaint/lego, etc.).

Additional context

  • Backend capability already exists: backend/python/ace-step/backend.py forwards model options into the generation payload, and GenerationParams accepts the fields above — this feature is endpoint + UI plumbing, not new inference work.
  • The audio.cpp backend already has a source_audio input path for editing routes (build_sound_generation_request), so this would align the Python ace-step backend with a pattern the codebase partially supports.

Environment

  • LocalAI v4.11.0 (binary), cpu-ace-step backend
  • ACE-Step 1.5 XL base + acestep-5Hz-lm-4B working via manual YAML
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions