Skip to content

Add AMD GPU secondary restoration (AMF Video SR + Real-ESRGAN) - #473

Open
lyx178238820-source wants to merge 54 commits into
Kruk2:mainfrom
lyx178238820-source:amd-secondary-upscale
Open

lyx178238820-source wants to merge 54 commits into
Kruk2:mainfrom
lyx178238820-source:amd-secondary-upscale

Conversation

@lyx178238820-source

@lyx178238820-source lyx178238820-source commented Oct 6, 2026 •

Copy link
Copy Markdown

Summary

Adds a secondary restoration backend for AMD GPUs, so --secondary-restoration
is no longer NVIDIA-only. Two engines are provided behind one option:

:: AMD's own Video SR (default) - AMF filter sr_amf, D3D11 on Windows
jasna --input in.mp4 --output out.mkv --secondary-restoration amd-upscale

:: Real-ESRGAN RRDBNet, in-process on the ROCm device
jasna --input in.mp4 --output out.mkv --secondary-restoration amd-upscale ^
      --amd-upscale-engine realesrgan --amd-upscale-scale 2

:: --amd-upscale-scale accepts 2 / 4 / 6 / 8; 6x and 8x act as supersampling
:: (the restored crop is blended back at the mosaic size)
jasna --input in.mp4 --output out.mkv --secondary-restoration amd-upscale --amd-upscale-scale 8

Motivation

--secondary-restoration offers unet-4x, tvai and rtx-super-res - all NVIDIA.
On an AMD GPU the stage simply could not be used (the factory raised for any
non-none value), so AMD users only got the primary BasicVSR++ pass.

Design

Both engines implement the existing SecondaryRestorer contract
(restore(batch) -> batch, 256x256 crop in, 256 * scale out, blended back by
the pipeline), so nothing outside session_factory needed to change.

amf-sr - AMD Video SR 1.0/1.1 via FFmpeg's AMF filter sr_amf:
format=nv12, hwupload=derive_device=amf, sr_amf=..., hwdownload. AMF is AMD's
media framework and initialises over D3D11 on Windows (not Vulkan/ROCm);
--amd-upscale-algorithm selects sr1-0/sr1-1 (ML) or the non-ML fallbacks
bicubic/bilinear/point, --amd-upscale-sharpness maps to the filter's
sharpness. One FFmpeg process per clip, with stdin closed as soon as the
frames are written: the AMF filters buffer the tail of the stream and only
release it when the input ends, so a long-lived pipe stalls one frame short of
every clip (closing stdin per clip flushes deterministically and subprocess
gives a hard timeout for free).

realesrgan - Real-ESRGAN RRDBNet in-process on the ROCm device
(prefers_cpu_input = False, tensors never leave VRAM - the same strategy as
the RTX Super Res backend). The architecture is inferred from the checkpoint
(num_block/num_feat/pixel_unshuffle), so x4plus, x2plus and the 6-block
anime weights all load; a network stronger than the requested factor is
area-downsampled. Weights are auto-detected in model_weights/ or given with
--amd-upscale-model-path.

Plumbing

  • new CLI group --amd-upscale-{engine,scale,algorithm,sharpness,model-path,ffmpeg-path,timeout}
  • SessionConfig fields + AmdUpscaleEngineName literal
  • session_factory: AMD branch, AMD-only guard relaxed
    (amd-upscale requires an AMD device; other backends still require NVIDIA)
  • GUI: engine/scale/algorithm/sharpness/timeout rows, single-choice entry, presets
  • locales: en / zh / ja / ko / th
  • docs: docs/zh/models.md, docs/zh/cli.md
  • tests: tests/test_amd_upscale_secondary_restorer.py (FFmpeg command + filter
    chain, mocked), tests/test_realesrgan_restorer.py (CPU, tiny checkpoints),
    plus GUI-settings coverage
  • benchmark record: benchmarks/2026-10-06_amd_secondary_upscale.{md,csv}

Testing

  • pytest on this branch: 76 passed, 2 skipped (the new backend tests are pure
    CPU / mocked FFmpeg, so they run without a GPU)
  • End-to-end on Windows 11 + RX 7900 XT (torch 2.9 ROCm, FFmpeg 8.1.2 with AMF),
    15 s 1080p clip, YOLO detection, 4 restored clips: both engines exit 0 and
    produce the expected output.

Benchmarks (RX 7900 XT, 256x256 crops)

engine scale output throughput notes
amf-sr 2x 512x512 ~84 fps
amf-sr 4x 1024x1024 ~55-60 fps no extra install, uses AMD's own model
amf-sr 6x 1536x1536 ~31 fps still super-resolved, acts as supersampling
amf-sr 8x 2048x2048 ~17-18 fps ditto
realesrgan 4x 1024x1024 ~16.5 fps fp16, batch 2 (batch 4 hits a pathological MIOpen kernel: 0.24 fps)

sr_amf takes a free output size, so 6x/8x needed proof that the engine does not
silently fall back to plain scaling. Comparing sr_amf against algorithm=bicubic at
the same output size (one testsrc2 256x256 frame): PSNR 36.8 / 38.0 / 38.6 / 38.8 dB
at 2/4/6/8x, with 1.6-3.1x the Laplacian variance of bicubic, and 8x SR differs from
"4x SR + plain resize" by 47.1 dB - the engine keeps doing its own work at the higher
factors. Because the blend step downsamples the restored crop back to the mosaic size,
the higher factors behave as supersampling (cleaner edges, less aliasing) rather than
as more output resolution, at a proportional time cost.

Notes / limitations

  • amf-sr needs an FFmpeg build with the AMF filters (sr_amf); the Windows
    Gyan "full" build has it. Error messages point this out.
  • realesrgan needs a Real-ESRGAN checkpoint in model_weights/ (not bundled);
    the error text says exactly where to put it.
  • Verified on RDNA3 (gfx110x). RDNA2 needs a ROCm build that ships gfx103x
    kernels (ROCm 7.14 / TheRock multi-arch wheels) - a wheel matter, not code.

Kruk2 added 30 commits September 28, 2026 13:12
… OSes

ROCm 10.0.0 (stable, 2026-08-26) publishes torch 2.12.0 and torchvision
0.27.0 cp312 wheels for Windows and Linux on
https://stable.repo.amd.com/rocm/whl-next/ with per-GPU kernel packs for
gfx1100-gfx1103, gfx1150/gfx1151 and gfx1200/gfx1201, so the Radeon 780M
(gfx1103) gets kernels alongside every target the ROCm 7.2.1 build had.

Both OSes now share one torch/torchvision pin, the Windows setup installs
the same kernel packs as the Linux image keeps, and the README names the
Adrenalin 26.8.1 driver ROCm 10 needs on Windows. The protection bump
moves the Linux image to ROCm 10 and bundles the ROCm payload on Linux.

Refs Kruk2#408
Bundle application and dependency license texts, record exact source and model correspondence, and include the legal payload in frozen builds.\n\nDocument the private protection source as the sole reproducibility exception.
18.1.0 contains the current-CUDA-context and explicit NVENC stream
changes the release builds compiled from a pinned PyAV commit. Pin it
exactly because the VALI release wheel links the same pyav-ffmpeg
8.1.2-1 libraries. Drop cython/delvewheel from the dev extra; only the
PyAV rebuild used them.
Patch releases within the 2.12 line that torch-tensorrt 2.12.0 accepts;
the dev environment already runs them.
Also records the protection commit that fixes the release scripts' modes.
Show where supporter keys are sold in the system-check wizard and in the
license dialog, and give translated messages for keys that are not
accepted.
Newer versions load a new checkpoint file name, so older versions keep
downloading the file they expect. The download skips the old file so
users fetch only the checkpoint this version needs.
Keep downloaded models usable after upgrading.
Explain activation failures and update supporter documentation.
…el-name ltx)

Inference-only restorer for the fine-tuned LTX-2.5 model, run in phased passes
with one model on the GPU at a time: scan (detect, cuts, plan) -> VAE encode
-> denoise -> decode + composite + encode.

- jasna/ltx: track planning with one zooming follow camera per track and
  121-frame windows at hop 88; video-only transformer port with prequantized
  INT8 ConvRot linears; lockstep sampler with overlap fusion and shared-prefix
  STG; block streaming from pinned RAM for low-VRAM GPUs; GPU crop/composite.
- jasna/models/ltx_vae: vendored pure-PyTorch subset of the ltx-core video VAE
  (causal encoder, single-step diffusion decoder, tiled decode).
- CLI: --restoration-model-name ltx, --ltx-large-canvas, --ltx-seed;
  --restoration-model-path now defaults per model.
- Licensing: LTX-2.x Community License text and notices.
Use shared model loading for all LTX components and report missing
access before engine setup. Clarify model access after app updates.
--ltx-fast uses an NVFP4 model on RTX 50-series GPUs. SageAttention, torch.compile and NATTEN are used when installed.
Sigma schedule and STG scale come from the file's metadata; STG 0 skips the perturbed pass.
…e warning

Progress advances after every sampler step, and the Hadamard matrix is built directly so Dynamo no longer warns.
Standard and LTX cards plus seed, fast mode and sharper large mosaics options. LTX is disabled without an NVIDIA GPU or its model files.
Segments without one follow the job's model, and overlapping ranges are split by the later one. Each LTX segment is planned from its own frames and all LTX spans run in one batch.
Each LTX pass reports its share of the run and the time left; the control bar shows the pass name in place of the FPS.
Kruk2 and others added 24 commits September 29, 2026 16:38
New ranges take the settings panel's model and seed, the selected range is edited in place, and "Use for all ranges" copies it. LTX ranges are teal on the timeline.
restore_spans runs prepare_spans, denoise_spans and compose_spans over one LtxRender, so prepared references can be denoised with another seed.
Try seed restores the selected range in the background like a job and keeps the prepared latents, so another seed reruns only denoise and decode. Use this seed applies a tried seed.
Inductor picked kernel configs by timing them, so the output changed between runs. The block forward now compiles in deterministic mode.
Start, End and the add button share a row when there is room, and the range settings scroll on their own.
An LTX frame counts as about 30 standard frames, and time left comes from the whole job.
cv2.fillPoly with several polygons uses the even-odd rule, so overlapping regions were left out of the blend mask. Fill each polygon separately.
blending.dilate_ellipse matches cv2.dilate with MORPH_ELLIPSE pixel for pixel and replaces the LTX tracing, LTX feather and SD 1.5 copies. LTX tracing runs about 2x faster.
… models

--ltx-model picks the model; each variant has its own file name. A missing model is offered for download with its size (y/N in the CLI); the download location is not set yet.
A Model menu picks distilled or undistilled. A missing model is offered for download with its size and blocks Start until done. One download worker now serves SD 1.5 and LTX.
…ense

LtxModelFiles.placeholder() serves seeded random tensors with the real names, shapes and dtypes (jasna/ltx/trial_shapes.py), so every pass runs as in a real job. The output is not restored.
A "Trial run" switch runs LTX with placeholder weights to test speed. It is forced on when no LTX model is installed and offered at Start when a license is missing.
…enable Flash Attention and Memory Efficient Attention
Enable LTX restoration for ROCM
…SRGAN)

Secondary restoration was NVIDIA-only (unet-4x / TVAI / RTX Super Res), so on an
AMD GPU the stage was unusable. This adds the AMD counterpart, reusing the
existing SecondaryRestorer contract:

* `--secondary-restoration amd-upscale --amd-upscale-engine amf-sr` (default)
  AMD Video SR 1.0/1.1 through FFmpeg's AMF filter `sr_amf`. AMF initialises over
  D3D11 on Windows. One FFmpeg process per clip: the AMF filters buffer the tail
  of the stream and only release it on EOF, so a long-lived pipe stalls one frame
  short of every clip.

* `--secondary-restoration amd-upscale --amd-upscale-engine realesrgan`
  Real-ESRGAN RRDBNet executed in-process on the ROCm device - no subprocess and
  the crops never leave VRAM (`prefers_cpu_input = False`, same strategy as the
  RTX backend). The network shape is inferred from the checkpoint (num_block /
  num_feat / pixel-unshuffle), so x4plus, x2plus and the 6-block anime weights all
  work; a stronger network than the requested factor is area-downsampled.

Plumbing: `--amd-upscale-*` CLI group, SessionConfig fields, the AMD-only guard
relaxed in `session_factory`, GUI section + 5 locales, zh docs, unit tests for
both backends (CPU only, mocked FFmpeg) and a benchmark record.

Measured on an RX 7900 XT (256x256 crops): AMF 4x ~59 fps, 2x ~92 fps,
Real-ESRGAN 4x ~16.5 fps. End to end on a 15 s 1080p clip with 4 restored clips:
secondary ~2.9 s (AMF) / ~8.4 s (Real-ESRGAN).
`sr_amf` takes a free output size and keeps running its SR engine at the higher
factors, so the 2x/4x restriction was arbitrary. Verified that it is real super
resolution and not a silent fallback to plain scaling by comparing against
`algorithm=bicubic` at the same output size (one testsrc2 256x256 frame):

  scale   PSNR(sr vs bicubic)   Laplacian var (sr / bicubic)
    2x            36.78 dB            245.6 / 156.8   (1.57x)
    4x            38.01 dB             57.0 /  19.6   (2.91x)
    6x            38.55 dB             21.5 /   9.6   (2.23x)
    8x            38.81 dB             12.8 /   4.1   (3.11x)

8x SR also differs from "4x SR + plain resize" (47.1 dB apart, sharpness 12.8 vs
6.9), so it is not chaining a resize. Absolute sharpness drops with the factor
simply because the same content is spread over more pixels; and since the blend
step downsamples the restored crop back to the mosaic size, the higher factors act
as supersampling (cleaner edges, less aliasing) rather than as more output
resolution.

Cost, measured in-process on an RX 7900 XT (30-frame clips, 256x256 crops):
~84 fps at 2x, ~55 at 4x, ~31 at 6x, ~18 at 8x. End to end on a 15 s 1080p clip
the wall clock only grows from ~21 s to 24 s (6x) and 29 s (8x), because the
secondary stage runs concurrently with detection.

Also fixes the stale `rocm` engine name in the zh docs (the engine is called
`realesrgan`) and records the 6x/8x numbers in the benchmark document.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants