Repository navigation
Add AMD GPU secondary restoration (AMF Video SR + Real-ESRGAN) - #473
Open
lyx178238820-source wants to merge 54 commits into
Open
lyx178238820-source wants to merge 54 commits into
lyx178238820-source wants to merge 54 commits into
Conversation
added 30 commits
September 28, 2026 13:12
… OSes ROCm 10.0.0 (stable, 2026-08-26) publishes torch 2.12.0 and torchvision 0.27.0 cp312 wheels for Windows and Linux on https://stable.repo.amd.com/rocm/whl-next/ with per-GPU kernel packs for gfx1100-gfx1103, gfx1150/gfx1151 and gfx1200/gfx1201, so the Radeon 780M (gfx1103) gets kernels alongside every target the ROCm 7.2.1 build had. Both OSes now share one torch/torchvision pin, the Windows setup installs the same kernel packs as the Linux image keeps, and the README names the Adrenalin 26.8.1 driver ROCm 10 needs on Windows. The protection bump moves the Linux image to ROCm 10 and bundles the ROCm payload on Linux. Refs Kruk2#408
Bundle application and dependency license texts, record exact source and model correspondence, and include the legal payload in frozen builds.\n\nDocument the private protection source as the sole reproducibility exception.
18.1.0 contains the current-CUDA-context and explicit NVENC stream changes the release builds compiled from a pinned PyAV commit. Pin it exactly because the VALI release wheel links the same pyav-ffmpeg 8.1.2-1 libraries. Drop cython/delvewheel from the dev extra; only the PyAV rebuild used them.
Patch releases within the 2.12 line that torch-tensorrt 2.12.0 accepts; the dev environment already runs them.
Also records the protection commit that fixes the release scripts' modes.
Show where supporter keys are sold in the system-check wizard and in the license dialog, and give translated messages for keys that are not accepted.
Newer versions load a new checkpoint file name, so older versions keep downloading the file they expect. The download skips the old file so users fetch only the checkpoint this version needs.
Keep downloaded models usable after upgrading.
Explain activation failures and update supporter documentation.
…el-name ltx) Inference-only restorer for the fine-tuned LTX-2.5 model, run in phased passes with one model on the GPU at a time: scan (detect, cuts, plan) -> VAE encode -> denoise -> decode + composite + encode. - jasna/ltx: track planning with one zooming follow camera per track and 121-frame windows at hop 88; video-only transformer port with prequantized INT8 ConvRot linears; lockstep sampler with overlap fusion and shared-prefix STG; block streaming from pinned RAM for low-VRAM GPUs; GPU crop/composite. - jasna/models/ltx_vae: vendored pure-PyTorch subset of the ltx-core video VAE (causal encoder, single-step diffusion decoder, tiled decode). - CLI: --restoration-model-name ltx, --ltx-large-canvas, --ltx-seed; --restoration-model-path now defaults per model. - Licensing: LTX-2.x Community License text and notices.
Use shared model loading for all LTX components and report missing access before engine setup. Clarify model access after app updates.
--ltx-fast uses an NVFP4 model on RTX 50-series GPUs. SageAttention, torch.compile and NATTEN are used when installed.
Sigma schedule and STG scale come from the file's metadata; STG 0 skips the perturbed pass.
…e warning Progress advances after every sampler step, and the Hadamard matrix is built directly so Dynamo no longer warns.
Standard and LTX cards plus seed, fast mode and sharper large mosaics options. LTX is disabled without an NVIDIA GPU or its model files.
Segments without one follow the job's model, and overlapping ranges are split by the later one. Each LTX segment is planned from its own frames and all LTX spans run in one batch.
Each LTX pass reports its share of the run and the time left; the control bar shows the pass name in place of the FPS.
New ranges take the settings panel's model and seed, the selected range is edited in place, and "Use for all ranges" copies it. LTX ranges are teal on the timeline.
restore_spans runs prepare_spans, denoise_spans and compose_spans over one LtxRender, so prepared references can be denoised with another seed.
Try seed restores the selected range in the background like a job and keeps the prepared latents, so another seed reruns only denoise and decode. Use this seed applies a tried seed.
Inductor picked kernel configs by timing them, so the output changed between runs. The block forward now compiles in deterministic mode.
Start, End and the add button share a row when there is room, and the range settings scroll on their own.
An LTX frame counts as about 30 standard frames, and time left comes from the whole job.
cv2.fillPoly with several polygons uses the even-odd rule, so overlapping regions were left out of the blend mask. Fill each polygon separately.
blending.dilate_ellipse matches cv2.dilate with MORPH_ELLIPSE pixel for pixel and replaces the LTX tracing, LTX feather and SD 1.5 copies. LTX tracing runs about 2x faster.
… models --ltx-model picks the model; each variant has its own file name. A missing model is offered for download with its size (y/N in the CLI); the download location is not set yet.
A Model menu picks distilled or undistilled. A missing model is offered for download with its size and blocks Start until done. One download worker now serves SD 1.5 and LTX.
…ense LtxModelFiles.placeholder() serves seeded random tensors with the real names, shapes and dtypes (jasna/ltx/trial_shapes.py), so every pass runs as in a real job. The output is not restored.
A "Trial run" switch runs LTX with placeholder weights to test speed. It is forced on when no LTX model is installed and offered at Start when a license is missing.
…enable Flash Attention and Memory Efficient Attention
Enable LTX restoration for ROCM
…SRGAN) Secondary restoration was NVIDIA-only (unet-4x / TVAI / RTX Super Res), so on an AMD GPU the stage was unusable. This adds the AMD counterpart, reusing the existing SecondaryRestorer contract: * `--secondary-restoration amd-upscale --amd-upscale-engine amf-sr` (default) AMD Video SR 1.0/1.1 through FFmpeg's AMF filter `sr_amf`. AMF initialises over D3D11 on Windows. One FFmpeg process per clip: the AMF filters buffer the tail of the stream and only release it on EOF, so a long-lived pipe stalls one frame short of every clip. * `--secondary-restoration amd-upscale --amd-upscale-engine realesrgan` Real-ESRGAN RRDBNet executed in-process on the ROCm device - no subprocess and the crops never leave VRAM (`prefers_cpu_input = False`, same strategy as the RTX backend). The network shape is inferred from the checkpoint (num_block / num_feat / pixel-unshuffle), so x4plus, x2plus and the 6-block anime weights all work; a stronger network than the requested factor is area-downsampled. Plumbing: `--amd-upscale-*` CLI group, SessionConfig fields, the AMD-only guard relaxed in `session_factory`, GUI section + 5 locales, zh docs, unit tests for both backends (CPU only, mocked FFmpeg) and a benchmark record. Measured on an RX 7900 XT (256x256 crops): AMF 4x ~59 fps, 2x ~92 fps, Real-ESRGAN 4x ~16.5 fps. End to end on a 15 s 1080p clip with 4 restored clips: secondary ~2.9 s (AMF) / ~8.4 s (Real-ESRGAN).
`sr_amf` takes a free output size and keeps running its SR engine at the higher
factors, so the 2x/4x restriction was arbitrary. Verified that it is real super
resolution and not a silent fallback to plain scaling by comparing against
`algorithm=bicubic` at the same output size (one testsrc2 256x256 frame):
scale PSNR(sr vs bicubic) Laplacian var (sr / bicubic)
2x 36.78 dB 245.6 / 156.8 (1.57x)
4x 38.01 dB 57.0 / 19.6 (2.91x)
6x 38.55 dB 21.5 / 9.6 (2.23x)
8x 38.81 dB 12.8 / 4.1 (3.11x)
8x SR also differs from "4x SR + plain resize" (47.1 dB apart, sharpness 12.8 vs
6.9), so it is not chaining a resize. Absolute sharpness drops with the factor
simply because the same content is spread over more pixels; and since the blend
step downsamples the restored crop back to the mosaic size, the higher factors act
as supersampling (cleaner edges, less aliasing) rather than as more output
resolution.
Cost, measured in-process on an RX 7900 XT (30-frame clips, 256x256 crops):
~84 fps at 2x, ~55 at 4x, ~31 at 6x, ~18 at 8x. End to end on a 15 s 1080p clip
the wall clock only grows from ~21 s to 24 s (6x) and 29 s (8x), because the
secondary stage runs concurrently with detection.
Also fixes the stale `rocm` engine name in the zh docs (the engine is called
`realesrgan`) and records the 6x/8x numbers in the benchmark document.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a secondary restoration backend for AMD GPUs, so
--secondary-restorationis no longer NVIDIA-only. Two engines are provided behind one option:
Motivation
--secondary-restorationoffers unet-4x, tvai and rtx-super-res - all NVIDIA.On an AMD GPU the stage simply could not be used (the factory raised for any
non-
nonevalue), so AMD users only got the primary BasicVSR++ pass.Design
Both engines implement the existing
SecondaryRestorercontract(
restore(batch) -> batch, 256x256 crop in,256 * scaleout, blended back bythe pipeline), so nothing outside
session_factoryneeded to change.amf-sr- AMD Video SR 1.0/1.1 via FFmpeg's AMF filtersr_amf:format=nv12, hwupload=derive_device=amf, sr_amf=..., hwdownload. AMF is AMD'smedia framework and initialises over D3D11 on Windows (not Vulkan/ROCm);
--amd-upscale-algorithmselectssr1-0/sr1-1(ML) or the non-ML fallbacksbicubic/bilinear/point,--amd-upscale-sharpnessmaps to the filter'ssharpness. One FFmpeg process per clip, with stdin closed as soon as the
frames are written: the AMF filters buffer the tail of the stream and only
release it when the input ends, so a long-lived pipe stalls one frame short of
every clip (closing stdin per clip flushes deterministically and
subprocessgives a hard timeout for free).
realesrgan- Real-ESRGAN RRDBNet in-process on the ROCm device(
prefers_cpu_input = False, tensors never leave VRAM - the same strategy asthe RTX Super Res backend). The architecture is inferred from the checkpoint
(
num_block/num_feat/pixel_unshuffle), sox4plus,x2plusand the 6-blockanime weights all load; a network stronger than the requested factor is
area-downsampled. Weights are auto-detected in
model_weights/or given with--amd-upscale-model-path.Plumbing
--amd-upscale-{engine,scale,algorithm,sharpness,model-path,ffmpeg-path,timeout}SessionConfigfields +AmdUpscaleEngineNameliteralsession_factory: AMD branch, AMD-only guard relaxed(
amd-upscalerequires an AMD device; other backends still require NVIDIA)docs/zh/models.md,docs/zh/cli.mdtests/test_amd_upscale_secondary_restorer.py(FFmpeg command + filterchain, mocked),
tests/test_realesrgan_restorer.py(CPU, tiny checkpoints),plus GUI-settings coverage
benchmarks/2026-10-06_amd_secondary_upscale.{md,csv}Testing
pyteston this branch: 76 passed, 2 skipped (the new backend tests are pureCPU / mocked FFmpeg, so they run without a GPU)
15 s 1080p clip, YOLO detection, 4 restored clips: both engines exit 0 and
produce the expected output.
Benchmarks (RX 7900 XT, 256x256 crops)
amf-sramf-sramf-sramf-srrealesrgansr_amftakes a free output size, so 6x/8x needed proof that the engine does notsilently fall back to plain scaling. Comparing
sr_amfagainstalgorithm=bicubicatthe same output size (one
testsrc2256x256 frame): PSNR 36.8 / 38.0 / 38.6 / 38.8 dBat 2/4/6/8x, with 1.6-3.1x the Laplacian variance of bicubic, and 8x SR differs from
"4x SR + plain resize" by 47.1 dB - the engine keeps doing its own work at the higher
factors. Because the blend step downsamples the restored crop back to the mosaic size,
the higher factors behave as supersampling (cleaner edges, less aliasing) rather than
as more output resolution, at a proportional time cost.
Notes / limitations
amf-srneeds an FFmpeg build with the AMF filters (sr_amf); the WindowsGyan "full" build has it. Error messages point this out.
realesrganneeds a Real-ESRGAN checkpoint inmodel_weights/(not bundled);the error text says exactly where to put it.
kernels (ROCm 7.14 / TheRock multi-arch wheels) - a wheel matter, not code.