Model
Kien T. Pham1, Yuhao Liu2, Chong Hou Choi2, Zhenqi He1, Yanghao Wang1, Chenyang Lei2, Rui Liu2, Qifeng Chen1, Long Chen1
1The Hong Kong University of Science and Technology 2Huawei Research
tkpham@connect.ust.hk · longchen@ust.hk
UniAVC is a unified model that natively handles text-conditioned joint audio-video generation, bidirectional cross-modal synthesis, and instruction-guided editing of either or both modalities without relying on task-specific adapters. To achieve this versatility, the model utilizes a progressive training paradigm that first masters unimodal generation and editing capabilities before dedicating scarce paired datasets entirely to establishing cross-modal correspondence and temporal synchronization. The framework further introduces an asymmetric noise-level scheduling technique that derives decoupled per-branch noise trajectories and explicitly supervises extreme boundary states to enable intent-aware editing and post-inference self-refinement. Extensive empirical evaluations demonstrate that this unified approach delivers performance comparable to specialized, state-of-the-art baselines across a wide range of generation and editing benchmarks.
One sample per editing family, applying the same instruction with UniAVC and with InstructAV2AV and MiniMax-H3.
Transform the man into a cyberpunk painting style with high-saturation neon colors and sharp digital illustration strokes, while keeping all characters, actions, and composition unchanged.
Add a person wearing white clothes, standing still in the center of the frame, looking at the camera with a blank expression.
Remove the woman wearing a white lab coat, a stethoscope, and looking confused or distressed from the source video, while preserving the background of a blurred office environment, a desk lamp, the wall calendar, and the shelving behind her.
Alter the character's dialogue to: "We were in the drum circle like, wow, this is where real musical magic happens right now!" while keeping visual elements and background sounds identical to the original scene.
Replace the background with a private study at late night, with quiet city night lights visible outside the window, while keeping the person's image, clothing, actions, and contemplative head-down expression in the foreground completely consistent.
Each grid holds the source clip and the results of three different instructions applied to that same clip.
Restyle the subject while preserving background, lighting and posture.
Insert a new person into the shot.
Erase a person and reconstruct what was behind them. This family ships two examples with one instruction each.
Replace a line of dialogue with new words and matched lip-sync.
Relocate the background scene, keeping subject appearance and expression intact.
conda create -n uniavc python=3.10 -y
conda activate uniavc
# Install a torch build matching your CUDA, then the rest.
pip install torch==2.8.0 torchvision==0.23.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128
pip install -r requirements.txt
pip install -e .
Tested environment: Python 3.10, CUDA 12.8, PyTorch 2.8, on a single 80 GB GPU.
bash scripts/download_weights.sh fetches the weights, plus Gemma-3 for text
conditioning:
bash scripts/download_weights.sh
Or by hand:
pip install -U huggingface_hub
# The released model.
hf download TK3105/UniAVC model.safetensors \
--local-dir models/UniAVC
# Gemma-3 text encoder.
hf download google/gemma-3-12b-it-qat-q4_0-unquantized \
--local-dir models/gemma-3-12b-it-qat-q4_0-unquantized
Once downloaded, the weights are stored as follows:
models/
├── UniAVC/
│ └── model.safetensors # the DiT + the VAEs, vocoder, post-modules
└── gemma-3-12b-it-qat-q4_0-unquantized/
Edit the video and its soundtrack from a single instruction. Each of the five families gets one clip and one instruction:
# Subject Editing
python inference/infer_tav2av.py \
--video samples/sample_1.mp4 \
--prompt "Replace the man's dark brown T-shirt with a blue cycling jersey with white stripes on the sleeves. Keep the pull-up bar, the beige wall and the lighting unchanged." \
-o sample_1_out.mp4
# Subject Addition
python inference/infer_tav2av.py \
--video samples/sample_2.mp4 \
--prompt "Include a man in a grey coat seated in the front seat, saying: \"We should have left an hour ago. Nobody checked the traffic or the weather.\"" \
-o sample_2_out.mp4
# Subject Removal
python inference/infer_tav2av.py \
--video samples/sample_3.mp4 \
--prompt "Remove the table lamp from the foreground and fill in the window and curtains behind it. Keep the man and his dialogue unchanged." \
-o sample_3_out.mp4
# Speech Editing
python inference/infer_tav2av.py \
--video samples/sample_4.mp4 \
--prompt "Have her say: \"I did not expect the city to be this quiet at dawn.\"" \
-o sample_4_out.mp4
# Background Editing
python inference/infer_tav2av.py \
--video samples/sample_5.mp4 \
--prompt "Replace the background with a warm, sunlit kitchen with wooden cabinets. Keep the man, his clothing and his speech unchanged." \
-o sample_5_out.mp4
The two headline modes are extra flags on this script.
Post-inference self-refinement: re-denoise only one branch against the other, held as an accepted reference.
python inference/infer_tav2av.py \
--video samples/sample_4.mp4 \
--prompt "Have her say: \"I did not expect the city to be this quiet at dawn.\"" \
--post-refine audio --refine-lambda 0.5 \
-o sample_4_refined.mp4
At the default --refine-lambda 0.5 this runs round(0.5 * 30) = 15 refinement steps.
The count scales with lambda so the refine pass keeps the main pass's per-step
resolution; --refine-steps overrides it.
Intent-aware editing: hold one branch intact by conditioning, and let it constrain the other.
python inference/infer_tav2av.py \
--video samples/sample_5.mp4 \
--prompt "The room is now a quiet library, with faint page turns and distant footsteps." \
--intent-aware video \
-o sample_5_audio_edit.mp4
No input media: both branches are generated from the prompt alone.
python inference/infer_t2av.py \
--prompt "A man in a grey shirt sits at a wooden table in a quiet cafe, a window with soft daylight behind him and a few empty chairs further back. He says: \"We can start whenever you are ready, but let me finish this first.\" Static medium shot." \
-o sample_t2av.mp4
Given a soundtrack, generate the matching video; the input audio is preserved.
python inference/infer_ta2v.py \
--audio samples/sample_4.mp4 \
--prompt "Medium close-up of a woman with long dark hair speaking directly to camera, standing in a sunlit kitchen with pale oak cabinets and a steaming kettle on the counter behind her, warm late-afternoon light through a side window, shallow depth of field, static shot." \
-o sample_4_ta2v.mp4
Given a video, generate the matching soundtrack; the input video is preserved.
python inference/infer_tv2a.py \
--video samples/sample_1.mp4 \
--prompt "The man is out of breath after a set of pull-ups. A barbell is racked nearby in a bare gym, with voices and movement echoing off a hard wall." \
-o sample_1_tv2a.mp4
Every option has a working default, and the commands above need nothing beyond
--video / --audio / --prompt.
Output
| Option | Default | Notes |
|---|---|---|
--height, --width | 704, 1280 | Both must be multiples of 32. |
--num-frames | 121 | Must be 8k+1. |
--frame-rate | 24 | |
-o, --output | <task>_output.mp4 |
Sampling and memory
| Option | Default | Notes |
|---|---|---|
--num-inference-steps | 30 | Denoising steps. |
--seed | 42 | |
--offload | on | --no-offload keeps all weights resident; see Hardware. |
--device | cuda | |
--no-tiled | off | Decode the video without VAE tiling. Needs more VRAM. |
Guidance, set per branch.
| Task | video-text | audio-text | video-reference | audio-reference |
|---|---|---|---|---|
tav2av | 3.0 | 2.0 | 1.5 | 3.0 |
t2av, ta2v, tv2a | 6.0 | 6.0 | 1.0 | 1.0 |
The flags are --video-text-cfg-scale, --audio-text-cfg-scale,
--video-reference-cfg-scale and --audio-reference-cfg-scale.
A Gradio app is included:
python app.py
The released checkpoint is a 19B-parameter DiT in bfloat16, and text conditioning
adds Gemma-3 12B. The two together are ~65 GB, so weights are offloaded by default:
each phase loads only what it needs (the text encoder while encoding, the DiT while
denoising, the VAEs while decoding) and the rest waits on CPU. Peak residency is the
DiT alone, ~39 GB, plus activations. --no-offload keeps everything on the card for
speed and needs substantially more VRAM than a single 80 GB GPU provides at the
default 720p resolution.
UniAVC is built on DiffSynth-Studio,
whose model-loading framework and LTX-2.3 implementation the vendored uniavc/ package
derives from. The base generative model is
LTX-2.3 from Lightricks. Text conditioning uses
Gemma-3 from
Google. We thank JAVEdit for the useful training data and their benchmark for evaluation;
MiniMax-H3, InstructAV2AV for the
comparison baselines; and the wider open-source audio-video community that make this work complete.
If you find UniAVC useful for your research, please consider giving a star ⭐ and citation 📝 :D
@article{pham2026uniavc,
title = {UniAVC: Unified Audio-Video Creation via Progressive Training and Asymmetric Scheduling},
author = {Pham, Kien T. and Liu, Yuhao and Choi, Chong Hou and He, Zhenqi and Wang, Yanghao
and Lei, Chenyang and Liu, Rui and Chen, Qifeng and Chen, Long},
journal = {arXiv preprint arXiv:2610.24791},
year = {2026}
}
LTX-2 Community License.