Omni-LiveAvatar: Minute-Level Real-Time Streaming Joint Audio-Video Avatar Generation

Qualitative comparison of minute-level generation by Omni-LiveAvatar

Qualitative comparison of minute-level generation. Compared with state-of-the-art baselines, Omni-LiveAvatar preserves consistent avatar and background appearance, and maintains tight audio-video alignment over a minute-long rollout, whereas OmniForcing and Hallo-Live exhibit severe appearance drift. In the bottom row, the syllable uttered at each frame is highlighted in red, and the insets show the corresponding mouth shapes for lip-audio alignment.

Abstract

Joint audio-video generative models serve as the foundation for immersive and interactive digital-human generation. Nevertheless, most existing models rely on bidirectional attention and multi-step denoising and can generate only short clips, making them unsuitable for real-time interaction over extended durations. We present Omni-LiveAvatar, the first framework for minute-level, real-time streaming joint audio-video avatar generation. Specifically, we propose (1) a progressive autoregressive distillation pipeline that transfers a large bidirectional joint audio-video diffusion model into a few-step autoregressive generator without auxiliary stabilization mechanisms; (2) a synchronized audio-video long-short-term memory that preserves global consistency under a bounded memory budget; and (3) a hierarchical rolling prompt planning strategy that enables coherent semantic evolution and seamless prompt transitions. Extensive experiments show that Omni-LiveAvatar generates high-quality, synchronized minute-level avatars in real time. In terms of speed, it achieves a 33× generation speedup over its teacher, LTX-2, on a single NVIDIA H200 GPU; in terms of generation quality, it outperforms accelerated baselines across visual quality, audio quality, cross-modal synchronization, and human fidelity.

Progressive Autoregressive Distillation

A three-stage pipeline (Few-Step Distillation → Trajectory-Aligned Causal Initialization → Joint Rolling Forcing) that converts a bidirectional teacher into a real-time streaming generator without auxiliary stabilization mechanisms.

Synchronized Audio-Video Long-Short-Term Memory

Periodically re-anchored long-term memory combined with a rolling KV cache preserves global consistency, recent context, and cross-modal alignment under a bounded memory budget.

Hierarchical Rolling Prompt Planning

Decomposes the full prompt into a fixed global prompt and block-level local prompts that advance with the rolling window, enabling smooth semantic transitions without conflicts.

Method Overview

Omni-LiveAvatar combines progressive autoregressive distillation, synchronized audio-video long-short-term memory, and hierarchical rolling prompt planning.

Overview of Omni-LiveAvatar. The proposed progressive autoregressive distillation pipeline converts a large bidirectional audio-video diffusion model into a few-step causal generator without any auxiliary stabilization mechanism (top). At inference, the synchronized audio-video long-short-term memory preserves global consistency while retaining recent context within a bounded memory budget for minute-level streaming inference (middle), and the hierarchical rolling prompt planning organizes global and local prompts for smooth long-form semantic evolution (bottom).

Trajectory-Aligned Causal Initialization

Visualization of denoising trajectories: DMD aligns output distribution but disrupts pointwise trajectory

Visualization of the denoising trajectories (left) and output distributions (right) of the multi-step teacher and few-step student. DMD aligns the output distribution but disrupts the pointwise denoising trajectory. Our Stage II generates ODE pairs from the distilled few-step model itself, avoiding trajectory mismatch and stabilizing causal initialization.

Quantitative Results

Omni-LiveAvatar achieves state-of-the-art performance across video quality, human fidelity, audio quality, and audio-video synchronization, with a 33× speedup over LTX-2.

33× Speedup & 21.99 FPS

Real-Time Streaming Generation on NVIDIA H200

All-Metric SOTA

Best Across All Evaluated Metrics on 5-Second and 60-Second Generation

Table 1. Quantitative Comparison on 5-Second Avatar Generation

Method	Speed	Video Quality					VideoAlign↑	Human Fidelity			Audio Quality			SyncNet Confidence↑
Method	FPS↑	Subject Consistency↑	Background Consistency↑	Aesthetic Quality↑	Imaging Quality↑	Quality Score↑	VideoAlign↑	Human Anatomy↑	Human Identity↑	Human Clothing↑	UTMOS↑	DNSMOS↑	NISQA↑	SyncNet Confidence↑
LTX-2	0.60	92.54	93.42	56.37	71.53	79.39	7.77	96.23	92.93	99.57	3.54	3.75	3.47	6.89
Ovi	1.41	97.10	95.29	33.13	65.83	76.14	6.51	98.55	97.13	99.19	3.10	3.72	2.63	6.88
Real-time Autoregressive Models
OmniForcing	16.11	98.05	94.47	60.14	70.75	80.05	8.11	88.76	98.51	100.00	2.46	3.84	3.03	1.60
Hallo-Live	16.50	98.20	93.64	27.97	66.04	74.21	8.04	98.39	97.42	100.00	2.96	3.81	2.95	4.50
Omni-LiveAvatar	19.57	99.28	97.52	63.61	72.11	81.72	9.08	98.76	100.00	100.00	3.19	3.95	3.06	6.16

Best results among real-time autoregressive models are highlighted in bold. ↑ indicates higher is better.

Table 2. Quantitative Comparison on Minute-Level (60s) Avatar Generation

Method	Speed	Video Quality				VideoAlign↑	Human Fidelity			Audio Quality			SyncNet Confidence↑
Method	FPS↑	Subject Consistency↑	Motion Smoothness↑	Aesthetic Quality↑	Imaging Quality↑	VideoAlign↑	Human Anatomy↑	Human Identity↑	Human Clothing↑	UTMOS↑	DNSMOS↑	NISQA↑	SyncNet Confidence↑
Real-time Autoregressive Models
OmniForcing	16.18	98.71	99.38	54.90	71.70	6.68	92.14	50.19	100.00	1.60	3.51	2.90	0.28
Hallo-Live	13.80	99.50	99.19	47.16	61.18	5.46	97.81	67.60	77.20	2.02	3.97	2.99	0.72
Omni-LiveAvatar	21.99	99.55	99.68	61.98	71.71	9.82	99.85	98.61	100.00	2.80	4.04	3.24	6.76

Best results among real-time autoregressive models are highlighted in bold. ↑ indicates higher is better.

Qualitative Comparisons

Omni-LiveAvatar generates realistic, temporally consistent avatars with stable appearance and accurate audio-video synchronization across both short and minute-long generation.

5-Second Generation

Qualitative comparison of 5-second avatar generation. Omni-LiveAvatar generates realistic and temporally consistent avatars with visual quality comparable to Ovi and LTX-2, whereas OmniForcing exhibits noticeable realism degradation and Hallo-Live suffers from facial and hand artifacts as well as color drift.

60-Second Generation

Additional qualitative comparisons on 60-second avatar generation. Omni-LiveAvatar maintains consistent avatar and background appearance throughout minute-long generation, while baselines exhibit severe appearance drift and degraded synchronization.

Ablation Study

Ablations validate the individual contributions of each proposed component.

Ablation on Training Strategies

Variant	Video Quality	Audio Quality			SyncNet Confidence↑	VideoAlign↑
Variant	Quality Score↑	UTMOS↑	DNSMOS↑	NISQA↑	SyncNet Confidence↑	VideoAlign↑
w/o RF & w/o TA-ODE	72.05	1.51	3.21	2.70	1.23	1.24
w/o RF	80.67	1.93	3.38	2.44	4.58	8.09
w/o TA-ODE	81.25	3.14	3.81	2.96	5.98	8.63
Ours (full)	81.72	3.19	3.95	3.06	6.16	9.08

We compare against three settings: (1) using joint self-forcing instead of joint rolling forcing and generating ODE pairs with the multi-step teacher (w/o RF & w/o TA-ODE); (2) using joint self-forcing instead of joint rolling forcing (w/o RF); and (3) generating ODE pairs with the multi-step teacher (w/o TA-ODE). Removing both components causes drastic degradation across all metrics. w/o RF mainly degrades audio quality and synchronization, while w/o TA-ODE primarily degrades video quality and text-video alignment. Only combining both achieves the best performance.

Ablation on Memory Mechanisms

Variant	Video Quality		Audio Quality			SyncNet Confidence↑	VideoAlign↑
Variant	Aesthetic Quality↑	Imaging Quality↑	UTMOS↑	DNSMOS↑	NISQA↑	SyncNet Confidence↑	VideoAlign↑
w/o LTM	58.61	71.16	2.79	4.03	3.22	5.79	9.70
w/o re-anchor	61.16	71.43	2.78	4.02	3.23	6.70	9.82
async LTM	61.93	71.69	2.76	4.03	3.21	6.61	9.80
Ours (full)	61.98	71.71	2.80	4.04	3.24	6.76	9.82

We compare against three settings: (1) without the long-term memory (w/o LTM); (2) without periodic RoPE re-anchoring (w/o re-anchor); and (3) with temporally misaligned audio and video long-term memory (async LTM). w/o LTM causes the largest degradation in both video quality and synchronization, indicating that video generation is more vulnerable to temporal drift. Removing re-anchoring mainly weakens video quality metrics, while async LTM primarily harms cross-modal alignment.

Ablation on Prompt Planning Strategies

Variant	Video Quality		Audio Quality			SyncNet Confidence↑	VideoAlign↑
Variant	Aesthetic Quality↑	Imaging Quality↑	UTMOS↑	DNSMOS↑	NISQA↑	SyncNet Confidence↑	VideoAlign↑
w/o rolling prompt	55.74	68.45	1.60	3.28	2.38	4.47	9.72
5s-interval switching	61.23	71.33	2.65	3.94	3.19	5.11	9.71
Ours (full)	61.98	71.71	2.80	4.04	3.24	6.76	9.82

We compare against two settings: (1) providing the full prompt to the model at once (w/o rolling prompt); and (2) updating the prompt every 5 seconds (5s-interval switching). Audio-related metrics are more sensitive to the prompt planning strategy than video metrics, confirming that audio requires a finer-grained and smoother prompt schedule. Providing the full prompt at once yields the largest drops across nearly all metrics, while switching every 5s mainly degrades audio quality and audio-video alignment. Our hierarchical rolling prompt planning enables smooth, continuous, and aligned audio-video generation under the joint rolling-forcing inference paradigm.

BibTeX

@misc{zhu2026omniliveavatar,
  title={Omni-LiveAvatar: Minute-Level Real-Time Streaming Joint Audio-Visual Avatar Generation},
  author={Lunjie Zhu and Xingtong Ge and Fangyu Lin and Yi Zhang and Zhening Liu and Mengfei Li and Yumeng Zhang and Guanglu Song and Yu Liu and Jun Zhang},
  year={2026},
  eprint={2608.13602},
  archivePrefix={arXiv},
  primaryClass={cs.MM},
  url={https://arxiv.org/abs/2608.13602}
}