A Real-Time Audio-Visual Generation Framework

Vorch-Streamer Extending Human Audio-Visual Generation to Real-Time Long-Form Streaming

Menglin Han^1,2*, Yang Ding^1*, Yulei Lu¹, Haoran Yu^1,3, Xin Ma¹, Junyi Chen^1,4,
Zhangkai Ni^2†, Lin Ma^†, Yaohui Wang^1†

¹Vorch Team ²Tongji University ³Harbin Institute of Technology, Shenzhen ⁴Shanghai Jiao Tong University

*Equal contribution †Corresponding authors

Paper Code (Coming soon)

Overview

Vorch-Streamer post-trains LTX2.3 for causal, block-wise generation and adds implicit LLM-based speech and audio planning. The model emits synchronized audio and video before future content exists, then remains stable while its own predictions become context for every next block.

Causal Audio-Video Forcing

Predict synchronized blocks under a bounded context window before future frames are available.

Long-Horizon Self Forcing

Adapts the model to its own generated context for stable identity and motion over long rollouts.

27.12 FPS Streaming

Native text-to-audio-video generation exceeds the 24 FPS real-time playback threshold.

LLM-Based Implicit Speech and Audio Planning

Plans speech and audio progression implicitly so long-form generation stays coherent and synchronized.

Generation Gallery

Long-form generations

Interviews, cinematic dialogue, product presentation, and everyday conversation across a growing set of real-time rollouts.

Related Works

Compare

Flat, same-condition comparisons across native T2AV pipelines.

Compare 01

Four methods video compare

Every method receives the same benchmark case.

Quantitative Comparison

Real-time speed without giving up long-horizon fidelity

Real-time throughput27.12 FPS

1.13× real-time playback at 24 FPS, with strong synchronization, speech accuracy, and identity preservation.

Method	Params	Task	Stream	FPS ↑	Sync-C ↑	WER ↓	Identity ↑
LTX2.3	22B	T2AV	-	1.83	6.80	7.59%	0.8733
JoyAI-Echo	22B	T2AV	-	3.08	2.44	9.34%	0.9789
OmniForcing	19B	T2AV	Yes	12.12	0.75	98.79%	0.8012
Hallo-Live	11.7B	T2AV	Yes	11.51	0.58	94.13%	0.8670
Vorch-Streamer Ours	22.8B	T2AV	Yes	27.12	6.62	7.92%	0.9996