Multimodal Large Language Models (MLLMs) have shown significant progress in offline video understanding. However, applying these models to real-world online scenarios, such as autonomous driving, augmented reality, and surveillance, presents unique challenges due to the need for real-time processing of continuous online video streams. This paper aims to alleviate this issue from three perspectives: evaluation benchmark, model architecture, and training strategy. First, we introduce OVBench, a comprehensive question-answering benchmark specifically designed to evaluate models' ability to perceive, memorize, and reason within online video contexts. It features six core task types across three temporal contexts—past, present, and future—forming 16 subtasks from diverse datasets. Second, we propose a novel Pyramid Memory Bank (PMB) that effectively retain key spatiotemporal information in video streams. Third, we proposed an offline-to-online learning paradigm, designing an interleaved dialogue format for online video data and constructing an instruction-tuning dataset tailored for online video training. This framework led to the development of VideoChat-Online, a robust and efficient model for online video understanding. Despite lower computational cost and higher efficiency, VideoChat-Online outperforms existing state-of-the-art offline and online models across popular offline video benchmarks and OVBench, demonstrating the effectiveness of our model architecture and training strategy.

An illustration of the model's inference process with the pyramid memory bank structure. m_main queues maintain balanced spatiotemporal information at different hierarchical levels, m_t is a high-frequency sampling queue for enhanced temporal detail preservation, and m_s queue is for spatial detail retention. The system supports simultaneous frame input to both the memory bank and KVCache, with synchronization mechanisms for maintaining consistency during memory modifications.