Towards Robust and Structured Audio-Visual Video Parsing under Weak Supervision

Abstract:
Audio-visual video parsing (AVVP) aims to detect and localize events in videos and determine whether each event is audible, visible, or jointly perceived from both modalities. It is commonly studied under weak supervision, where only video-level labels are available during training, while segment-level and modality-specific predictions are required during inference. This supervision gap makes AVVP challenging, as the model must infer fine-grained temporal evidence from coarse labels. The task is further complicated by audio-visual asynchrony, modality-exclusive events, background noise, and partially observable event cues.

This thesis investigates weakly supervised AVVP through three complementary perspectives: fine-grained perception, robust learning, and semantic-temporal reasoning. First, a fine-grained and reliability-aware perception framework is proposed to improve the quality of local evidence. It introduces segment-level attention and reliability-aware fusion mechanisms to highlight informative temporal segments and suppress unreliable audio-visual cues. By estimating reliability from uncertainty, cross-modal similarity, and feature strength, the model can adaptively regulate modality contributions during fusion.

Second, a robust segment-level learning framework is developed to address unstable supervision under weak labels. It introduces an exponential moving average teacher-student strategy to generate stable pseudo supervision, a class-aware cross-modal agreement loss to avoid unreliable alignment between asynchronous modalities, and an event-span multiple instance learning objective to encourage temporally coherent event evidence. These components provide more reliable training signals and reduce the negative effects of noisy pseudo labels and indiscriminate cross-modal alignment.

Third, a text-enriched and category-aware temporal reasoning framework is proposed to strengthen semantic and temporal reasoning. In this framework, the Bi-directional Text Fusion module uses modality features as semantic anchors to calibrate pseudo-text embeddings, reducing semantic mismatch between static textual representations and the corresponding audio-visual content. The Category-Aware Temporal Graph module models event-dependent temporal structures by adaptively selecting multi-scale temporal neighbours according to predicted category semantics and learning category-specific temporal decay factors.

Experiments on LLP, AVE, and UnAV-100 datasets demonstrate the effectiveness of the proposed methods. The results show that reliability-aware perception improves fine-grained evidence selection, robust learning stabilizes weakly supervised training, and semantic-temporal reasoning enhances modality-aware event understanding. Overall, the results suggest that, beyond the design of audio-visual interaction, local evidence perception, the learning process under weak labels, and semantic-temporal reasoning each provide an effective route to improving weakly supervised AVVP.