Multi-Modal Narrative Understanding for Videos

Abstract:

"With the rapid growth in online video content, understanding what happens in a video has become a central problem in computer vision. Video captioning attempts to describe this content in natural language, but existing methods focus on describing isolated actions rather than capturing how events are connected, why they occur, or what they mean socially. This limits the utility of video understanding for applications such as accessibility tools that explain events to visually impaired users, instructional systems that must identify which steps are prerequisites for others, robotics systems that plan and diagnose sequential tasks, and social robotics that must interpret nonverbal communication between people. This thesis addresses multi-modal narrative understanding for videos through five contributions. Architecturally, the work traces a progression from LSTM-based sequential modelling through attention-based cross-modal alignment to pretrained transformer-based vision-language models, with each stage motivated by limitations identified in the previous one.

The first contribution addresses grammatical and semantic correctness in video captioning. Existing methods either capture global visual information while missing fine-grained local detail, or generate word-by-word descriptions that do not align with grammatical structure. We address both by introducing SEM-POS, a global-local fusion network built on bidirectional LSTMs with four parts-of-speech (POS) blocks that align visual features with linguistic components at the local level. A Global-Local Fusion Block (GLFB) combines these representations with global features to generate captions that are both grammatically well-formed and semantically correct to the video content. The method is applied to the ForecasterFlexOBM production-quality broadcast media dataset, demonstrating that the learned alignment transfers to professional content and supports automatic captioning for accessibility and object-based media production workflows.

The second contribution addresses the problem that audio and visual streams carry complementary information that existing methods do not align at the spatial, temporal, and semantic levels simultaneously. We introduce CAD, a contextual multi-modal alignment network for audio-visual question answering, with dedicated components for each level of misalignment. We show that these contributions transfer to existing methods without additional architectural complexity. We further introduce Attend-Fusion, a compact attention-based audio-visual fusion architecture for video classification, demonstrating that strong performance can be achieved with a model nearly 80% smaller than competing baselines. This establishes that attention-based audio-visual alignment is not only effective but also deployable in resource-constrained environments, motivating the move toward transformer-based architectures in subsequent work.

The third contribution identifies that existing video captioning benchmarks describe observable actions without capturing causal, temporal, or spatial narrative structure. Models trained on these benchmarks cannot articulate how one event leads to another or how spatial arrangements drive causal relationships. We introduce NarrativeBridge, the first Causal-Temporal Narrative (CTN) captions benchmark dataset and a Cause-Effect Network (CEN) trained through contrastive learning on CLIP-ViT features. We then introduce Spatial-NarrativeBridge, which extends the benchmark with spatial grounding through a vision-language model annotation generation pipeline. A central finding is that the Cause-Effect Network learns spatial understanding from spatially enhanced captions without architectural modification, supporting data-centric principles over model complexity. These benchmarks enable models that can explain why events occur rather than merely describe what happens.

The fourth contribution addresses the fragmented treatment of nonverbal communication in computer vision. Facial expression recognition, gesture recognition, and gaze estimation are studied in isolation despite the fact that nonverbal signals account for the majority of information exchanged in face-to-face interaction. We introduce NVI-47, a benchmark for fine-grained nonverbal interaction detection that expands the existing taxonomy from 22 to 47 categories with near-exhaustive annotation coverage, addressing training pathologies in the baseline dataset caused by sparse supervision. We introduce language-grounded detection through cross-attention between visual features and text embeddings of category names, combined with a Graph Attention Network for explicit spatial reasoning between individuals. This enables applications in social robotics, human-computer interaction, and assistive technology where understanding the full nonverbal narrative of a social scene is essential.

Finally, the fifth contribution extends causal understanding from short clips with single cause-effect pairs to long-form videos with complex causal structures, including parallel chains and convergent causality. We find that existing vision-language models (VLMs) hallucinate causal relationships in the majority of failure cases, even after fine-tuning, predicting causality from temporal proximity rather than causal necessity. We address this through two contributions. The Dense Causal Captions (DCC) benchmark dataset provides the first large-scale resource for dense causal captioning. Causal-VLM is a two-stage architecture that fuses visual and semantic features through curriculum learning and a multimodal causal head, demonstrating that combining these modalities allows models to distinguish causality from temporal correlation. An important secondary finding is that causal supervision also improves caption quality and temporal localisation, as reasoning about causal structure forces the model to localise event boundaries precisely. The model transfers to the causal-temporal narrative benchmarks, outperforming NarrativeBridge and demonstrating that training on complex long-form causal structures generalises to simpler scenarios."