Integrating Auxiliary Information for Visual Scene Understanding and Generation

Abstract:
Understanding and generating complex visual scenes requires models to reason about multiple complementary forms of information. These include semantic information about objects and actions, instance-level information about individual entities, spatial and geometric information about their locations and relationships, and quantitative information about how many instances are present. In many computer vision systems, however, these forms of information are modeled separately, with action recognition, object counting, and image generation often relying on task-specific representations. This thesis investigates whether these limitations can be reduced by integrating auxiliary information from pre-trained foundation models into visual representations. The central premise is that foundation models encode complementary semantic, geometric, spatial, instance-level, and cross-modal knowledge that can provide structure missing or only weakly represented in the primary task formulation.

The thesis develops this idea across four contributions. First, multi-label action recognition is reformulated as a detection-style set-prediction problem with multimodal semantic queries, where learnable action queries are enriched using visual and textual representations from pre-trained vision-language models. This provides an actor-agnostic formulation that can recognize multiple concurrent actions across human and animal domains without relying on actor-specific pose representations. Second, open-vocabulary multi-label object counting is addressed through semantic-geometric priors for spatially grounded counting, combining semantic segmentation, monocular depth estimation, and promptable instance segmentation. This enables multiple object categories to be counted without task-specific training while retaining information about their spatial locations and individual instances. Third, count-faithful image generation is developed through spatial planning, instance-aware attention composition, and iterative count verification, so that object quantity and arrangement are explicitly considered during synthesis rather than being left entirely to the implicit representation of the generative model. Finally, count understanding and generation are unified through cycle-consistent feedback between understanding and generation, where spatially grounded count reasoning is used not only for analyzing visual scenes but also to provide direct feedback for improving count-controlled image synthesis.

Overall, the thesis shows that foundation models are most effective when complementary semantic, geometric, spatial, instance-level, and quantitative information is integrated in a structured manner, providing more grounded and transferable representations for visual scene understanding and generation.