From Strokes to Scenes: Understanding Freehand Sketches for Image and Multi-View Generation

Abstract:
This thesis investigates freehand scene sketches as a control signal for visual generation. Such sketches convey rich semantic and spatial information with only a few strokes, but their abstraction and geometric distortions are challenging for models trained primarily on natural images. We address this problem through three contributions that progress from sketch understanding to image generation and geometrically consistent multi-view synthesis.

First, we introduce a CLIP-based, language-supervised scene sketch segmentation method trained on bitmap sketches and captions. Two-level hierarchical training and value-value self-attention increase Acc@P from 28.82% for zero-shot CLIP to 85.54%, an improvement of 56.72 percentage points on FS-COCO. A user study identifies the remaining gap between human and machine performance.

Second, SketchingReality conditions latent diffusion on semantic sketch features. We extract the features from the specialized sketch encoder we designed in the first project. A modulation network then injects them into denoising, while attention supervision driven by visual-token/text-category similarity enables training without pixel-aligned ground truth. In the user study, it is preferred over the tested ControlNet and T2I-Adapter settings for sketch adherence and its balance with realism; ControlNet SDXL is preferred for realism alone.

Third, we define the Sketch-to-Multi-View task of generating geometrically consistent views of a scene from one freehand sketch. We curate 9,222 scenes with 33 views each and equip a frozen video diffusion transformer with lightweight camera-aware attention adapters that inject relative projective geometry at 2.7% additional parameters. A sparse correspondence loss built from Structure-from-Motion lets the model generate all views jointly in one denoising pass, without per-scene optimisation. Additional validation on real posed imagery shows that the camera adapters and correspondence supervision transfer beyond the generated sketch dataset. They achieve the best PSNR among the compared methods for three-input reconstruction and single-input long-trajectory synthesis, and the resulting views support downstream 3D reconstruction.

Together, these contributions show that freehand scene sketches when used adequately provide sufficient semantic and spatial information to control image and multi-view generation.