Interpreting and Building from Human Drawn Sketches

Abstract:
A quick drawing often says what a sentence cannot. People sketch to communicate because sketching is expressive, direct, and needs no training. Two things stand in the way of a machine doing the same: a drawing varies with subjective human interpretation of real world objects, and often conveys part of human intent. This thesis studies how machines can work with real, imperfect drawings, both interpreting a sketch they are given and generating using one as a control signal. Importantly, these roles require efforts in different directions. Interpretation must absorb sketch variation while upholding sketch intent; generation must decide how much of spatial control is to be taken from the drawing.

Sketch variation runs along three axes: style, abstraction, and length. Style and abstraction come first, and coping with them takes knowledge of the objects being drawn rather than of drawings themselves; the first contribution borrows that knowledge from a foundation model. We test an off-the-shelf VLM: CLIP, observing that it performs poorly on abstract drawings, degrading even further with more abstraction. We adapt CLIP rather than train from scratch: keeping the model frozen preserves its generalisation, while learned visual prompts and a raster-to-vector auxiliary task make it sketch-aware. For abstraction, we learn a codebook of coarse abstraction levels and mix its codes to cover the spectrum in between, injecting the mixture as an additional prompt. The adapted model then reaches categories it was never trained on, and abstraction levels it was never shown.

The second contribution turns to length, and to the strokes that produce it. Sketch pipelines inherit formats that are bulky (raster), unbounded in size (vector), or unreliable on long drawings (learned auto-regressive models). SketchINR treats a sketch as an implicit neural function of time and strokes. Because the code it decodes from has a fixed width, a longer drawing costs no more to store. It reconstructs complex sketches faithfully, compresses far beyond raster and vector formats, and can re-render one drawing at whatever level of detail is wanted, as a person would vary it.

Generation is a different problem. Neither exact geometry nor exact motion is conveyed in a freehand drawing, so a generator must construct both, while depending on the sketch for identity cues. The third contribution generates 3D shapes by keeping the output-space restricted: a diffusion model is trained in a part-aligned latent shape space, with a separate model per object category, so that nothing is asked of the sketch beyond the arrangement of parts; sketches reach that space through synthetic edgemap translations. Rough doodles then produce precise 3D shapes, with local editing included, and no paired data linking human sketches to shapes is ever collected.

The final contribution faces the reverse problem. What an animation needs is motion, and sketch videos are scarce, so FlipSketch takes its motion from a pre-trained text-to-video diffusion model. To prevent motion priors from overwriting input sketches, the model is fine-tuned on line-drawing frames and generation is anchored to the input sketch through inversion. The animation moves freely while keeping the appearance of the input sketch.

The first two contributions deliver sketch interpretation and representation that generalise across categories, abstraction, and length; the last two build generators that turn one drawing into a precise shape or animation, one weakening the sketch's control and one strengthening it. Together they cover both roles of the sketch, each in two settings. In every one of them the same decision recurs, over how much of the drawing to keep and how much to let the model build.