press release
Published: 17 June 2024

Surrey researchers teach AI to spot what you’re sketching

A new way to teach artificial intelligence (AI) to understand human line drawings – even from non-artists – has been developed by a team from the University of Surrey and Stanford University.  

The new model approaches human levels of performance in recognising scene sketches. 

People of all ages and backgrounds use drawings to explore new ideas and communicate. Yet, AI systems have historically struggled to understand sketches.  

AI has to be taught how to understand images. Usually, this involves a labour-intensive process of collecting labels for every pixel in the image. The AI then learns from these labels.   

Instead, the team taught the AI using a combination of sketches and written descriptions. It learned to group pixels, matching them against one of the categories in a description.

The resulting AI displayed a much richer and more human-like understanding of these drawings than previous approaches. It correctly identified and labelled kites, trees, giraffes and other objects with an 85% accuracy. This outperformed other models which relied on labelled pixels.  

As well as identifying objects in a complex scene, it could identify which pen strokes were intended to depict each object. The new method works well with informal sketches drawn by non-artists, as well as drawings of objects it was not explicitly trained on.    

Drawing and writing are among the most quintessentially human activities and have long been useful for capturing people's observations and ideas.

This work represents exciting progress towards AI systems that understand the essence of the ideas people are trying to get across, regardless of whether they are using pictures or text.
Professor Judith Fan, Assistant Professor of Psychology at Stanford University
portrait of judith fan

The research forms part of Surrey’s Institute for People-Centred AI, where we draw on Surrey's domain expertise across different disciplines encouraging the creation and sharing of innovative ideas and approaches to foster new research directions and collaborations in AI. 

The SketchX programme seeks to understand the way we see the world by the way we draw it.

The findings will be presented at the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024. It takes place in Seattle from 17-21 June 2024. 

The paper is available for free on arXiv

Related sustainable development goals

Image for Industry, Innovation, and Infrastructure Sustainable Development Goal