Ben Canini

Sketches provide a sparse yet expressive modality for human–machine interaction, but their use in 3D scene understanding remains largely unexplored. While sketch-based image retrieval (SBIR) and text-driven 3D localisation have made significant progress, the problem of localising specific object instances in full 3D scenes from free-hand sketches has received comparatively little attention. We introduce a framework for sketch-driven 3D object localisation in reconstructed indoor environments. Given a human sketch, our method retrieves candidate detections across a pose-aligned image bank, ranks them using a sketch-conditioned similarity model, and lifts the highest-ranked observations into 3D using calibrated camera geometry. Multi-view frustum aggregation over a structured primitive scene representation then produces a geometrically consistent object hypothesis. Experiments across multiple apartment-scale environments demonstrate that sketches provide a powerful complement to text-based retrieval, substantially reducing localisation errors in scenes containing multiple visually similar object instances. These results establish sketch-driven localisation as a promising modality for resolving instance-level ambiguity in large-scale 3D environments.

Camera pose estimation has traditionally relied on RGB images, assuming access to rich visual detail and low-level features. In this paper, we introduce a novel framework for estimating 6-DoF camera pose directly from sketch inputs. This new setting is both highly challenging—sketches are abstract, sparse, and stylistically diverse—and uniquely valuable, enabling pose estimation when photos are unavailable or when structural abstraction is preferable (e.g., design, ideation, cultural heritage).We present Sketch-a-Pose (SAP), a lightweight framework that bridges the modality gap between drawings and geometry through three components: (i) a sketch adapter that maps CLIP embeddings into the pose-sensitive DINO space for robust initialization, (ii) a sparse Gaussian edge representation that captures sketch-aligned structure and supports efficient rendering, and (iii) an analysis-by-synthesis refinement loop guided by a structural LPIPS loss.Across the Tanks and Temples and MIP-NeRF 360 datasets, SAP demonstrates that sketches alone can yield accurate camera pose recovery, often rivaling or even surpassing RGB-based baselines such as iNeRF, NeMO+VoGE, and 6DGS. Beyond raw numbers, SAP reveals a surprising insight: structural alignment from sketches can outperform photorealistic cues, suggesting a new paradigm of semantic-guided geometric vision.

Robust 3D edge reconstruction from in-the-wild data remains a key challenge in robotics and SLAM, where scenes are often noisy, unstructured, and captured under imprecise conditions. Recent methods based on neural implicit representations offer compact reconstructions but suffer from high computational costs and poor localization of fine edge detail due to reliance on NeRF-style volumetric rendering. While Gaussian Splatting offers a fast and accurate alternative, its use in edge extraction has been limited to clean or synthetic datasets with precise initialization.We present EdgeSplats, a method for 3D edge reconstruction that operates effectively on real-world data without requiring clean geometric priors. Our approach leverages the 3DGS-MCMC pipeline to train edge-aligned Gaussians directly from posed images. A graph-based clustering stage then extracts edge-consistent splats by exploiting spatial and directional coherence.We demonstrate that EdgeSplats improves both reconstruction quality and training speed over existing methods, demonstrating strong performance on noisy, real-world inputs.