Building Vision Encoders for Efficient and Effective Learning from Biomedical Images

Abstract:
The vision encoder is a fundamental component of modern computer vision, conventionally built for natural photographic images. Biomedical imaging data has grown rapidly with large-scale screening, and comprises subdomains with characteristics natural images do not possess. Off-the-shelf vision encoders apply directly to most of them. The exceptions lie in microscopy imaging, where the image properties do not satisfy what a general purpose encoder requires. In computational pathology, images span gigapixel scale and cannot be processed in a single pass due to computational cost. In multi-channel microscopy, each channel is a distinct semantic entity and the channel set varies across datasets, so the image cannot be processed due to architectural mismatch. This thesis addresses both constraints. The thesis first establishes that the internal representation of an encoder should be adapted to the form of the downstream task rather than inherited from natural images, shown on multi-label chest radiography. That principle is applied to each constraint in turn. Where the image is too large to process at once, the encoder is ordinarily excluded from training, so its representations are fixed before the task is known. To address this, the encoder is kept in the training loop at gigapixel scale, which matches or exceeds compute-efficient methods built on in-domain encoders, and improves further when applied on top of one. Where the channel configuration varies, an encoder defined for one set of channels cannot be applied to another. The fact that channels are semantically distinct is leveraged at three stages. During pre-training, the semantic relationship between channels is encoded as a prior, yielding representations that transfer to an unseen channel set. During frozen-encoder adaptation, preserving the distinction while the representation is read matches fine-tuning from scratch. During full fine-tuning, the distinction is maintained within the encoder and channel exchange is made adaptive, which improves substantially on the existing protocol. These mechanisms extend to dense prediction, highly multiplexed images and novel training strategies, and are evaluated across histopathology, fluorescence microscopy and imaging mass cytometry.