Zheng Zhang

Solution for addressing real-world image classification challenges. Human-AI collaborative classification (HAI-CC) aims to synergise the efficiency of machine learning classifiers and the reliability of human experts to support decision making. Learning to defer (L2D) has been one of the promising HAI-CC approaches, where the system assesses a sample and decides to defer to one of human experts when it is not confident. Despite recent progress, existing L2D methods rely on the strong assumption of ground truth label availability for training, while in practice, most datasets often contain multiple noisy annotations per data sample without well-curated ground truth labels. In addition, current L2D methods either consider the setting of a single human expert or defer the decision to one human expert, even though there may be multiple experts available, resulting in a suboptimal utilisation of available resources. Furthermore, current HAI-CC evaluation frameworks often overlook processing costs, making it difficult to assess the trade-off between computational efficiency and performance when bench-marking different methods. To address these gaps, this paper introduces LECOMH-a new HAI-CC method that learns from noisy labels without depending on clean labels for training, simultaneously maximising collaborative accuracy with either one or multiple human experts, while minimising the cost of human collaboration. The paper also introduces benchmarks featuring multiple noisy labels per data sample for both training and testing to evaluate HAI-CC methods. Through quantitative comparisons on these benchmarks, LECOMH consistently outperforms HAI-CC methods and baselines, including human experts alone, multi-rater learning and noisy-label learning methods across both synthetic and real-world datasets.

The advent of learning with noisy labels (LNL), multi-rater learning, and human-AI collaboration has revolutionised the development of robust classifiers, enabling them to address the challenges posed by different types of data imperfections and complex decision processes commonly encountered in real-world applications. While each of these methodologies has individually made significant strides in addressing their unique challenges, the development of techniques that can simultaneously tackle these three problems remains underexplored. This paper addresses this research gap by integrating noisy-label learning, multi-rater learning, and human-AI collaboration with new benchmarks and the innovative Learning to Complement with Multiple Humans (LECOMH) approach. LECOMH optimises the level of human collaboration during testing, aiming to optimise classification accuracy while minimising collaboration costs that vary from 0 to M, where M is the maximum number of human collaborators. We quantitatively compare LECOMH with leading human-AI collaboration methods using our proposed benchmarks. LECOMH consistently outperforms the competition, with accuracy improving as collaboration costs increase. Notably, LECOMH is the only method enhancing human labeller performance across all benchmarks.

With the development of Human-AI Collaboration in Classification (HAI-CC), integrating users and AI predictions becomes challenging due to the complex decision-making process. This process has three options: 1) AI autonomously classifies, 2) learning to complement, where AI collaborates with users, and 3) learning to defer, where AI defers to users. Despite their interconnected nature, these options have been studied in isolation rather than as components of a unified system. In this paper, we address this weakness with the novel HAI-CC methodology, called Learning to Complement and to Defer to Multiple Users (LECODU). LECODU not only combines learning to complement and learning to defer strategies, but it also incorporates an estimation of the optimal number of users to engage in the decision process. The training of LECODU maximises classification accuracy and minimises collaboration costs associated with user involvement. Comprehensive evaluations across real-world and synthesized datasets demonstrate LECODU’s superior performance compared to state-of-the-art HAI-CC methods. Remarkably, even when relying on unreliable users with high rates of label noise, LECODU exhibits significant improvement over both human decision-makers alone and AI alone (Supported by the Engineering and Physical Sciences Research Council (EPSRC) through grant EP/Y018036/1). Code is available at https://github.com/zhengzhang37/LECODU.git.