Uncertainty Quantification in CLIP using Ensemble Methods

Abstract

The multimodal image-text model CLIP, trained with a constrastive objective, shows impressive generalization abilities in downstream tasks, such as zero-shot classification. However, the model’s performance is less promising in several complex tasks, such as traffic sign and satellite image classification. In this thesis, we study the quantification of CLIP’s data bias by examining its learned embedding space. In addition, we investigate methods to quantify the uncertainty in CLIP’s zero-shot predictions. To do this, we conduct experiments on five datasets on which CLIP has a wide range of zero-shot classification accuracies, using baseline methods and extending the use of existing data augmentation methods. Furthermore, we pursue and develop a novel approach of transferring ensemble methods to CLIP, and present three computationally efficient variants. The results show that the presented methods are capable of quantifying model uncertainty for given inputs, but also suggest that there is room for improvement. Regarding our novel approach of using ensembles, the results show that we can match, and in some cases exceed by over 6%, the accuracy of the original CLIP model. We also show that the ensembles can detect the datasets with high uncertainty, indicating CLIP’s biases.