Functional Monitoring of Object Detectors via CCS: A Label - Free Evaluation Metric for Embedded AI

Kurzfassung

Abstract. Evaluating object detection (OD) models often relies on ground truth annotations, which are unavailable in many real-world scenarios. To address this, we propose the Cumulative Consensus Score (CCS), a metric that enables DevOps-style monitoring and iterative refinement of object detectors without human annotations. CCS leverages TestTime Data Augmentation (TTDA) to estimate spatial uncertainty, allowing comparison between existing and newly developed object detection models. We validate CCS through controlled experiments by comparing its congruence with established ground-truth-based metrics, including F1 score, pairwise Probabilistic Detection Quality (pPDQ), and Optimal Correction Cost (OC-cost). Unlike dataset-level metrics such as mAP, these metrics and CCS help identify underperforming cases for targeted evaluation. Experiments across multiple model architectures demonstrate CCS's robustness and alignment with standard metrics. We also outline future directions for improving CCS.