Optimal Model Placement in Heterogeneous Edge AI Environments

Abstract

With the rise of high-resolution sensors in Earth observation satellites, wildlife cameras, and autonomous vehicles, the volume of edge-generated data has significantly increased. Analyzing this data using Deep Learning (DL) models often requires moving it to remote computing facilities, leading to longer execution times and higher energy consumption. To minimize unnecessary data transfer, Edge AI accelerators like the Edge TPU (ETPU) offer fast, low-power inference close to data sources. However, due to limited on-chip memory and support for various neural network operations, these accelerators are often used with power-efficient CPUs in single-user, single-model scenarios. In this work, we tackle the automatic mapping of multiple DL models to heterogeneous resources (CPUs and ETPUs) for efficient inference in multi-user, multi-model environments. We present Maggie, which optimally allocates DL models to minimize latency and maximize ETPU memory utilization. Our results indicate that Maggie achieves up to 7 times lower latency compared to a CPU-only baseline and about 3 times lower latency than an ETPU-only baseline across various neural network architectures.