Prediction of Neural Network Latency on Embedded GPU Accelerators

Abstract

AI systems are transforming many application areas, but resource constraints prevent many especially time critical embedded applications. In particular, a fast prediction of expected latency and energy costs can enable the hardware-aware design of such systems. Thus, we started developing a latency model for the NVIDIA Jetson platform, which, with its GPU-based architecture is interesting in e.g. automotive vision use-cases. Using latency models in automatic optimization flows yields benefits, compared to abstract metrics like parameter count. However, it needs to be fast, for optimization algorithms to cover large search spaces. We developed a layer-wise model using the parameters of the layer (like input size) and a number of hardware measurements to produce a model of the network latency. On the Jetson platform the characterization yielded a model, that in first tests provided three key insights: Using only a few measurements we can already accurately predict upper and lower bound of the execution time. Using more information, we can, under ideal conditions, estimate latency with a mean average percentage error below 3%. Lastly, we found that sometimes a small change of input parameters can lead to up to 10% reduction in latency for that layer.