Analyzing Big Data Workloads Using Discrete Simulation

Abstract

Virtualized IT platforms abstracting from physical infrastructure offer easy scaling of storage and compute resources. This is a great progress for handling dynamic big data workloads but holds the danger of solving performance issues just with upscaling, without looking at causes. We propose to analyze the behavior of complex systems handling big data workloads using system modelling and discrete event simulation. It can be very revealing to see a simulated system in action by running simulations in different configurations, showing the impact of a modified system structure on its runtime behavior and resource consumption. With a toolbased system workload simulation example in the Earth Observation data domain we show how this approach can lead to significant resource and cost savings.