Evaluation of Open Source Static Analysis Security Testing (SAST) Tools for C

Abstract

The goal of SAST-tools is to help developers coding software in a more secure fashion by pointing early at suspicious constructs, insecure API usage or dangerous run-time errors. There exists a variety of tools which claim more or less to do a "security analysis". The goal of this evaluation is to measure to which extent open source static analysis-tools can find vulnerabilities in C language based code. Therefore we investigate which kind of bugs exist as vulnerabilities in C language based software found in a major linux distribution. Then we evaluate several open source static analysis tools for C on a major benchmark data set which includes the vulnerabilities we identified before. It comes out that there is a great variety in the precision and recall of the tools. We show that pure "linters" miss almost all of the vulnerabilities, however sound tools like Frama-C perform best when it comes down to finding difficult-to-spot numeric errors and buffer overflows.