Static Analysis Technologies Evaluation Criteria

Download this Document in PDF Format

List of Static Code Analysis Technologies

Download An Evaluation Worksheet


Spanish Translation 

Russian Translation 


Table of Contents:




Static code analysis is the analysis of software source or binary code. It aims at automating code analysis to find as many common software security weaknesses as possible. There are several open source and commercial static code analysis tools and services available in the market for organizations to choose from. 


Static code analysis is rapidly becoming an essential part of most software organizations' application security assurance program. Mainly because of their ability to analyze large amounts of source code in considerably shorter amount of time than a human could, uncover potential weaknesses, in addition to the ability to automate security knowledge and workflows.


The goal of the SATEC project is to create a vendor-neutral set of criteria to help guide application security professionals during the process of acquiring a static code analysis technology that is intended to be used during source-code driven security programs. This document provides a comprehensive list of criteria that should be considered during the evaluation process. Different users will place varying levels of importance on each feature, and the SATEC project provides the user with the flexibility to take this comprehensive list of potential criteria, narrow it down to a shorter list which contains the most important or most relevant set of criteria, assign weights to each criterion, and conduct a formal evaluation to determine which scanning solution best meets the user's needs.


The aim of this document is not to define a list of requirements that all static code analysis vendors must provide in order to be considered a "complete" solution. In addition, evaluating specific products and providing the results of such an evaluation is outside the scope of the SATEC project.  Instead, this project provides criteria and documentation to enable anyone to evaluate static code analysis tools and services and choose the product that best fits their needs. NIST Special Publication 500-283, "Source Code Security Analysis Tools Functional Specification Version 1.1", contains minimal functional specifications for static code analysis tools. This document can be found at



Target Audience: 

The target audience of this document is the technical staff of software organizations who are looking to automate parts of their application security assurance programs using one or more static code analysis technologies, as well as application security professionals who are responsible for performing application security reviews. The document will take into consideration those who would be evaluating the technology and those who would actually be using it. 



The purpose of this document is to develop a set of criteria that should be taken into consideration while evaluating static code analysis tools or services for security testing. The vendor-neutral criteria defined in this document are selected using a consensus-driven review process comprised of volunteer subject matter experts. Every organization is unique and has a unique software development environment, this document aims to help organizations achieve their application security goals through acquiring the most suitable tool for their own unique environment. The document will strictly stay away from evaluating or rating vendors. However, it will focus on the most important aspects of static code analysis technologies that would help the target audience identify the best technology for their environment and development needs. 




Aaron Weaver (Pearson Education)

Abraham Kang (HP Fortify)

Alec Shcherbakov (AsTech Consulting)

Alen Zukich  (Klocwork)

Arthur Hicken (Parasoft)

Amit Finegold (Checkmarx)

Benoit Guerette (NorthSec)

Chris Eng (Veracode)

Chris Wysopal (Veracode)

Dan Cornell (Denim Group)

Daniel Medianero (Buguroo Offensive Security)

Dinis Cruz (SecurityInnovation)

Gamze Yurttutan

Herman Stevens

Janos Drencsan

James McGovern (HP)

Jean-Marc Atchison (Centauri Technologies))

Jim Manico (WhiteHat Security)

Joe Hemler (Gotham Digital Science)

Jojo Maalouf (Hydro Ottawa)

Laurent Levi  (Checkmarx)

Mushtaq Ahmed (Emirates Airlines)

Ory Segal (Akamai)

Philippe Arteau 

Sherif Koussa (Software Secured) [Project Leader]

Srikanth Ramu (University of British Columbia)

Romain Gaucher  (Coverity)

Sneha  Phadke (eBay)

Wagner Elias (Conviso)




Participation in the Static Analysis Technologies Evaluation Criteria is open to all.  If you have any questions about the evaluation criteria, please contact Sherif Koussa ( sherif dot koussa at gmail dot com)




1. Deployment: 

Static code analysis technologies often represent a significant investment by software organizations looking to automate parts of their application security assurance programs. Not only do these technologies represent a monetary investment, but  they demand time and effort by staff members to setup, operate, and maintain them. In addition, staff members are required to check and act upon the results generated by the technology. Understanding the ideal deployment environment will maximize the derived value, help the organization uncover more potential security flaws and could avoid unplanned hardware purchase cost. The following factors are essential to understanding the technology's capabilities and hence ensuring its proper utilization. 



1.1 Deployment Model:

Vendors deliver static code analysis technologies through one or both of the following models:

This document will refer to Desktop-based static code analysis technologies as "tools" and will refer to SaaS-based static code analysis technologies as "services". The document could use the term "technology" to reference both desktop-based tools and SaaS-based services.


1.2 Tool Installation Support: 

A static code tool should provide the following :


1.3 Deployment Architecture: 

Vendors provide various deployment options. Clear description of the different deployment options must be provided by the vendor to better utilize the tool within an organization. In addition, the vendor must specify the optimal operating conditions. At a minimum the vendor should be able to provide:


1.4 Setup and Runtime Dependencies: 

The vendor should be able to state whether the tool or the service uses a compilation based analysis or source code based analysis.


2. Technology Support: 

Most organizations use more than one programming language internally within their applications portfolio. In addition, more software frameworks are becoming mature enough for development teams to leverage and use across the board as well as a score of 3rd party libraries which are used both on the server and client side. Once these technologies, frameworks and libraries are integrated into an application, they become part of it and the application inherits any vulnerability within these components. 


2.1 Standard Languages Support: 

Most of the technologies available today support more than one programming language. However, an organization looking to use a static code analysis tool or service should make an inventory of all the programming languages, and their versions, used within the organizations as well as third party applications that will be scanned as well. After shortlisting all the programming languages and their versions, an organization should compare the list against the vendor's supported list of programming languages and versions. Vendors provide several levels of support for the same language, understanding what level of support the vendor provides for each programming language, and their versions, in addition to the frameworks used and their versions, is key to understanding the coverage and depth provide by the technology for each language. One way of understanding the level of support for a particular language is to inspect the tool's signatures (AKA Rules or Checkers) for that language.  


2.2 Programming Environment Support: 

Once an application is built on a top of a framework, the application inherits any vulnerability in that framework. In addition, depending on how the application leverages a framework or a library, it can add new attack vectors. Understanding the relationship between the application and the frameworks/libraries is key in order to detect vulnerabilities resulting from the application's usage of the framework or the library, and the following in particular:


2.3 Technology Configuration Support: 

Several tweaks provided by the tool could potentially uncover serious weaknesses. 



3. Scan, Command and Control Support: 

The scan, command and control of static code analysis tools has a significant influence on the user’s ability to configure, customize and integrate the tool into the organization's Software Development Lifecycle (SDLC). In addition, it affects both the speed and effectiveness of processing findings and remediating them.


3.1 Command line support: 

The user should be able to perform scans using the command line which is a desirable feature for many reasons, e.g. avoiding unnecessary IDE licensing, build system integration, custom build script integration, etc. For SaaS based services, the vendor should be able to indicate whether there are APIs to initiate the scan remotely, this becomes a desirable feature for scenarios involving large number of applications.


3.2 IDE integration support: 

The vendor should be able to enumerate which IDEs (and versions) are being supported by the technology being evaluated, as well as what scanning using the IDE will incorporate (e.g. what exactly will get scanned). For example, does an Eclipse plugin scan JavaScript files and configuration files, or does it only scan Java and JSP files.


3.3 Build systems support:

Organizations usually utilize static analysis technologies differently. Some organizations scan the code as part of their daily build system. The vendor should be able to enumerate the build systems supported and their versions (Ant, Make, Maven, etc). In addition, the vendor should be able to describe what gets scanned exactly in this context.


3.4 Customization: 

The tool usually comes with a set of signatures (AKA as rules or checkers), this set is usually followed by the tool to uncover the different weaknesses in the source code. Static code analysis tools should offer a way to extend these signatures in order to customize the tool's capabilities of detecting new weaknesses, alter the way the tool currently detect weaknesses or stop the tool from detecting a specific pattern. The tool should allow users to:


3.5 Scan configuration capabilities: 

This includes the following capabilities:



3.6 Testing Capabilities: 

Scanning an application for weaknesses is an important functionality of the tool or service. It is essential for the tools and services to be able to understand, accurately identify and report the following attacks and security weaknesses.


3.7 Industry Standards Aided Analysis: 

Providing industry-standard-based scanning becomes a desirable feature for many reasons. For example, OWASP Top 10CWE/SANS Top 25WASC Threat ClassificationDISA/STIG etc provide organizations with starting points to their software security gap analysis and in other cases these classifications become metrics of minimum adherence to security standards. Providing industry standard-aided scanning becomes a desirable feature for many reasons. 


4. Product Signature Update:   

Product signatures (AKA rules or checkers) are what the static code analysis tools use to identify security weaknesses. When making a choice of a static analysis tool, one should take into consideration the following:


4.1 Frequency of signature update: 

Providing frequent signature update to a static code analysis tool ensure the tool's relevance to threat landscape. Hence, it is important to understand the following about a tool's signature update:


4.2 User signature feedback: 

The tool must provide a way for users to submit feedback on bugs, flawed rules, rule enhancement, etc.


5. Triage and Remediation Support: 

A crucial factor in a static code analysis tool or service is the support provided in the triage process and the accuracy, effectiveness of the remediation advice. This is vital to the speed in which  findings are assessed and remediated by the development team.


5.1 Findings Meta-Data:  

Finding's meta-data is the information provided by the tool or service around the finding. Good finding meta-data helps the auditor or the developer to understand the weakness and act upon it quicker. The tool or service should provide the following a long with each finding:


5.2 Meta-Data Management: 


5.3 Remediation Support:



6. Reporting Capabilities: 

The tool or service reporting capability is one of its most visible functionalities to stakeholders. The tool or service should provide different ways to represent the results based on the target audience. For example, developers will need as much details as possible in order to be able to remediate the weakness properly in a timely fashion. However, upper management might need to focus on the report's high level summary, or the risk involved more so than the details of every weakness. 


6.1 Support for Role-based Reports:  

The tool or service should be able to provide the following types of reports with the ability to mix and match:



6.2 Report Customization: 

The tool or service should be able to support report customization. The tool or service should be able to provide the following:


6.3 Report Formats: 

The vendor should be able to enumerate the report formats they support (PDF, XML, HTML, etc)



7. Enterprise Level Support:  

When making a choice on a static analysis tool or service in the Enterprise, one should take into consideration the ability to integrate the tool or service into various enterprise systems, such as bug tracking, reporting, risk management and data mining.


7.1 Integration into Bug Tracking Systems: 

Vendors should be able to enumerate the supported bug tracking applications, in particular the following criteria should be defined by the vendor:


7.2   Integration into Enterprise Level Risk Management Systems: 

Information security teams and organizations need to present an accurate view of the risk posture of their applications and systems at all times. Hence, the tool or service should provide integration into enterprise level risk management systems. The vendor should be able to provide


7.3   Ability to Aggregate Projects: 

This pertains to the ability to add meta-data to a new scan. This data could be used to aggregate and classify projects, which could be used to drive intelligence to management. For example, this can help in identifying programming languages that seem to generate more findings thus better utilizing training budge for instance.

Projects are built using a certain set of technologies and/or frameworks. These can be commercial, open source or built in-house. Certain projects may tend to have more security flaws as compared to others based on a technology or framework used or based on the how the technology\framework is used within a given business context. Static code analysis tools and services could be used to configure similar projects with additional metadata to detect these patterns. This will build intelligence around them which could help then detect which application components have more security weaknesses and why.


7.4   Licensing Scheme: 

Static Code Analysis tools and services varies in their licensing schemes. Usually, the following factors decide on the technology's total cost of ownership.


Index A: Static Code Analysis Preparation Cheat Sheet: 

Taking a decision regarding the best static code analysis tool or service to acquire could be a daunting task. However, preparation  for such a task could be very helpful. Every technology is unique so as your corporate environment. The following is a set of information you need to gather which could make the decision much easier to take:


Index B: References