Static Application Security Testing (SAST): Code Quality vs Code Security

& A Brief Discussion of Three SAST Industry Leading Tools - by Gregg Zepp
When looking for tools to conduct secure code reviews, be careful not to confuse tools meant for reviewing
code quality (sometimes referred to as coding violations) versus those designed to review for code
vulnerabilities, often also referred to generically as code security. What is the difference? Code Quality
reviews focus on coding styles, how well the code follows its industry standard(s), proper logic, well defined
interfaces, ease of build, code comments, etc. Code Security reviews focus on types of code vulnerabilities,
measure their ease of their exploitation (aka, how likely the vulnerability could be exploited), and score the
potential exploitation results (i.e., critical finding = administrator level access).
AppScan, Checkmarx, Fortify, and VeraCode are the Gartner SAST Leadership Quad tools for 2018.1 The
annual Gartner Static Application Security Testing (SAST) Report is a good place to start if your team needs to
select a tool for SAST.
Figure 1: Magic Quadrant for Application Security Testing 2

These tools look directly at an application’s source code, aka, inside the application (or white box testing).
They are not to be confused with those used for Dynamic Application Security Testing (DAST). DAST tools
examine an application from the outside (or black box testing). Think of SAST tools as a code level reviews

1
Gartner SAST Tools report 2018: https://www.gartner.com/doc/3868966/magic-quadrant-application-security-testing
2
Gartner SAST Tools report 2018: https://www.gartner.com/doc/3868966/magic-quadrant-application-security-testing
2018 © Gregg Zepp, All Rights Reserved
which could reveal up to project architectural issues while DAST tools look across an application’s external
surface which could lead down to revealing some code level issues.
Often referenced is the NIST “Source Code Security Analyzers” website:
https://samate.nist.gov/index.php/Source_Code_Security_Analyzers.html, which has been around for a
number of years. Among the tools listed on the site are code security analyzers; however, many of the tools
listed measure code quality and other measurable source coding aspects (i.e., code complexity, etc).
Unfortunately, having the word “Security” in the page’s title can be misleading often sending development
teams down the wrong path of picking several tools to review for analyzing code security when, in fact, they
are measuring more code quality issues instead of vulnerability issues.
Fortify remains a popular tool, and IBM AppScan, albeit a SAST tool too, has been seen in the industry as being
better for reviewing higher-level code architecture issues, while Fortify is effective at analyzing deeper, down-
to-the-line coding issues. Albeit, HPE’s Fortify was purchased in 2017 by the UK’s Micro Focus software
company (making it UK’s largest software company) 3 it continues to evolve along the similar cycles as when it
was owned by HPE. Some Fortify product team members have been with the team since before Hewlett
Packard purchased it and remain with it today. A few of these same folks assisted with putting together some
of the benchmarks found at: https://samate.nist.gov/SRD/testsuite.php 4 confirming that these benchmarks
are indeed for code quality reviews – not code security.
Unfortunately, I’ve seen development teams too many times mistakenly leverage the NIST Samate code sets
for examining a set of SAST tools thinking they are ideal to test a set of security tools. Albeit, the benchmarks
have some vulnerabilities among them; however, they should be leveraged as benchmarks to review tools
which assess code quality issues, such as code styles to industry standards, code logic, building, and other
quality issues. A number of tools measure these quality metrics, examples include Clang-Tidy, CPPCheck,
Bandit, Brakeman, PyCharm, and Pylint to name a few.
Rarely discussed is what the tools are not finding (aka, false negatives). To lessen the impact of false negatives,
ensure to leverage a code benchmark which has a lot of various types of vulnerabilities. How many types of
vulnerabilities? As many different types as possible. An excellent benchmark for learning about various types
of code vulnerabilities scan be found at: https://vulncat.hpefod.com (click “advanced” then choose the
language to filter the finding categories based on a language). 5 This site filters for a number of vulnerabilities
across most of the major languages, as well as, a “Universal” setting for a general set of coding vulnerabilities.
What are some criteria to consider when putting together a methodology to test and select a SAST tool?
Below are some points to consider…

 Often times, a publicly available code benchmark isn’t as good a representative of code styles or
various languages the team would be actually be reviewing. In other words, if you’re going to pick a
publicly available code benchmark, ensure it is representative of the types of project code(s) you
would be scanning.
 Whenever it is possible, SAST tools should be tested scanning actual project code versus publicly
available benchmarks and if possible, using the same code set(s) to test all tools being considered.

3
HP Enterprise strikes $8.8 billion deal with Micro Focus for software assets: https://www.reuters.com/article/us-hpenterprise-software-
microfocus-idUSKCN11D2EU
4
Micro Focus vendor presentation
5
Fortify Taxonomy: Software Security Errors: https://vulncat.hpefod.com/en
2018 © Gregg Zepp, All Rights Reserved
  SAST tool analysis should review code with as many security issues as possible. For example, the
Toyota ITC benchmark (a C++ benchmark from the NIST Samate benchmark website) has 51 categories
of findings, and a number of these findings are code quality related (not security nor vulnerability
related). While the long running website for code vulnerability assessments
(https://vulncat.hpefod.com/en/weakness), when filtered for C++ vulnerabilities, has 110 security
categories.
Leadership of development and cybersecurity teams should be vigilant that when they task their teams with
transforming their DevOps process to one of DevSecOps, much too often, the developers or even the
development team leads will either knowingly or unknowingly undermine the security tools integration
process by:

 Tainting the security tools selection process by reviewing tools meant to check for code quality issues
instead of selecting tools engineered to review code vulnerabilities and security.
 Tainting the security tools selection process by leveraging code benchmarks with code quality issues
and few, if any, code vulnerability issues.
 Recommending or selecting a tool not meant for code quality reviews yet still referring to these
reviews the tool or tool set performs as though its checking for code vulnerabilities - when it’s really
checking for code quality issues (more likely among smaller development teams).
On a side note, one tool from a common set of code quality tools, ParaSoft, does has a stronger security slant
to it than other code quality review tools. However, it does not display information as to why a finding is
vulnerable, how it can be exploited, or how to mitigate a finding in its stand-alone product. ParaSoft’s tool
leverages information they publish to the internet. If it does not have internet access it cannot show such
information. In the end, coding standards and quality measuring tools should be part of a separate Code
Quality Assessment report and not a SAST Tool Assessment.
Three leading SAST industry tools, Checkmarx, AppScan, and Fortify, all have their findings’ information built
in, including as why a finding is vulnerable, how it can be exploited, potential result of a successful
exploitation, and mitigation recommendations making them better out of the box solutions for all types of
environments, especially for stand-alone environments.
Other notables from vendor presentation sessions:

 Polyspace is largely a code quality tool with limited security analysis. Also, its limited in how well one
can drill down on findings, its reporting, functionality, etc. 6 Moreover, it scans only C, C++, and Ada
programming languages. 7
 Checkmarx has stated a number of their customers had expressed a desire for improvement in their
tool’s ability to effectively analyze C\C++ code vulnerabilities. As a result, they were working on
improving this with anticipation of the upgraded capability being available perhaps later this year.
Moreover, their licensing tracks the number of projects its scans, which could be a potentially limiting
situation given some “applications” have multiple compiled project or solution files, something they
themselves admit could present licensing challenges.8

6
Polyspace vendor presentation
7
Wikipedia: Polyspace: https://en.wikipedia.org/wiki/Polyspace
8
Checkmarx vendor presentation
2018 © Gregg Zepp, All Rights Reserved
  IBM AppScan Standard SCA does not support Python and there are no plans to change this as they
focus more on their Enterprise (aka, Cloud) offering. Moreover, IDE plugins such as those for Eclipse
and Visual Studio are each an additional cost over and above the cost for the product itself. 9
 Fortify supports the most languages, APIs, IDEs, and the plugins for a number of IDEs comes with their
license purchase. Having a history of supporting and running in federal environments, it’s the only SAST
tool with PIV Card support too. 10 11 12

Of the leading SAST Tools, the following charts detail the total number of and types of languages and IDE
integrations Checkmarx, AppScan, and Fortify support.
13 14 15
Table 1: Code Languages Supported

Languages: IBM AppScan (17) Checkmarx (21) Fortify (27)

Java Yes Yes Yes
JSP Yes Yes Yes
.Net Yes - All .Net, C#, VB.net Yes - C#, VB.net Yes
ASP Yes Yes Yes
VB Yes Yes Yes
C\C++ Yes Yes - but weak (2018 upgrades planned) Yes
Cobol Yes No Yes
PHP 5.5, 5.6, and 7.0 Yes 5.3, 5.4, 5.5, 5.6, 7.0, 7.1
Ruby No Yes Yes
Javascript Yes Yes Yes
VBScript Yes Yes Yes
Perl Yes Yes No
Android Yes Yes Yes
iOS Yes Yes Yes
HTML5 Yes Yes Yes
SQL Yes Yes Yes
Ajax ASP.Net only Javascript only Yes
Python No Yes Yes
Rudy No Yes Yes - 1.9.3
Coldfusion Yes No Yes
others No Yes (i.e., Scala) Yes (i.e., Scala)
16 17 18
Table 2: IDEs Supported

9
IBM Appscan vendor presentation
10
Micro Focus Fortify presentation
11
Micro Focus Fortify: Overview: https://software.microfocus.com/en-us/products/static-code-analysis-sast/overview
12
Micro Focus Fortify: v17.20: https://community.softwaregrp.com/dcvta86296/attachments/dcvta86296/fortify-software-
1720/5/1/HPE_Whats_New_17.20.pdf
13
Checkmarx: Language support:
https://checkmarx.atlassian.net/wiki/spaces/KC/pages/141328390/8.5.0+Supported+Code+Languages+and+Frameworks
14
IBM Appscan: Language support:
https://www.ibm.com/support/knowledgecenter/en/SSS9LM_9.0.3/com.ibm.rational.appscansrc.install.doc/topics/system_requirements_languag
e_support.html
15
Micro Focus Fortify: 17.20 What’s New: https://community.softwaregrp.com/dcvta86296/attachments/dcvta86296/fortify-software-
1720/5/1/HPE_Whats_New_17.20.pdf
16
Checkmarx: IDE support: https://checkmarx.atlassian.net/wiki/spaces/KC/pages/4259843/IDE+Plugins
2018 © Gregg Zepp, All Rights Reserved
 Platforms: IBM AppScan Checkmarx Fortify
* Eclipse versions 4.6, 4.7 *
* Eclipse versions 3.6, 3.7, 3.8, 4.2, 4.2.x, 4.3, 4.3.1, 4.3.2 and 4.4
* Eclipse versions 3.6, 3.7, 3.8, 4.2, 4.2.x, 4.3, 4.3.1, 4.3.2, 4.4, up to * Visual Studio 2013, 2015, 2017 *
* IBM Rational Application Developer (RAD) V8.0.x, V8.5, V8.5.1,
4.7 * IntelliJ 15, 2016.x, 2017.x *
Windows V8.5.5, V9.0, V9.0.1 and V9.1
* Visual Studio 2010, 2012, 2013, 2015, 2017 * Android Studio 2.3.x
* Visual Studio 2010, 2012, 2013
* IntelliJ 11-16 * WebStorm
* Oracle Jdeveloper 12c
* Eclipse versions 3.6, 3.7, 3.8, 4.2, 4.2.x, 4.3, 4.3.1, 4.3.2 and 4.4
* Eclipse versions 4.6, 4.7 *
Linux * IBM Rational Application Developer (RAD) V8.0.x, V8.5, V8.5.1, None
* IntelliJ 15, 2016.x, 2017.x *
V8.5.5, V9.0, V9.0.1 and V9.1

* Eclipse versions 3.6, 3.7, 3.8, 4.2, 4.2.x, 4.3, 4.3.1, 4.3.2 and 4.4 * Eclipse versions 4.6, 4.7 *
Mac * IBM Rational Application Developer (RAD) V9.0, V9.0.1 and V9.1
None
* IntelliJ 15, 2016.x, 2017.x *

Solaris Limited Limited Limited

HP-UX\AIX Limited Limited Limited

* older versions support available in prior Fortify versions which are

downloadable with the purchase of a 2018 license

Conclusion
With CA Technologies recently purchasing VeraCode, it’s future status remains to be determined. AppScan and
Fortify have a history supporting multiple federal agencies and large commercial customers making Fortify’s
market share beyond that of AppScan’s. All three are priced to compete with each other with their first-year
licenses ranging from $50-$60k. However, given the broadest technical support across languages, IDEs, and
environments is offered by Fortify along with a more flexible application scanning licensing model, Micro
Focus’s Fortify is recommended for purchase over the other two. Moreover, Fortify incorporates data from
resources such as Bugzilla resulting in many of its Low and Informational findings to be related to code quality,
and as a result, recommend purchasing Fortify first, then deciding later if a complimentary code quality or bug
finding tool is needed.

17
IBM Appscan: IDE support:
https://www.ibm.com/support/knowledgecenter/en/SSS9LM_9.0.2/com.ibm.rational.appscansrc.install.doc/topics/install_developer_eclipse.html
18
Micro Focus Fortify: IDE support:
https://marketplace.microfocus.com/fortify/category/plugins?product=All%20products&version=All%20versions&company=All%20companies&su
bcat
2018 © Gregg Zepp, All Rights Reserved
List of Resources

Checkmarx (Atlassian)
 Overview: https://checkmarx.atlassian.net/wiki/spaces/KC/pages/59211846/Checkmarx+CxSAST+Overview
 Plugins: https://checkmarx.atlassian.net/wiki/spaces/KC/pages/4259843/IDE+Plugins

IBM AppScan
 Standard – Overview: https://www.ibm.com/us-en/marketplace/appscan-standard
 System requirements: https://www-01.ibm.com/support/docview.wss?uid=swg27024155
 Plugins:
https://www.ibm.com/support/knowledgecenter/en/SSS9LM_9.0.2/com.ibm.rational.appscansrc.install.doc/topics/install_
developer_eclipse.html
 Language support:
https://www.ibm.com/support/knowledgecenter/en/SSS9LM_9.0.3/com.ibm.rational.appscansrc.install.doc/topics/system
_requirements_language_support.html

Micro Focus Fortify SCA

 Overview: https://www.microfocus.com/documentation/fortify-static-code/1720/#
 17.20 List of Documents: https://www.microfocus.com/documentation/fortify-static-code-analyzer-and-tools/1720/
 Installation: https://www.microfocus.com/documentation/fortify-static-code-analyzer-and-
tools/1720/HPE_SCA_Install_Help_17.20/index.htm
 Plugins:
https://marketplace.microfocus.com/fortify/category/plugins?product=All%20products&version=All%20versions&company
=All%20companies&subcat
 Build Servers: https://marketplace.microfocus.com/fortify/category/build-
servers?product=All%20products&version=All%20versions&company=All%20companies&subcat

2018 © Gregg Zepp, All Rights Reserved