Many vendors of application security testing tools are classifying their tools based on the used testing techniques, e.g., static application security testing (SAST), dynamic application security testing (DAST) or, more recently, interactive application security testing (IAST). Is this really the information users of security testing tools actually need?

Most likely not. Customers are much more interested to get the following questions answered:

1. What expertise to I need to use the tool?
2. What type of security vulnerabilities can be detected?
3. What type of software (frameworks, technologies, programming languages) can I test?

Answering these questions gives a nice, three-dimensional, framework for categorizing security testing tools:

In more detail, the three axis are:

1. Target User Group (x-axis): Does the tool target developers that do not (and should not need!) have a deep security expertise or does the tool require a high level of security expertise. Security tools for developers need to encapsulate the security knowledge and must the developer-friendly (this includes a low false positive rate as well as specific and actionable fix instructions). In contrast, a security testing tool for security experts can focus on a low false negative rate and/or reporting and trending tools that help in the overall risk assessment.
2. Vulnerability Scope (y-axis): How many vulnerability types can (potentially) be detected by the tool. This is often expressed in the number of covered Common Weakness Enumeration (CWE) categories.
3. Technology Scope (z-axis): How many different technologies are supported by the tool. For example, in case of static tools (SAST), this might be expressed in the number of programming languages (and frameworks/libraries) that can be analyzed; in case of dynamic tools (DAST), this might be the number of protocols (or interaction models) that are supported.

Actually, if we merge the “vulnerability scope” and “technology scope” axis, we end up with a nice magic quadrant for security testing tools.

The magic quadrant has the two axis:

1. Target User Group (x-axis): This is the same as in the three-dimensional case.
2. Security and Technology Scope (y-axis): This axis combines the coverage of technologies and the vulnerabilities scope.

This two axis capture, from a central software security team of a software vendor, a very important insight:

• Tools in the right upper quadrant are tools (the “generalists for developers”) that are candidates for being purchased centrally (and centrally supported), as they serve a large developer base and require a cooperatively low security skills level. Examples for tools in this quadrant are Checkmarx’s SAST solution), Coverity or HP Fortify. Also, dynamic tools such as might, depending on the maturity and the focus of a software security programs (as they, usually, do not provide detailed fix recommendations, we discuss them in the next category), be put in this category. Besides the mentioned “on-premise” tools, there are also cloud offerings available that fit into this category, e.g., from Veracode.

• Tools in the left upper quadrant are tools (the “generalists for security experts”) that are candidates for being purchased by a central security team: they cover a wide range of technologies and security aspects but requires a higher security expertise than the developer-friendly tools in the right upper quadrant. This might include traditional vulnerability scanners (such as Metasploit but depending on the maturity of the security training for developers might also include dynamic tools such as Coverity Seeker or HP WebInspect.

• Tools in the right lower quadrant (the “specialists for developers” are usually specialist tools for developers, e.g., they might require a high level of development expertise but only a low to medium level of security expertise (e.g., they only cover one CWE that is specific to the use case of the developed product). One could, for example, argue that BDD-Security is a candidate for this category.

• Tools in the lower left quadrant (the “specialists for security experts”) are specialist tools that serve individual experts and team. Thus, they are candidates for a local purchase by the actual development team using them. Examples for tools in this quadrant are Burp Suite, DOM Inspector, OWASP ZAP, or sslyze.

Of course, such a classification is never “black-and-white-only” and depends not only on the actual properties of the tool but also on the type of application security program in your organization as well as the level of maturity. In general, if you are just starting to introduce application security testing tools to your developers, focus on developer-friendly tools that support a wide range of vulnerabilities and technologies (i.e., the right-upper quadrant).

Concluding, if you are talking to application security testing tool vendors, prioritize your needs (the type of vulnerabilities you want to detect, the type of technologies you need to support, etc.) over actual technology used for finding them. Of course, the different application security testing techniques differ in much more aspects: this article is not a comprehensive buyer’s guide - it only should give some ideas how to decide if a tool is more suitable to be bought by a local development team or if it is more suitable to be bought be a central organization supporting a large group of developers.

And last but not least, the same gap between what users need and what vendors (respectively, their marketing department) offers also exists for other software security technologies such as Runtime Application Self-Protection (RASP).