Posted on by Achim D. Brucker, licensed under CC BY-ND 4.0.

Security Testing for DevOps: Requirements

The DevOps model promises to allow software companies to significantly faster (i.e., more frequently) shipping updates to their customers. A key requirement for this is a high degree of test automation: This does not only apply to testing functional testing, it is at least as important for all security testing activities – which are still often done manually or semi-automated.

In a traditional Secure Development Lifecycle (SDL), as, e.g., used by Microsoft or SAP, security testing is usually applied in two phases:

  1. During the implementation phase: Usually, static code analyzer (SAST) are used for supporting developers in applying defensive programming techniques [1].
  2. When the product is prepared for shipment (e.g., as part of the security validation in case of SAP): (manual) penetration tests are used to ensure that the final product does not contain vulnerabilities.

This separation is usually not adequate for the fast release cycles of the DevOps strategy. Thus, we need to evaluate additional security testing strategies as well as investigate how tools that have been shown to be successful in more traditional development models can be adapted to fit the needs of a DevOps workflow.

The DevOps Workflow

A successful integration of security testing tools and strategies (see, e.g., [2] and [3] for an overview of security testing techniques and strategies) into a DevOps workflow (to achieve SecDevOps or DevSecOps) requires that the security experts listen to the requirements of the development and operations experts and adapt the security testing tools to their needs. This affects various phases of an agile SDLC:

  • Development: To avoid vulnerabilities right from the beginning, developers should have access (and use) tools that support them to avoid insecure programming patterns. Such (usually static) tools should be integrated into the development environment (IDE) and provide instant feedback – similar to the spelling and grammar corrections in modern word processors.

  • Build: On the build (and commit/repository) servers, two strategies should be implemented:

    • Commit checks: To allow for immediate feedback for security tests that are not suitable for “instant IDE feedback” (e.g., because they take too much time) as well as for checking that the issues reported by the IDE integrated tools are fixed, both static and dynamic security tests should be run at each code commit.
    • Nighly Build Checks: In particular complex static checks are usually require more time and resources than available “at commit time”. Thus, they should be run as part of the nightly build – together with a daily routine in analysing and fixing them each morning.
  • Milestone Releases: The previously discussed test strategies are target towards developers. Thus, the focus is on tools that are easy to use and require no or only a low security expertise. While such tools usually help to detect the vast majority of vulnerabilities early in the development, they might miss rare but severe issues that are hard to detect (or to analyze). Thus, the regular but not daily (e.g., every three months) milestone releases should be used for applying more thorough security tests (e.g., penetration tests) that require a high-level of security expertise. Issues found during these tests should be fixed (and thus, effort needs to be planned) during the following milestone release cycle.

  • Operations: The fact that the one team is developing and operating an application enables the seamless extension of security testing, monitoring, and enforcing across the whole application lifecycle:

    • Monitoring and Intrusion-Detection are already used by the operations team. In very rare cases, however, feedback is provided to the development teams. Such an information exchange is vital to leverage the advantages of DevOps (and are a key factor to make your cloud business successful): issues detected by the monitoring and intrusion detection systems should result in improvements of the security testing tools used in the previous phases to reduce the number of findings that occur during operations. Similarly, the need for monitoring tools should already be considered during system design to enable the monitoring with at least runtime effort (and integration effort) as possible.
    • An active defense, as for example, Runtime Application Self Protection (RASP) provides, is much more likely to have success within a SecDevOps strategy: the close integration of development and operations activities makes it possible to introduce RASP tools during the development (i.e., during the execution of the system test) and, as a next step, its seamless transition into the production system. The use of RASP in development allows, on the one hand, to fix found issues early, and, on the other hand, to optimize the developed application so that RASP can be applied without (or only a negligible) a runtime overhead.

Of course, not all security tests are useful or necessary for all products. A risk-based assessment (including the legal and regulatory requirements) that weights the operational risks with the development and operational costs and the expected revenue should form the basis of an application-specific security test plan.

Development Commit Nightly Build Milestone Release Operations
automation high high high low high
acceptable max. runtime < 500ms < 10s < 12h several days < 1ms
acceptable false positives very low very low low high medium
acceptable false negatives high high medium low medium
integration IDE code repository build server not necessary runtime (application server)
required security expertise low low medium high medium
user group developer developer developer (and security experts) security experts developer, sysadmins
example Find Security Bugs BDD Checkmarx, Coverity, HP Fortify, HP WebInspect, Seeker (potentially using a focused test scope/configuration to ensure low false positive rate) Checkmarx, Coverity, HP Fortify, HP WebInspect, Seeker (full/large test scope), manual penetration tests (and tools like OWASP ZAP) HPE Application Defender, Qualys

The table below summarizes the different requirements for security test tools and methods as part of a (Sec)DevOps strategy. Depending on which phase of the development and operation lifecycle security testing is applied, different configuration/combinations of security test tools are required. In particular in early phases of the development, tools need to be easy to use for developers that are security aware they should not require a deep security expertise (thus, they need to have a low false positive rates, fast turn-around times, etc.). Similarly, tools used during operations should not require a deep security expertise during normal operations (thus, they share a lot of their requirements with the tools targeting developers). In contrast, the security tests for milestone releases should be done by security experts and, thus, can target the full range of security testing tools and methods (ranging fro0m manual design reviews, SAST/DAST/IAST, vulnerability scanners, security proxies, or human expert knowledge).

Of course, in addition to the requirements targeting DevOps teams, the well-known requirements for selecting security test tools should be considered. Finally, secure development and operations covers all parts of the development, operations, and maintenance lifecycle, i.e., it starts already with the training (and product design) and also includes a comprehensive response and patch & update process.


DevOps and secure software development are not contradictory. Still, the introduction of security testing tools needs more planing (and more adaption of the tools) than introducing the same or similar tools in traditional development strategies. Following the “Customer First” strategy, security experts should focus on the needs of the software developers: the goal of security testing tools is to support developers to develop secure software, they should not be seen as an obstacle.


A. D. Brucker and U. Sodan, Deploying static application security testing on a large scale,” in GI sicherheit 2014, Mar. 2014, vol. 228, pp. 91–101.
R. Bachmann and A. D. Brucker, “Developing secure software: A holistic approach to security testing,” Datenschutz und Datensicherheit (DuD), vol. 38, no. 4, pp. 257–261, Apr. 2014, doi: 10.1007/s11623-014-0102-0.
M. Felderer, M. Büchler, M. Johns, A. D. Brucker, R. Breu, and A. Pretschner, “Security testing: A survey,” Advances in Computers, vol. 101, pp. 1–51, Mar. 2016, doi: 10.1016/bs.adcom.2015.11.003.
A. D. Brucker, Sicherheitstests für secure DevOps (SecDevOps),” OBJEKTspektrum, May 2016,

Welcome to the blog of the Software Assurance & Security Research Team at the University of Exeter. We blog regularly news, tips & tricks, as well as fun facts about software assurance, reliability, security, testing, verification, hacking, and logic.

You can also follow us on Twitter: @logicalhacking.




academia ai android apidesign appsec bitcoin blockchain bpmn browser browserextensions browsersecurity bug certification chrome composition cordova dast devops devsecops dom dsbd efsm epsrc event extensions fixeffort floss formaldocument formalmethods funding hol-ocl hol-testgen humanfactor hybridapps iast industry internetofthings iot isabelle/hol isabelledof isadof latex logic maintance malicous mbst mobile mobile apps modelinference modeling monads monitoring msc ocl ontology opensource owasp patches pet phd phdlife phishing policy protocols publishing reliability research safelinks safety sap sast sdlc secdevops secureprogramming security securityengineering securitytesting semantics servicecomposition skills smartcontract smartthings softwareeinginering softwaresecurity softwaresupplychain solidity staff&positions statemachine studentproject tcb test&proof testing tips&tricks tools transport tuos uk uoe upgrade usability verification vulnerabilities vulnerableapplication webinar websecurity


blog whole site