Posted on by Achim D. Brucker, licensed under CC BY-ND 4.0.

Security Testing for DevOps: Requirements

The DevOps model promises to allow software companies to significantly faster (i.e., more frequently) shipping updates to their customers. A key requirement for this is a high degree of test automation: This does not only apply to testing functional testing, it is at least as important for all security testing activities – which are still often done manually or semi-automated.

In a traditional Secure Development Lifecycle (SDL), as, e.g., used by Microsoft or SAP, security testing is usually applied in two phases:

  1. During the implementation phase: Usually, static code analyser (SAST) are used for supporting developers in applying defensive programming techniques [1].
  2. When the product is prepared for shipment (e.g., as part of the security validation in case of SAP): (manual) penetration tests are used to ensure that the final product does not contain vulnerabilities.

This separation is usually not adequate for the fast release cycles of the DevOps strategy. Thus, we need to evaluate additional security testing strategies as well as investigate how tools that have been shown to be successful in more traditional development models can be adapted to fit the needs of a DevOps workflow.

The DevOps Workflow
The DevOps Workflow

A successful integration of security testing tools and stragegies (see, e.g., [2] and [3] for an overview of security testing techniques and strategies) into a DevOps workflow (to achieve SecDevOps or DevSecOps) requires that the security experts listen to the requirements of the development and operations experts and adapt the security testing tools to their needs. This affects various phases of an agile SDLC:

  • Development: To avoid vulnerabilities right from the beginning, developers should have access (and use) tools that support them to avoid insecure programming patterns. Such (usually static) tools should be integrated into the development environment (IDE) and provide instant feedback – similar to the spelling and grammar corrections in modern word processors.
  • Build: On the build (and commit/repository) servers, two strategies should be implemented:
    • Commit checks: To allow for immediate feedback for security tests that are not suitable for “instant IDE feedback” (e.g., because they take too much time) as well as for checking that the issues reported by the IDE integrated tools are fixed, both static as well as dynamic security tests should be run at each code commit.
    • Nighly Build Checks: In particular complex static checks are usually require more time and resources than available “at commit time”. Thus, they should be run as part of the nightly build – together with a daily routine in analysing and fixing them each morning.
  • Milestone Releases: The previously discussed test strategies are target towards developers. Thus, the focus is on tools that are easy to use and require no or only a low security expertise. While such tools usually help to detect the vast majority of vulnerabilities early in the development, they might miss rare but severe issues that are hard to detect (or to analyse). Thus, the regular but not daily (e.g., every three months) milestone releases should be used for applying more thorough security tests (e.g., penetration tests) that require a high-level of security expertise. Issues found during these tests should be fixed (and thus, effort needs to be planned) during the following milestone release cycle.
  • Operations: The fact that the one team is developing and operating an applications enables the seamless extension of security testing, monitoring, and enforcing across the whole application lifecycle:

    • Monitoring and Intrusion-Detection are already used by the operations team. In very rare cases, however, feedback is provided to the development teams. Such an information exchange is vital to leverage the advantages of DevOps (and are a key factor to make your cloud business successful): issues detected by the monitoring and intrusion detection systems should result in improvements of the security testing tools used int he previous phases to reduce the number of findings that occur during operations. Similarly, the need for monitoring tools should already be considered during system design to enable the monitoring with at least runtime effort (and integration effort) as possible.
    • An active defence, as for example, Runtime Application Self Protection (RASP) provides, is much more likely to have success within a SecDevOps strategy: the close integration of development and operations activities makes it possible to introduce RASP tools during the development (i.e., during the execution of the system test) and, as a next step, its seamless transition into the production system. The use of RASP in development allows, on the one hand, to fix found issues early, and, on the other hand, to optimise the developed application so that RASP can be applied without (or only a negligible) a runtime overhead.

Of course, not all security tests are useful or necessary for all products. A risk-based assessment (including the legal and regulatory requirements) that weights the operational risks with the development and operational costs and the expected revenue should form the basis of an application-specific security test plan.

Development Commit Nightly Build Milestone Release Operations
automation high high high low high
acceptable max. runtime < 500ms < 10s < 12h several days < 1ms
acceptable false positives very low very low low high medium
acceptable false negatives high high medium low medium
integration IDE code repository build server not necessary runtime (application server)
required security expertise low low medium high medium
user group developer developer developer (and security experts) security experts developer, sysadmins
example Find Security Bugs BDD Checkmarx, Coverity, HP Fortify, HP WebInspect, Seeker (potentially using a focused test scope/configuration to ensure low false positive rate) Checkmarx, Coverity, HP Fortify, HP WebInspect, Seeker (full/large test scope), manual penetration tests (and tools like OWASP ZAP) HPE Application Defender, Qualys

The table below summarizes the different requirements for security test tools and methods as part of a (Sec)DevOps strategy. Depending at which phase of the development and operation lifecycle security testing is applied, different configuration/combinations of security test tools are required. In particular in early phases of the development, tools need to be easy to use for developers that are security aware they should not require a deep security expertise (thus, they need to have a low false positive rates, fast turn-around times, etc.). Similarly, tools used during operations should not require a deep security expertise during normal operations (thus, they share a lot of their requirements with the tools targeting developers). In contrast, the security tests for milestone releases should be done by security experts and, thus, can target the full range of security testing tools and methods (ranging fro0m manual design reviews, SAST/DAST/IAST, vulnerability scanners, security proxies, or human expert knowledge).

Of course, in addition to the requirements targeting DevOps teams, the well-known requirements for selecting security test tools should be considered. Finally, secure development and operations covers all parts of the development, operations, and maintenance lifecycle, i.e., it starts already with the training (and product design) and also includes a comprehensive response and patch & update process.


DevOps and secure software development are not contradictory. Still, the introduction of security testing tools needs more planing (and more adaption of the tools) than introducing the same or similar tools in traditional development strategies. Following the “Customer First” strategy, security experts should focus on the needs of the software developers: the goal of security testing tools is to support developers to develop secure software, they should not be seen as an obstacle.


1. Brucker, A. D. and Sodan, U. “Deploying Static Application Security Testing on a Large Scalegi Sicherheit 2014 228, (2014): 91–101. URL:

2. Bachmann, R. and Brucker, A. D. “Developing Secure Software: A Holistic Approach to Security TestingDatenschutz und Datensicherheit (DuD) 38, no. 4 (2014): 257–261. doi:10.1007/s11623-014-0102-0, URL:

3. Felderer, M., Büchler, M., Johns, M., Brucker, A. D., Breu, R., and Pretschner, A. “Security Testing: A SurveyAdvances in Computers 101, (2016): 1–51. doi:10.1016/bs.adcom.2015.11.003, URL:

4. Brucker, A. D. “Sicherheitstests Für Secure DevOps (SecDevOps)OBJEKTspektrum (2016): URL:

Welcome to the blog of the Software Assurance & Security Research Team at The University of Sheffield. We blog regularly news, tips & tricks, as well as fun facts about software assurance, reliability, security, testing, verification, hacking, and logic.

You can also follow us on Twitter: @logicalhacking.




academia appsec cordova dast devops devsecops event fixeffort floss hol-ocl hol-testgen iast industry isabelle/hol logic mbst mobile modelling monads ocl opensource owasp research sap sast sdlc secdevops security securityengineering securitytesting staff&positions test&proof testing tips&tricks tools tuos uk verification webinar


blog whole site