Security Testing for DevOps: Requirements
The DevOps model promises to allow software companies to significantly faster (i.e., more frequently) shipping updates to their customers. A key requirement for this is a high degree of test automation: This does not only apply to testing functional testing, it is at least as important for all security testing activities – which are still often done manually or semi-automated.
In a traditional Secure Development Lifecycle (SDL), as, e.g., used by Microsoft or SAP, security testing is usually applied in two phases:
- During the implementation phase: Usually, static code analyzer (SAST) are used for supporting developers in applying defensive programming techniques [1].
- When the product is prepared for shipment (e.g., as part of the security validation in case of SAP): (manual) penetration tests are used to ensure that the final product does not contain vulnerabilities.
This separation is usually not adequate for the fast release cycles of the DevOps strategy. Thus, we need to evaluate additional security testing strategies as well as investigate how tools that have been shown to be successful in more traditional development models can be adapted to fit the needs of a DevOps workflow.
A successful integration of security testing tools and strategies (see, e.g., [2] and [3] for an overview of security testing techniques and strategies) into a DevOps workflow (to achieve SecDevOps or DevSecOps) requires that the security experts listen to the requirements of the development and operations experts and adapt the security testing tools to their needs. This affects various phases of an agile SDLC:
Development: To avoid vulnerabilities right from the beginning, developers should have access (and use) tools that support them to avoid insecure programming patterns. Such (usually static) tools should be integrated into the development environment (IDE) and provide instant feedback – similar to the spelling and grammar corrections in modern word processors.
Build: On the build (and commit/repository) servers, two strategies should be implemented:
- Commit checks: To allow for immediate feedback for security tests that are not suitable for “instant IDE feedback” (e.g., because they take too much time) as well as for checking that the issues reported by the IDE integrated tools are fixed, both static and dynamic security tests should be run at each code commit.
- Nighly Build Checks: In particular complex static checks are usually require more time and resources than available “at commit time”. Thus, they should be run as part of the nightly build – together with a daily routine in analysing and fixing them each morning.
Milestone Releases: The previously discussed test strategies are target towards developers. Thus, the focus is on tools that are easy to use and require no or only a low security expertise. While such tools usually help to detect the vast majority of vulnerabilities early in the development, they might miss rare but severe issues that are hard to detect (or to analyze). Thus, the regular but not daily (e.g., every three months) milestone releases should be used for applying more thorough security tests (e.g., penetration tests) that require a high-level of security expertise. Issues found during these tests should be fixed (and thus, effort needs to be planned) during the following milestone release cycle.
Operations: The fact that the one team is developing and operating an application enables the seamless extension of security testing, monitoring, and enforcing across the whole application lifecycle:
- Monitoring and Intrusion-Detection are already used by the operations team. In very rare cases, however, feedback is provided to the development teams. Such an information exchange is vital to leverage the advantages of DevOps (and are a key factor to make your cloud business successful): issues detected by the monitoring and intrusion detection systems should result in improvements of the security testing tools used in the previous phases to reduce the number of findings that occur during operations. Similarly, the need for monitoring tools should already be considered during system design to enable the monitoring with at least runtime effort (and integration effort) as possible.
- An active defense, as for example, Runtime Application Self Protection (RASP) provides, is much more likely to have success within a SecDevOps strategy: the close integration of development and operations activities makes it possible to introduce RASP tools during the development (i.e., during the execution of the system test) and, as a next step, its seamless transition into the production system. The use of RASP in development allows, on the one hand, to fix found issues early, and, on the other hand, to optimize the developed application so that RASP can be applied without (or only a negligible) a runtime overhead.
Of course, not all security tests are useful or necessary for all products. A risk-based assessment (including the legal and regulatory requirements) that weights the operational risks with the development and operational costs and the expected revenue should form the basis of an application-specific security test plan.
| Development | Commit | Nightly Build | Milestone Release | Operations | |
|---|---|---|---|---|---|
| automation | high | high | high | low | high |
| acceptable max. runtime | < 500ms | < 10s | < 12h | several days | < 1ms |
| acceptable false positives | very low | very low | low | high | medium |
| acceptable false negatives | high | high | medium | low | medium |
| integration | IDE | code repository | build server | not necessary | runtime (application server) |
| required security expertise | low | low | medium | high | medium |
| user group | developer | developer | developer (and security experts) | security experts | developer, sysadmins |
| example | Find Security Bugs | BDD | Checkmarx, Coverity, HP Fortify, HP WebInspect, Seeker (potentially using a focused test scope/configuration to ensure low false positive rate) | Checkmarx, Coverity, HP Fortify, HP WebInspect, Seeker (full/large test scope), manual penetration tests (and tools like OWASP ZAP) | HPE Application Defender, Qualys |
The table below summarizes the different requirements for security test tools and methods as part of a (Sec)DevOps strategy. Depending on which phase of the development and operation lifecycle security testing is applied, different configuration/combinations of security test tools are required. In particular in early phases of the development, tools need to be easy to use for developers that are security aware they should not require a deep security expertise (thus, they need to have a low false positive rates, fast turn-around times, etc.). Similarly, tools used during operations should not require a deep security expertise during normal operations (thus, they share a lot of their requirements with the tools targeting developers). In contrast, the security tests for milestone releases should be done by security experts and, thus, can target the full range of security testing tools and methods (ranging fro0m manual design reviews, SAST/DAST/IAST, vulnerability scanners, security proxies, or human expert knowledge).
Of course, in addition to the requirements targeting DevOps teams, the well-known requirements for selecting security test tools should be considered. Finally, secure development and operations covers all parts of the development, operations, and maintenance lifecycle, i.e., it starts already with the training (and product design) and also includes a comprehensive response and patch & update process.
Conclusion
DevOps and secure software development are not contradictory. Still, the introduction of security testing tools needs more planing (and more adaption of the tools) than introducing the same or similar tools in traditional development strategies. Following the “Customer First” strategy, security experts should focus on the needs of the software developers: the goal of security testing tools is to support developers to develop secure software, they should not be seen as an obstacle.