We had 2 code analyzers, 4 dynamic testing tools, our own crafts, and 250 scripts. It's not that everything was needed in the current process, but since I started implementing DevSecOps, itâs necessary to see it through to the end.
. Character authors: Justin Roiland and Dan Harmon.
What is SecDevOps? And DevSecOps? What are the differences? Application Security â what is it about? Why doesnât the classical approach work anymore? The answers to all these questions are known. Yuri Shabalin from Swordfish Security. Yuri will answer everything in detail and analyze the challenges of transitioning from the classical Application Security model to the DevSecOps process: how to properly integrate secure development into the DevOps process without breaking anything, how to go through the main stages of security testing, which tools can be used, how they differ, and how to configure them correctly to avoid pitfalls.
About the speaker: Yuri Shabalin â Chief Security Architect at Swordfish Security. He is responsible for implementing SSDL, for the overall integration of application analysis tools into a unified development and testing ecosystem. 7 years of experience in information security. He has worked at Alfa-Bank, Sberbank, and Positive Technologies, which develops software and provides services. A speaker at international conferences such as ZerONights, PHDays, RISSPA, OWASP.
Application Security: what is it about?
Application Security â this is a branch of security that focuses on application security. It does not relate to infrastructure or network security, but specifically to what we write and what developers work on â it concerns the flaws and vulnerabilities of the application itself.
Direction  â Security Development Lifecycle â developed by Microsoft. The diagram shows the canonical SDLC model, whose main task is to involve security at every stage of development, from requirements to release and production deployment. Microsoft realized that there were too many bugs in production, they were becoming more frequent, and something needed to be done about it, leading to the proposal of this approach, which became canonical.
Application Security and SSDL are focused not on detecting vulnerabilities, as is commonly believed, but on preventing their emergence. Over time, the canonical approach by Microsoft has been improved and developed, resulting in a deeper and more detailed approach.
The canonical SDLC is detailed in various methodologies - OpenSAMM, BSIMM, OWASP. The methodologies differ, but overall, they are similar.
Building Security In Maturity Model
I am most fond of BSIMMÂ â . The foundation of the methodology is the division of the Application Security process into 4 domains: Governance, Intelligence, SSDL Touchpoints, and Deployment. Each domain includes 12 practices presented as 112 activities.
Each of the 112 activities has 3 maturity levels: initial, intermediate, and advanced. All 12 practices can be studied by sections, selecting important aspects for you, understanding how to implement them, and gradually adding elements such as static and dynamic code analysis or code review. You draft a plan and work calmly within the framework of implementing the selected activities.
Why DevSecOps
DevOps is a comprehensive process where security must be considered.
Initially was expected to conduct security checks. In practice, the number of security teams was significantly lower than it is today, and they acted not as participants in the process but as a control and oversight body that imposes requirements and checks product quality at the end of the release. This is a classic approach where security teams were separated from development and did not participate in the process.
The main problem is precisely that information security is set apart from development. Typically, it is some sort of security perimeter with 2-3 large and expensive tools. Every six months the source code or application arrives for review, and annual . All of this leads to delays in production timelines, and a huge number of vulnerabilities from automated tools are dumped on the developer. It is impossible to sort and fix this because the results from the previous six months have not yet been dealt with, and now there is a new batch.
In the course of our company's work, we see that security in all sectors and industries understands that it's time to step up and align with development in the same wheel - in . The DevSecOps paradigm fits perfectly into agile development methodology, ensuring involvement, support, and participation in every release and iteration.
Transition to DevSecOps
The most important word in the Security Development Lifecycle is "process"You need to understand this before thinking about purchasing tools.
Simply incorporating tools into the DevOps process is not enoughâit's important to have interaction and understanding among the participants of the process.
People are more important than tools.
Often, the planning of a secure development process starts with the selection and purchase of a tool and ends with attempts to integrate the tool into the existing process, which often remain just attempts. This leads to unfortunate outcomes, as every tool has its own features and limitations.
A common scenario is when the security department selects a good, expensive tool with extensive capabilities and approaches developers to integrate it into the process. But it doesn't workâit turns out the process is structured in a way that the limitations of the purchased tool do not fit into the current paradigm.
First, describe the outcome you want and how the process will look. This will help clarify the roles of the tool and security in the process.
Start with what is already being used.
Before purchasing expensive tools, take a look at what you already have. Every company has security requirements imposed on development, there are checks, pentestsâwhy not transform all this into a clear and convenient format for everyone?
Usually, requirements are a paper tome that sits on a shelf. There was a case when we visited a company to review processes and asked to show the security requirements for the software. The specialist responsible for this searched for a long time:
â Right now, somewhere in the notes, there was a path to where this document is located.
In the end, we received the document after a week.
For requirements, checks, and other matters, create a page, for example, on Confluence â this is convenient for everyone.
It's easier to reformat what already exists and use it as a starting point.
Use Security Champions.
Usually, in a medium-sized company with 100-200 developers, there is one security person who performs multiple functions and physically cannot check everything. Even if they try their hardestâthey cannot review all the code generated by development alone. For such cases, the concept of .
Security Champions are individuals within the development team who are invested in the security of your product.
A Security Champion is both an entry point into the development team and a security evangelist.
Typically, when a security expert joins the development team and points out a bug in the code, they receive a surprised response:
â And who are you? Iâm seeing you for the first time. Everything is fine for me â my senior colleague approved it during the code review, and we're moving on!
This is a common scenario because developers trust senior colleagues or team members they work with closely and frequently in code review much more. If instead of a security expert, a Security Champion points out a bug and its consequences, their word carries more weight.
Developers also know their code better than any security expert. For someone with at least 5 projects in a static analysis tool, itâs usually hard to remember all the nuances. Security Champions are familiar with their product: what interacts with what and what to prioritizeâthey are more effective.
So, consider implementing Security Champions to expand the influence of the security team. Itâs also beneficial for the champion: professional development in a new area, broadening technical horizons, enhancing technical, managerial, and leadership skills, and increasing market value. They serve as your âeyesâ in the development team.
Testing Stages
 states that 20% of efforts yield 80% of results. These 20% represent application analysis practices that can and should be automated. Examples of such activities include static analysis â SAST, dynamic analysis â DAST, and open source monitoring. I will elaborate on the activities and the tools, along with the specific challenges we typically encounter when implementing them into the process and how to do it correctly.
Key Issues with Tools
I will highlight the relevant issues that all tools face, which require attention. I will discuss them in detail to avoid repetition.
Long analysis times. If it takes 30 minutes from commit to production for all tests and builds, then checks for information security will take a day. No one is going to slow down the process like that. Keep this in mind and draw conclusions.
A high level of False Negatives or False Positives. All products are different; they use various frameworks and coding styles. Different codebases and technologies can show varying levels of False Negatives and False Positives. So, take a look at what's specific in your company and for your applications that will yield good and reliable results.
No integrations with existing tools. Look at the tools from the perspective of integrations, considering what you are already using. For example, if you have Jenkins or TeamCity, check the integration of the tools specifically with that software, not with GitLab CI, which you do not use.
Lack of or overly complex customization. If the tool has no API, then what is its purpose? Everything that can be done in the interface should be available through the API. Ideally, the tool should have the capability to customize checks.
No product development roadmap. Development is always evolving; we constantly use new frameworks and features, rewriting old code in new languages. We want to be assured that the tool we purchase will support new frameworks and technologies. Thus, it is essential to know that the product has a genuine and proper roadmap for development.
Process features
In addition to the features of the tools, consider the peculiarities of the development process. For instance, interfering with development is a typical mistake. Let's look at what other features need to be taken into account and what the security team should pay attention to.
To avoid derailing development and release timelines, establish different rules and various show stoppers â criteria for halting the build process when vulnerabilities are present â for different environments.For example, we understand that the current branch is heading to the development stage or UAT, which means we do not stop and do not say:
â You have vulnerabilities here; youâre not going any further!
At this stage, it is essential to inform developers that there are security issues that need attention.
The presence of vulnerabilities is not an obstacle for further testing.whether manual, integration, or manual testing. On the other hand, we need to improve the product's security somehow, and ensure that developers do not overlook the findings of the security team. So sometimes we do this: when a release goes to the development environment, we simply notify the developers:
â Guys, you have issues, please pay attention to them.
At the UAT stage, we again show warnings about vulnerabilities, and at the production launch stage we say:
â Guys, we warned you several times, you did nothing â we will not release with this.
When it comes to code and dynamics, we need to show and warn about vulnerabilities only for those features and code that have just been written in that feature. If a developer moved a button 3 pixels and we tell them that they have an SQL injection and need to fix it urgently â that is incorrect. Only look at what has been written now and at the changes coming into the application.
Suppose we have some functional defect â that is, how the application should not work: money is not transferred, clicking on a button does not lead to the next page, or the product does not load. Security defects are similar defects, but not in the context of application functionality, rather in terms of security.
Not all software quality issues are security problems. However, all security problems are related to software quality. Sherif Mansour, Expedia.
Since all vulnerabilities are similar defects, they should be located where all development defects are. So forget about reports and scary PDFs that no one reads.
When I worked at a company that was involved in development, I received a report from static analysis tools. I opened it, was horrified, made coffee, flipped through 350 pages, closed it, and went back to work. Large reports are dead reports.Usually they go nowhere, emails get deleted, forgotten, lost, or the business says it accepts the risks.
What to do? The confirmed defects that have been found are simply converted into a format suitable for development, for example, stored in the backlog in Jira. We prioritize and eliminate defects in order of priority along with functional defects and testing defects.
Static Analysis - SAST
This is code analysis for vulnerabilities., but it is not the same as SonarQube. We check not only against patterns or styles. A range of approaches is applied during analysis: by vulnerability tree, by , by analyzing configuration files. This all relates directly to the code.
Advantages of the approach: identifying vulnerabilities in the code at an early stage of development, when there are still no environments or ready tools, and the possibility of incremental scanning: scanning the part of the code that has changed, and only the feature we are currently working on, which reduces scanning time.
Cons â there is no support for the necessary languages.
Necessary integrations, which should be in the tools, in my subjective opinion:
- Integration tools: Jenkins, TeamCity, and Gitlab CI.
- Development environment: Intellij IDEA, Visual Studio. It's easier for the developer not to navigate an unfamiliar interface that needs to be remembered, but to see all necessary integrations and vulnerabilities discovered directly in their own development environment at their work.
- Code review: SonarQube and manual review.
- Defect trackers: Jira and Bugzilla.
In the picture are several of the best representatives of static analysis.
Itâs not the tools that matter, but the process; therefore, there are Open Source solutions that are also good for testing the process.
SAST Open Source won't find a huge number of vulnerabilities or complex DataFlow, but when building the process, they can and should be used. They help understand how the process will be structured, who will be responsible for the bugs, who will report, and who will provide reports. If you want to go through the initial phase of building security into your code, use Open Source solutions.
How can this be integrated if you are just starting out and have nothing: no CI, no Jenkins, no TeamCity? Letâs consider integrations in the process.
Integration at the CVS level
If you have Bitbucket or GitLab, you can integrate at the .
By event â pull request, commit. You scan the code and in the build status, you indicate whether the security check passed or failed.
Feedback. Certainly, feedback is always necessary. If you just performed it on the security side, put it in a box and told no one about it, and then at the end of the month dumped a bunch of bugs â thatâs wrong and not good.
Integration with the code review system
Once, we set the technical user AppSec as the default reviewer in several important projects. Depending on whether problems were identified in the new code or not, the reviewer assigns a status on the pull request as 'accept' or 'need work' â either all is OK, or it needs some improvement along with links to what exactly needs to be improved. For the integration with the version going to production, we had a merge prohibition enabled if the security test did not pass. We included this in manual code review, and other participants in the process could see statuses pertaining to security specifically for that process.
Integration with SonarQube
Many have for code quality. The same applies here â you can create the same gates just for SAST tools. It will have the same interface, the same quality gate, but it will be called security gate. And likewise, if you have a process set up using SonarQube, you can easily integrate everything there.
Integration at the CI level
Here too, everything is quite straightforward:
- At the same level as automated tests, unit tests.
- Separation by development stages: dev, test, prod. Different sets of rules can be included or different fail conditions: stop the build, do not stop the build.
- Synchronous/asynchronous execution. We wait for the results of the security tests or we do not wait. That is, we just start them and move on, and then we receive a status indicating whether everything is good or bad.
This is all in an ideal pink world. In real life, this does not happen, but we strive for it. The results of security checks should be analogous to the results of unit tests.
For example, we took a large project and decided that we would now scan it using SASTâOK. We put this project into SAST, it reported 20,000 vulnerabilities, and with a decisive action, we accepted that everything was fine. 20,000 vulnerabilitiesâthat's our technical debt. We'll box this debt up, gradually tackle it, and log bugs in defect trackers. We'll hire a company, do everything ourselves, or be assisted by Security Championsâand the technical debt will reduce.
All newly identified vulnerabilities in the new code must be addressed just like errors in unit or automated tests. Let's say a build was launched, two tests failed, and two security-related tests also failed. OKâwe went, looked at what happened, fixed one issue, fixed the other, and ran the tests againâeverything is fine, no new vulnerabilities appeared, and the tests passed. If the task is deeper and needs thorough understanding, or if fixing vulnerabilities affects large parts of what's under the hood: a bug is logged in the defect tracker, prioritized, and fixed. Unfortunately, the world is not perfect, and tests sometimes fail.
An example of a security gate is akin to a quality gate, based on the presence and quantity of vulnerabilities in the code.
We integrate with SonarQubeâa plugin is installed, everything is very convenient and great.
Integration with the development environment
Integration capabilities:
- Launching scans from the development environment even before committing.
- Viewing results.
- Analyzing results.
- Synchronization with the server.
This is what retrieving results from the server looks like.
In our development environment a new item simply appears that indicates that such vulnerabilities were discovered during scanning. You can immediately correct the code, review recommendations, and . Everything is located at the developer's workstation, which is very convenientâyou don't have to navigate through other links and look for additional information.
Open Source
This is my favorite topic. Everyone uses Open Source librariesâwhy write a ton of workarounds and reinvent the wheel when you can use a ready-made library where everything is already implemented?
Of course, that's true, but libraries are also written by people and include certain risks, with vulnerabilities that are periodically or constantly reported. Therefore, the next step in Application Security is the analysis of Open Source components.
Open Source Analysis - OSA
The tool consists of three main stages.
Searching for vulnerabilities in libraries. For example, the tool knows that we're using a certain library, and that in or in bug trackers there are vulnerabilities related to this version of the library. When attempting to use it, the tool will issue a warning that the library is vulnerable and recommends using another version that does not have these vulnerabilities.
Analysis of license compliance. This is not particularly popular for us yet, but if you work overseas, you can occasionally face penalties for using an open-source component that cannot be used or modified. According to the licensing policy of the library, we cannot do that. Or, if we have modified it and are using it, we must publish our code. Of course, no one wants to make their product code public, but there are ways to protect against this as well.
Analysis of components used in the industrial environment. Let's imagine a hypothetical situation where we have finally completed development and released the final version of our microservice into production. It lives wonderfully there for a week, a month, a year. We don't collect it, we don't conduct security checks, everything seems fine. But suddenly, two weeks after the release, a critical vulnerability is found in the Open Source component that we use in this build, in production. If we don't keep a record of what and where we use, we simply won't see this vulnerability. Some tools offer the ability to monitor vulnerabilities in the libraries currently used in production. This is very useful.
Features:
- Different policies for different stages of development.
- Monitoring components in the industrial environment.
- Control of libraries within the organization's perimeter.
- Support for various build systems and languages.
- Analysis of Docker images.
A few examples of industry leaders that engage in Open Source analysis.
The only free one among them is from OWASP. It can be enabled in the early stages to see how it works and what it supports. Basically, this covers all cloud products and on-premise solutions, but they still send their base data to the internet. They send not your libraries, but hashes or their computed values, along with fingerprints, to their server to get notifications about vulnerabilities.
Integration into the process
Control of libraries within the perimeter, which are downloaded from external sources. We have both external and internal repositories. For instance, inside Event Central, there is Nexus, and we want our internal repository to be free from vulnerabilities rated as 'critical' or 'high'. Proxying can be set up using the Nexus Firewall Lifecycle tool to filter out such vulnerabilities and prevent them from entering the internal repository.
Integration into CI. At the same level as automated tests, unit tests, and phase separation: dev, test, prod. At each stage, any libraries can be downloaded and anything can be used, but if there's something severe rated as 'critical', it might be worth bringing to the developers' attention before production release.
Integration with artifact repositories: Nexus and JFrog.
Integration into the development environment. The tools you choose should integrate with development environments. Developers should have access to scanning results from their workstations or have the ability to scan and check code for vulnerabilities before committing to CVS.
Integration into CD. This is a cool feature that I really like and have already mentionedâmonitoring the emergence of new vulnerabilities in the production environment. It works roughly like this.
We have Public Component Repositories â some external tools, and our internal repository. We want only trusted components in it. When proxying a request, we check that the downloaded library has no vulnerabilities. If it falls under certain policies that we establish and must agree upon with development, then it won't be downloaded, and a notification comes in to use a different version. Accordingly, if there is something truly critical and bad in the library, then the developer will not receive it at the installation stageâ they should use a higher or lower version.
- During the build process, we check that no one has slipped in anything malicious, that all components are safe, and that no one has brought anything dangerous on a flash drive.
- Our repository contains only trusted components.
- During deployment, we double-check the package itself: war, jar, DL, or Docker image, to ensure it complies with the policies.
- Upon going live, we monitor what happens in the production environment: whether critical vulnerabilities appear or not.
Dynamic analysisâDAST
Dynamic analysis tools differ significantly from everything mentioned earlier. Itâs a kind of simulation of a user interacting with the application. If this is a web application, we send requests mimicking client behavior, click buttons on the front end, and send artificial data from forms: quotes, brackets, characters in different encodings, to see how the application operates and processes external data.
This system also allows checking for template vulnerabilities in Open Source. Since DAST doesnât know which Open Source we are using, it simply throws âmaliciousâ patterns and analyzes the server responses:
â Aha, thereâs a deserialization problem here, and there isnât one there.
There are significant risks involved because if you conduct this security test on the same environment that the testers are operating onâ unpleasant things can happen.
- High load on the application server network.
- No integrations.
- Ability to change settings of the analyzed application.
- No support for the required technologies.
- Complexity of configuration.
We had a situation where we finally launched AppScan: it took a long time to gain access to the application, we got 3 accounts and were happy â finally, we could check everything! We started the scan, and the first thing AppScan did was access the admin panel, clicked all the buttons, changed half of the data, and then completely crashed the server with its -requests. Development with testing said:
â Guys, are you kidding?! We provided you with accounts, and you just brought the stand down!
Consider the possible risks. Ideally, prepare a separate stand for testing information security, which should be isolated from the rest of the environment at least somewhat, and it's preferable to check the admin panel manually. This is a pentest â those remaining percentages of effort that we are not considering right now.
It's worth noting that this can be used as an equivalent of load testing. In the first phase, you can enable a dynamic scanner with 10-15 threads and see what happens, but usually, as practice shows, nothing good comes of it.
Several resources that we usually use.
It's worth highlighting  â this is the 'Swiss Army Knife' for any security professional. Everyone uses it, and it's very convenient. A new demo version of the enterprise edition has just been released. Previously, it was just a standalone utility with plugins, but now the developers are finally creating a large server that will allow managing multiple agents from it. This is great; I recommend trying it out.
Integration into the process
Integration occurs quite well and easily: starting the scan after a successful installation of the application on the stand and scanning after successful integration testing.
If the integrations are not working or if there are stubs and mock functions in place, this is pointless and useless â no matter what pattern we send, the server will respond the same way.
- Ideally â a separate stand for testing.
- Before starting the test, record the login sequence.
- System administration testing is only manual.
Process
A bit of a general overview of the process in general and about the operation of each tool, in particular. All applications are different â one works better with dynamic analysis, another with static analysis, with OpenSource analysis, pentests, or something entirely different, for example, events with .
Every process needs to be monitored.
To understand how the process works and where it can be improved, you need to gather metrics from all accessible areas, including production metrics, metrics from tools, and from defect trackers.
Any data is useful. You should look at various angles to see where a particular tool is best applied, and where the process is specifically faltering. It might be worth considering development response time to identify areas for process improvement based on timing. The more data you have, the more perspectives you can gain, from high-level overviews down to the details of each process.
Since all static and dynamic analyzers have their own APIs, methods of execution, principles, some have schedulers while others do not, we are developing a tool AppSec Orchestrator, which provides a single entry point into the entire process from the product and allows management from one location.
For managers, developers, and security engineers, there is one entry point from which they can see what is running, configure and initiate scans, receive scan results, and present requirements. We aim to move away from paperwork and translate everything into a human-readable format used by developmentâpages on Confluence with statuses and metrics, defects in Jira or various defect trackers, or integration into synchronous/asynchronous processes in CI/CD.
Key Takeaways
Tools are not the main focus. First, think through the processâthen implement the tools. Tools are good but costly, so it's better to start with the process and establish interaction and understanding between development and security. From a security standpoint, it is not advisable to 'cross everything off' indiscriminately; from a development perspective, if something is high mega super critical, it needs to be addressed rather than ignored.
Product Quality is the common goal for both security and development. Weâre working towards the same objective, ensuring everything operates correctly to mitigate reputational and financial risks. Thatâs why we advocate for a DevSecOps and SecDevOps approach to facilitate communication and enhance product quality.
Start with what already exists: requirements, architecture, partial checks, training, guidelines. You donât need to apply all practices to all projects at once â move iteratively. There is no single standard â experiment and try different approaches and solutions.
Between information security defects and functional defects, there is an equal sign.
Automate everything, that moves. Everything that doesn't move â make it move and automate it. If something is done manually, it's not a good part of the process. You might want to reconsider it and also automate.
If the information security team is small â use Security Champions.
Perhaps what Iâve talked about wonât fit you and youâll come up with something of your own â and thatâs fine. But choose tools based on the requirements specific to your process. Donât look at what the community says; that this tool is bad and that one is good. It may turn out to be quite the opposite for your product.
Requirements for tools.
- Low level of False Positives.
- Adequate analysis time.
- Ease of use.
- Availability of integrations.
- Understanding of the product development roadmap.
- Possibility of tool customization.
Yuri's report was chosen as one of the best at DevOpsConf 2018. To see even more interesting ideas and practical cases, come to Skolkovo on May 27 and 28 at as part of . And even better, if you are ready to share your experience, then for a report by April 21.
Source: habr.com
