The global state of DevSecOps

The state of DevSecOps

Development and operations (DevOps) teams are embracing new approaches of embedding application security testing (AST) into their development pipelines to marry development, security and operations (DevSecOps). Developer and security teams agree that a single vulnerability scan late in the software development life cycle (SDLC) is insufficient, can result in delayed software releases, and often becomes unnecessarily expensive.

As with any type of security testing, discovering vulnerabilities late in the pipeline means it takes more time and effort to remediate them. Hence, it is highly desirable to shift AST as far left as possible by integrating security into the tooling developers use.

However, DevSecOps is still in its early stages. Part of the problem is budget, and part of it has to do with a talent shortage. Smaller companies especially lack the security expertise and tools larger companies have.

Some of the key findings from our survey are:

• Half of respondents plan to add more application security staff in the next year.
• 43 percent say their biggest frustration is security testing done late in the SDLC because it delays launches.
• Of all the application security types, API testing is the most popular among two-thirds of respondents.
• Two-thirds lack a security champion.

State of application security team

This report is based on a global online survey of application security professionals that took place during late May and June 2021. Sixty percent are located in EMEA, followed by APAC and the Americas. Forty-five percent hold director, VP or C-level titles.

How big is your application security team?

There are always far fewer application security team members than developers. Developers outnumber security professionals 100 to one in large organizations. More than half of respondents (53 percent) said they employ one to five application security team members, and generally speaking, the lower number is generally associated with small and medium businesses (SMBs).

However, typically, as the number of application security team members rise, so does the size of the organization. The number of companies employing 21 or more such professionals outnumbers those that employ 11 to 20.

Do you plan on adding additional staff to your application security team within the next year?

Half of respondents plan to add more security staff, with the majority (38 percent) adding just one to five more. A non-trivial factor is the shortage of security talent.

Stephen Gates, security evangelist and senior solutions specialist at Checkmarx shares that a real challenge is that "there is always a shortage of experienced people".

“That is a hurdle organization are going to have to deal with even though they want to double the size of their staff,” he explains.

How do you measure a successful application security testing program?

Nearly three-quarters (72 percent) view vulnerability reduction as the key measure of success. Fewer (46 percent) are focused on particular vulnerabilities they have identified previously.

The scary statistic is that 44 percent measure success based on security vulnerability identification, not remediation. About the same number (42 percent) are achieving release speed and quality by integrating security testing and automation, typically in a CI/CD pipeline that is highly automated otherwise.

Three in 10 have reduced bug bounties because they likely do not want to advertise the fact that they have software vulnerabilities, and they do not want people hacking their systems to find bugs. On an industry level, companies generally are decreasing bug bounties.

Also read: DevOps Secure Coding Education Infographic

What kind of reporting is given to the CISO?

Half of respondents are providing the CISO with  a PDF report of security scan findings, some of which include custom charts. The people who are not providing the CISO with any report may lack the necessary tools or time to do it, or the company may not have a CISO in the first place.

What is your biggest frustration with application security testing?

There are several frustrations with application security testing, and one of the most prominent is testing too late in the life cycle (43 percent). A quick vulnerability scan is not enough to address all application vulnerabilities, and it often results in interpersonal conflicts between developers and security since it requires developers to revisit code after they have moved onto another project. From an overhead perspective, it is more time-consuming and expensive.

“When people say a vulnerability is being identified too late, is a clear indication that integration and automation of AST solutions within their development pipelines is less than perfect,” said Gates. “Then, just before deployment, they launch a scan and then say, ‘Oh wow, we have lots more work to remediate the vulnerabilities we have discovered’.”

An equal number (43 percent) are frustrated by their inability to interpret the scan findings, how risky a suspected vulnerability may be, and how to quickly fix it. There is a gap between understanding a vulnerability and its remediation.

A related issue is tracking the vulnerability status to make sure it is actually being fixed (36 percent). Teams often lack the tools or tool integration they need to track the status of vulnerability remediation in a quick and easy way. Also at 36 percent are false positives, which waste precious time. Alternatively, when false positives are too frequent, people tend to ignore them (which is called “alert fatigue”). Alert fatigue is not only very real, but also potentially dangerous because some true positives may be ignored.

How are you currently (or planning) automating application security testing?

Time to market pressure necessitates automated AST, but it also helps ensure code quality. Those with CI/CD pipelines (36 percent) are automating or plan to automate security testing as part of a larger automation strategy since CI/CD and automation go hand in hand. Another 15 percent are relying on SCM integration.

“Source code management integration is something people either do not know about or they do not have the correct application security testing tools that can integrate directly within their SCM tools of choice,” Gates explains. “Scan early and scan often is the way to work. Within the SCM is the best location to launch scans since it is the often the farthest left you can go.”

Automation is a maturity issue that requires AST integration within the dev tools in use, which some organizations clearly lack (31 percent). However, modern AST solutions can easily be integrated and fully automated.

Also read: Traits of a Modern CISO

Where is the best place developers consuming application security results?

Half the respondent base considers a bug tracking tool like Jira the best place to consume application security results because they already have the tool in place, with both develops and security already using it.

A distant second (20 percent) is in an IDE, as modern IDEs show coding errors in real time, though they will not catch everything because they were not designed specifically to identify application vulnerabilities.

About the same number (19 percent) prefer a pull/merge request.

“If you automate the pull request, then you can actually get the results right back in the SCM tool you are working on, and what is nice about that is it is very fast,” Gates shares. “Developers have the opportunity to remediate quickly when they are sitting in a branch of code or working on it. It is just expected.”

What type of security testing are you performing?

Two-thirds of survey participants (67 percent) are doing application programming interface (API) testing because it is becoming an increasing threat. In fact, Gartner predicts that APIs will become the most popular attack vector by 2022. Another reason API testing is popular is because modern applications have a lot of dependencies. 

Today’s applications rely on a number of outside functions and data to operate as intended. Those dependencies, if they contain security flaws, may infect an application. In addition, cloud native development and the use of microservices are growing exponentially, so API vulnerabilities are of particular concern.

Static application security testing (SAST) is more than twice as popular (44 percent) as dynamic application security testing (DAST, at 21 percent) because DAST does not always fit well into modern application development practices, which emphasize speed.

When organizations start decomposing applications into microservices and using containers—or building new applications with containers—they eventually realize they need special tooling and updated practices since traditional application security tooling and practices will not work. Each container represents an individual attack surface, unlike a monolithic application. The 36 percent using container scanning realize this.

Finally, the use of software composition analysis (SCA) tools is on the rise because of open-source security concerns. Today’s applications use more third-party commercial and open-source components and libraries than ever before because of time to market mandates and an increase in application complexity. Without SCA, it can be more difficult to pinpoint open-source vulnerabilities and license risks.

Also read: DevSecOps Is not Optional Anymore

Do you use any open-source tooling specifically for finding security vulnerabilities?

Despite the increased use of open-source code, 58 percent said they are not using open-source security testing tools.

“That is a concern because of the increase in the number of open-source modules, packages, and libraries these teams are using,” notes Gates.

What improvements in application security testing would you like to see?

Most respondents (62 percent) chose better integrations as the top improvement they would like to see because of all the friction a lack of integration causes. Insufficient integration precludes visibility across tooling, and it prevents teams from achieving the level of automation they need to improve release velocity and product quality. Also causing friction are less-than-optimal user experiences (52 percent). Faster scans (50 percent) and fewer false positives (45 percent) are also about speed.

How do you communicate with developers on bugs found?

Jira tickets are the top means of communicating with developers (48 percent) since so many DevOps teams already use it.

“Integration and automation into Jira systems is probably vastly desired because of the need for speed, which extends to triage, troubleshooting, and remediation,” shares Gates. “There is a desire to have automation and integration with the actual scans to open and close tickets automatically.”

Email is an antiquated way of communicating information about code issues, but it is still popular among 31 percent of respondents since everyone uses email but not everyone uses Jira.

While email can accommodate all the information a developer will need, unlike Slack, emails are not a real-time communication medium, and they can get lost easily in a universe of other emails. It can also be difficult to search through emails to find out what the status of remediation is.

Group meetings have the benefit of getting everyone on the same page (13 percent), but they take time away from other day-to-day tasks.

Do you currently use a Security Champion type of model within your organization?

Another scary statistic is that two-thirds of respondents (66 percent) lack a security champion, when that is such an important role. The security champion evangelizes security within the organization, which helps create a cyber-aware culture. Since they are security experts and also good communicators, they are a good resource for mentoring developers. The security champion is the designated go-to person developers and others can tap when they have a security-related question.

What do you currently do for security awareness?

Wikis are alive and well in a security context. Nearly half of survey participants (47 percent) are using a wiki as the primary vehicle for security awareness, with just slightly fewer (44 percent) saying they use instructional videos.

Fewer still are those using the Open Web Application Security Project (OWASP), an online community providing freely available web application security articles, methodologies, documentation, tools, and technology. Then again, OWASP is specific to web application security, and the guidelines tend to be quite general.

About one-third have integrated real-time training, which helps accelerate application release speed and quality. Unlike videos, which take time away from coding, real-time training teaches developers application security within the context of writing code. If a developer does not know how to remediate an issue, a five-minute tutorial will often solve the problem.

What skills will be important for security professionals to have in the future?

Subject matter expertise tops the list (61 percent) of what respondents think security professionals will need to have in the future. Security professionals tend to be driven to upskill themselves because they want to advance their careers, often into the role of a security champion or CISO.

However, to become an effective security champion or CISO, they need to have technical expertise and soft skills (56 percent), such as communication and collaboration, because the role interacts with other roles in the company.

Advanced programming skills (52 percent) does not mean that security professionals are morphing into developers (though some may choose to do so). It is more about understanding the application side of application security as much as the security side of application security and being able to tell a developer what to do.

AI and machine learning are also considered important knowledge to have (48 percent), given their growing popularity throughout enterprises. At first, enterprises raced to implement AI and ML to reap the benefits, but more realize now that they must also manage the potential risks not only with the help of data scientists, but security professionals as well. Meanwhile, IoT and blockchain use is emerging, as reflected by the 35 percent.

Is the speed of an application security testing scan more valuable than the quality of results?

It is encouraging that 64 percent of respondents consider the quality of application testing more important than speed, because it speaks to the desire to provide higher quality software and avoid the headaches of a customer or bad actor discovering security issues in production.

The result may seem counterintuitive based on some of the other survey responses that demonstrate a desire to remove obstacles to speed. However, given the choice between speed and quality, quality prevails because it equates to more secure code.

Moving from DevOps to DevSecOps

Application security continues to rise in importance because it represents legal, regulatory, and brand issues. End users expect applications to be of high quality, which includes security. It is not enough to simply build and deliver applications faster. Speed and quality must go hand in hand.

As always, individual respondent companies are at different stages of maturity because speeding delivering and increasing code security simultaneously is a journey. Moving from DevOps to DevSecOps helps.

Read a PDF of the Report Here