A lack of communications between the SecOps and DevOps teams leads to potential security lapses and, in a worst case scenario, breaches and data loss. Because the SecOps and DevOps teams are often working at cross purposes, getting everyone on the same page and working towards the same goal can range from challenging to impossible. How do you get all your teams to move in tandem towards the same goal of a highly secure organization while still allowing your teams to operate efficiently and effectively? Ensuring communications are actively occurring between stakeholder groups is the most effective way to ensure your organization meets its cybersecurity and productivity goals.
On Oct 30, 2023 the US Securities and Exchange Commission (SEC) announced a lawsuit1 against SolarWinds, Inc and their CISO, Timothy G. Brown. The complaint (which runs to 68 pages) is pretty interesting both in what the SEC is saying and what they are not saying. There are plenty of examples of what appears to be extremely bad behavior by both SolarWinds and Brown (of course none of this is proven in a court of law and everyone is presumed innocent until proven guilty) via apparent lies, fudging data, insider trading, and more.
This article isn’t intended to dissect the SEC case, SolarWinds, or anything along those lines. In this article I am going to discuss tensions that arise between Security Operations (SecOps) and developers (DevOps) teams at organizations (very generally), why it is a problem, and how to fix it.
What Happens?
Organizations, especially as they grow and mature, struggle with the inherent dichotomy between DevOps and SecOps - though in all sincerity the issue is beyond that of just developers vs. cybersecurity2 .
There is an inherent divergence between the SecOps and DevOps teams as they often work at odds with each other. SecOps is focused on securing basically everything as much as possible whereas DevOps, while often sensitive to security needs, wants to be able to get their jobs done as efficiently and productively as possible.
The more secure an environment is, the more things are locked down, the more restrictive access is, and the harder it is to access resources (by design). But, when you are a developer and you need access to lots of different resources, having more and more restrictions leads to increased frustration, reduced productivity, and general annoyance. These restrictions and frustrations then lead to the inevitable next step of people developing workarounds to facilitate what they see as their needs3 . When DevOps puts mechanisms in place to avoid security they reduce the effectiveness of any and all security measures that have been put in place.
There are lots of statistics available on cybersecurity but a few that I find interesting4 and relevant include5 :
- 95 percent of cybersecurity breaches are caused by human error. (WorldEconomic Forum)
- 43 percent of all breaches are insider threats, either intentional or unintentional. (CheckPoint) and 30 percent of data breaches involve internal actors. (Verizon)
- Cyber fatigue, or apathy to proactively defending against cyberattacks, affects as much as 42 percent of companies. (Cisco)
- 15 percent of companies found 1,000,000+ files open to every employee, on average every employee has access to 11 million files, and 17 percent of all sensitive files are accessible to all employees. (Varonis)
Granted not all of these statistics are directly related to a developer bypassing a security measure to make their life easier. They are all, however, indicative of the problem. If the vast majority of breaches are caused by human error (according to the World Economic Forum) and on average every employee has access to 11 million files (including 17 percent of all sensitive files) that means that any time someone does anything that reduces security even a little bit they are increasing the chances of an issue occurring. So, an organization that does not want to pay for individual accounts on a service decides to share a single account amongst the entire development team resulting in a weak, easily crackable password which, in turn, increases the likelihood of a security breach (and even if the password is complicated because it is widely shared its security drops off drastically not to mention a likely lack of two-factor authentication)..
Why Does It Happen?
There are a myriad of reasons why these security issues occur. The most likely and common, in my view, is a lack of communication between the SecOps and DevOps teams. SecOps is tasked with securing things. DevOps is tasked with developing things. If the two don’t communicate effectively and efficiently on their needs, plans, and intentions they are going to be working at cross purposes anywhere from some to all of the time.
This lack of communication means that SecOps does not always (or ever) understand exactly what DevOps is doing, how they are doing it, or what the team's needs are. It also means that while it is likely DevOps understands the need for security in a general sense they do not appreciate the exact reasoning and needs for specific controls or restrictions. This lack of communication results in two (or more) groups of people working at cross purposes and neither getting what they need or doing what they should.
The failure to properly and effectively communicate is not necessarily intentional. The reasons can be anything from no one telling the teams they need to work together (really since we are all adult professionals no one should have to tell you to work with stakeholders but here we are) to there was a major breach and the organization is in full panic mode just locking everything down (this is a normal reaction and likely fine as long as after the lockdown is complete things are reevaluated to ensure they make sense) to internal politics and infighting preventing people and teams from working together (this one can be common and is almost always stupid). Unfortunately, the fact that there may be a legitimate reason for the lack of communication does not excuse the failure.
How Do You Fix It?
I’ve used the word communicate (or some variation thereof) many times in this short article and that is because this problem is almost always a result of either miscommunication or a failure to communicate. SecOps and DevOps (and, as mentioned at the beginning of the article, other teams) need to be communicating on a regular basis on what each of them needs. This may be frequent at the beginning of a project (daily or weekly meetings) and less frequent as things move forward (once everything is deployed communications may reduce to monthly or quarterly unless there is a specific need to be addressed).
Part of the requirement for communication is that each team understands what the other team is doing and (perhaps most importantly) why they are doing it. If SecOps understands how DevOps works, what tools they use, what their processes are, and why those processes, tools, etc. are in use it means SecOps can design programs that allow DevOps to continue doing their jobs productively. Similarly, if DevOps understands exactly what SecOps is trying to achieve, how they are going to achieve it, and why they need to do it, DevOps is more likely to not struggle with new security measures.
There are numerous ways to achieve improved communications between teams. While there is no single “right way” to do things there are different approaches that can be taken. I outline two different techniques that can be used to help overcome these communication gaps below. It is important to remember that these are merely two examples out of dozens of possible methodologies and that, because every organization is unique, what works for you and your organization will differ from other organizations (and may require portions of multiple different techniques). With that in mind, two approaches to consider are:
Organize regular “brown-bag” lunch sessions between teams. During each session a member of one team (SecOps or DevOps) covers a specific area of their job/operations. This could include some of the following:
- Discussing the entire development process from a high level (what programming languages are in use, what tools are used by what teams and how, do you use an agile or waterfall approach, how are development decisions made, etc.). It may not be realistic to cover all these topics in a single session and additional drill-downs into specific topics (such as tools in use) may be required for a fuller understanding.
- Discussing why specific security measures are being put in place. DevOps may not keep up to date on the latest cybersecurity news so when SecOps introduces a new set of restrictions it may appear to be arbitrary. Sitting down and discussing the reasons behind the changes can help everyone understand what is happening and why it is happening. Additionally, feedback from the end users can be invaluable as it is likely to bring up concerns and considerations that were missed.
- Present a SecOps roadmap that covers where things are currently with security as well as where they are planned to be over the next two to five years. Providing this information to others allows everyone to see what the plans are so other decisions can be made in an informed way. If the SecOps plan is to migrate to a zero trust framework beginning in twelve months with an additional six month rollout path providing that information in advance to the other teams will allow them to plan their processes around these changes before the changes are implemented.
- Perhaps most importantly, if you are providing lunch to employees in an office you should also provide lunch to remote attendees (whether they are in a remote office or working from home) via a delivery service gift card. No one likes to feel left out and multi-disciplinary team building is the entire point of this exercise.
Schedule a regular cadence of meetings with the plan to cover specific items that are actively in planning. Unlike brown-bags discussed previously, these meetings would focus on actual projects that are in process. These meetings should happen on a cadence that makes sense depending on where in the process things are:
- Early in the process when plans are still being developed (i.e. requirements set, vendors identified, stakeholders identified, etc.) meetings should happen fairly frequently (weekly or alternating weeks at a minimum) to allow for rapid iteration of plans and understanding of specific needs.
- In the middle of the process, when evaluating vendor responses, choosing vendors, products, features, etc. meetings should continue on a regular cadence though with a bit more flexibility. Analyzing responses and making decisions requires a balance of discussions and individual analysis. Team members should review the returned documents individually, making notes along the way, so the entire team can meet and discuss the results. By having members review documents in advance meetings can focus on specific needs/questions/concerns instead of reading through an entire RFP proposal on a call.
- During and around the active implementation time of the solution meetings should be at least weekly and possibly more frequent. Because things may change on a day-to-day (or even hour-to-hour) basis during a deployment any long delay between team meetings has the potential to result in delays (for example if a new endpoint agent is being deployed but there is a conflict with another piece of software waiting a week to deal with that issue can be highly problematic).
- After successful implementation and sign-off on the deployment there may be a temptation to simply stop meetings related to the project. This would be a mistake however. It is likely that meetings can be reduced in frequency (instead of weekly, perhaps monthly or quarterly) while still allowing for sufficient communications. Maintaining a regular (but reduced) cadence of meetings is important for many reasons.
Maintaining your cross-functional teams after successful completion and deployment of the project is critical for a variety of reasons. For example:
- Upgrades are a near universal requirement. These upgrades could be new versions of software, hardware, etc. While upgrades should be smooth and easy to accomplish there are always risks related to them. Being able to discuss these risks with all relevant stakeholders is critical. SecOps making updates to a security tool without understanding how that update impacts the DevOps team makes it that much more likely that things will go horribly wrong.
- Needs change over time and tracking those changing needs and how they interact with other parts of the organization is critical. Just because software (for example) was successfully deployed and is in active use today does not mean problems will not occur down the road. The DevOps team is likely to have changing requirements. If DevOps and SecOps do not continue to maintain open lines of communications they are going to struggle as each team's needs are adjusted to meet the new realities of the organization.
Over time there will be new projects. Some may be replacing existing tools/solutions while others may be entirely new. Having an effective working team across multiple groups gives you a leg up on these new projects. Instead of having to spin a team up from scratch, build trust, establish understandings, learn about each other, you have an existing team that has done most of that already. Sure, people may swap in and out of the team (maybe there is a requirement for everyone to spend at least a year on the team from every associated division) but overall there will be an effective and efficient group ready to take on the next project.
---------------------------------
[1] Christopher Bruckmann (SDNY barno. CB-7317) attorney for ... - sec.gov. (n.d.). https://www.sec.gov/files/litigation/complaints/2023/comp-pr2023-227.pdf
[2] One of my father's favorite anecdotes is the battle at airlines between the division responsible for selling as many seats as possible and the division responsible for giving frequent flier seats as many rewards as possible. The seller wants to sell all the seats and not give any away. The rewards person wants to reward people as much as possible to make their program better. The two have to meet somewhere in the middle. The same is true of SecOps and DevOps.
[3] One of the best examples of this I’ve seen was on a US military base where someone had set up a route between the military
and national guard networks both running on the same base to get around outbound Internet filters on the military network. We found it when the colonel told us Facebook was not accessible from the military network and I brought up Facebook on a computer connected to the military network. They were not happy.
[4] Sobers, Rob. “166 Cybersecurity Statistics and Trends [Updated 2022].” Varonis, Varonis, 3 Aug. 2022, www.varonis.com/blog/cybersecurity-statistics#data-breach-hacking.
[5] Keep in mind that because these statistics are pulled from different sources they overlap in weird ways so the percentage numbers may not always be entirely logical.
---------------------------------
Josh Chessman is an Advisor at Lionfish Tech Advisors.