At the end of last year two security breaches made all the headlines. JPMorgan announced that information about 83 Million accounts had been obtained because a single server lacked two-factor authentication. Sony Pictures fell victim to a Malware attack that allowed attackers to access corporate emails and employee information. Then over Christmas the Playstation Network and Xbox Live were taken down by huge DDOS attacks. Since those attacks I’ve read a few blog posts presenting certain tools organisations can adopt to improve security. The mainstream IT press have focussed on the processes and procedures organisations should have in place in the event of a security breach. There’s a pattern to this reporting and it mirrors the patterns I see in IT security in most organisations. There’s a great deal of focus on technological solutions and a lot of focus on what to do once an incident occurs but very little focus on how organisations should behave day-to-day to improve their security. Unfortunately this situation has necessitated a very long article so if you’re short on time you can skip directly to the conclusion.

The perennial IT security problem

It’s been my experience that IT security is often considered as an additional set of tasks, separate from day-to-day work. In smaller businesses security is just considered another responsibility of the IT engineers in larger firms security is often devolved to security departments while management reiterate that security is everyone’s responsibility and include a couple of paragraphs in the IT systems policy in the employee handbook.

The security department, devoid of any real authority over technical implementation define their role as providing audits and reminding engineering teams of best practices. Engineering teams are then left to implement solutions in isolation while trying to keep on top of all their other work. Development teams are often given little guidance and so are left to do their best to secure the applications they create and the only time they’re given any feedback is after a penetration test is performed or an incident occurs. If I might paraphrase The Incredibles if security is everyone’s responsibility then it’s no-ones. Security is often considered only at the beginning of a project and then ignored until someone decides to run an audit or there’s a security breach. In smaller businesses it is sometimes just ignored altogether in favour of building the business faster. Unfortunately when it’s time to address security again the task is herculean because the number of systems and services has multiplied. In one company I worked at the security department ran an audit on some servers and presented the engineering team with a list of 19,000 recommendations. Clearly this was the output from some automated tool. The various recommendations weren’t prioritised. No attempt was made to rationalise the actions to determine if any tasks needed to be performed on every system. The security team felt they had done their job by highlighting the problems and they moved on, they didn’t even try to work with the engineering teams and their customers to try to get the security work prioritised alongside the engineering team’s other work. Then on a regular basis they’d send an email to the manager of the team to ask how the actions were progressing. This is the very definition of activity rather than progress. A box was ticked and nothing significant was improved. It seems to me that security is another area where DevOps can make a difference. By breaking down silos, fostering collaboration and providing measurable improvements a DevOps approach can turn security from activity into progress. While the ideas I’m going to present below certainly can be useful in organisations not following DevOps practices I think they are almost infinitely harder to implement in a non-Agile, non-iterative, silo’d world.

The security strategy is vital!

Security starts, as all large initiatives should with a strategy. The trick then is to use that strategy to create tactical approaches that can in-turn be used to create action plans with measurable outputs. In my book Next Gen DevOps: Creating the DevOps Organisation I devote an entire chapter to creating strategies and using them to create tactical approaches and detailed plans. I won’t repeat that entire chapter here. What I wanted to do with this article was provide a more specific example, using security, that demonstrates how IT leaders can drive forward the security agenda, liberate improved productivity and make security everyone’s responsibility while at the same time ensuring everyone in the IT organisation understands their responsibilities to the security agenda. I regard a strategy as being: “…a statement that describes how goals will be achieved and success will be determined. It should, for clarity’s sake, include a statement about what those goals are, but the important word is ‘how’.” A typical modern IT business provides ongoing services to it’s customers. It stores some sensitive information about those customers including their name, address, phone number and other personally identifiable information (PII). In addition it will probably store a transaction history and provide some secure purchasing method. This could describe a taxi firm with an online booking app, or a utility provider with some online services or even a games company. A suitable strategy might be expressed thus: “We regard our customers data as important as our own and will treat it as such. We will permit only those accesses that are absolutely essential. We will safeguard this data and will follow the latest best practice guidelines for data storage and systems and network access. While there is no end to security provision we will deem ourselves successful when only those accesses that are absolutely necessary are present and every failed access attempt is audited.” While I admit this strategy is a bit wooly and vague because it’s just an example one it fulfils the basic criteria. It states what our goals are, how we’ll achieve them and how we’ll know when we’ve been successful. The next stage is to get the heads of our department and some senior people together to create the tactical approaches so that our strategy can become plans. In the case of security these tactical approaches are should be guidelines that describe the tools and technologies we’ll use and any configuration options we consider essential. The guidelines may also define any processes we need such as change management or how we’ll handle access requests.

What comes after the strategy?

This is where DevOps can vastly improve our productivity. Rather than considering the domains security needs to be applied to as skill-sets i.e. Data centres, networks, systems and applications DevOps can help us take a product approach. By having our network, software, operations and test engineers collaborate we can look at how our important data is used, how it moves, where it’s stored and who needs access to it and what they need to do with it. By defining our domains this way we don’t risk having tasks fall between the cracks that exist between silos. We also have the opportunity to make decisions that improve security and reduce the workload and time to implement by allowing the engineers from all the disciplines work together. It’s not that this isn’t possible in a traditional environment but negotiating priorities with the half-dozen different managers, building support, trading favours and then trying to bridge the responsibility gaps between each group absolutely kills productivity and often causes larger initiatives to stall indefinitely effectively stopping them completely. It’s then quite likely that certain tasks will require multiple groups to work together and here again DevOps smooths that process too. By focussing on the task and what a successful implementation of that task looks like the organisation is far more likely to actually achieve the desired result rather than having to compromise the implementation. Customers usually provide their data to us using their computer and an application we have created. So we need a secure connection to their computer. We also need a secure application, hosted on a secure system on a secure network to make that connection to their computer. Secure in this case meaning many things but starting with only making or allowing those connections that are absolutely necessary. Getting network, software, system and test teams working together looking at these requirements we can determine the most efficient route forward. It may be obvious to that group that a browser making an SSL connection to a web server requires the least effort and provides an adequate level of security. That then informs the network engineers that a certain type of traffic will need to be allowed for. It also informs the system engineers that certificates will be required and some web server software is needed. It also informs them that certain ports will need to be opened to receive connections from certain protocols. Test engineers now have a whole bunch of things they can test for the presence of and the absence of. This may lead the group to conclude that they need a tool for managing ACLs, that team has the right mix of people to determine if such a tool will be useful and how such a tool can be built, hosted and tested and how that capability can be monitored. Even at this early stage it’s now obvious that guidelines are needed for network connectivity, systems security, SSL connections and web accessible applications. These guidelines provide the tactical approaches that define how security will be provided. These guidelines can mandate certain operating systems and the mechanisms that will be used to ensure they are secure. They can mandate how systems receive internet connections and the firewalls and ACLs that will be in place on the routers and switches between the internet connections and the systems. The guidelines can mandate the Certificate Authorities that will be used and the types of certificates that will be used. They could also mandate particular applications and where to get verified versions of those applications. They might even mandate particular configuration options the group agree are necessary. We now have practical pragmatic guidelines, created by people whose teams will be a part of the actual implementation. They own these guidelines they understand the decisions that led to them and as a result they are responsible for them and for abiding by them. With the guidelines in place projects can now be formulated to provide the solutions necessary. Networks and systems can be designed and built.

From strategy to guidelines to a plan!

The strategy and the guidelines provide sufficient information to allow a project manager or scrum master with sufficient understanding of the product to work with the engineering teams to create plans. If the teams in the organisation are arranged by skill-set rather than product focus there will still be the problem of prioritisation of tasks. It’s still possible for certain teams to delay the project because of their prioritisation choices. At least tasks can’t be missed though. Because the project is focussed on the successful execution of certain tests the senior leadership of the organisation can focus on those tests. That allows prioritisation to be very easily executed. They are either prioritised or not and leadership carries the responsibility for those priority calls. If the operations team are slow to update the configuration management system then the data cannot come in to the organisation securely. There’s no ambiguity there. If the developers chose an implementation method that passes the data insecurely the tests will demonstrate that and the test will not be passed. This gives the organisation a myriad of options, they can create tiger teams, they can hire in temporary staff or they can reprioritise certain work. By using DevOps methods for security implementation each assertion is tested for. Those tests are automated and can execute against the development, test and live environments every minute if so desired. This ensures that no regression bug can break the security mechanism.

Conclusion / TL:DR

As usual the real value of the DevOps approach comes from the automated test capability. Security like any assertion we make in IT needs to be tested. In a world of rapid release cycles it needs to be tested regularly and often. If security is to be everyone’s responsibility then those responsibilities need defined by the people who will take the responsibility and they need the concomitant authority to go along with that responsibility. If that starts with strategy then that responsibility starts with the CTO. That responsibility can filter through the organisation with guidelines, plans and finally implementation steps. By going through each step in order those responsibilities receive further definition at the right level. Senior managers and directors can own the processes for formulating and updating guidelines. Project managers and scrum masters can own the implementation plans and engineers can own the implementation itself. Each group taking and owning appropriate levels of responsibility and authority. Finally by segregating teams and projects by skill-sets we miss out on amazing opportunities to further improve security and productivity. The choices each team makes can have huge consequences on the work of the others. There are also amazing opportunities for each team to help each other for very little additional effort. The whole IT security process can be a good and positive experience for everyone in the organisation rather than an exercise in frustration, negotiation and excuses.

If you’d like to read more you can find my book Next Gen DevOps: Creating The DevOps Organisation on Amazon: