What’s next for NEXT GEN DEVOPS

You know I never actually intended to write a book. That sounds strange doesn’t it? It’s not like I tripped one day, fell on my keyboard and NEXT GEN DEVOPS: Creating The DevOps Organisation appeared on the screen.

When I was trying to bring DevOps to EA I read a bunch of people’s DevOps manifestos. I read a lot of blog articles about tools and technologies. I read about how Development and Operations could collaborate, I read about full-stack developers and of course I read about Netflix and Etsy’s successes. I spent days researching DevOps and in all that time and in all those articles I didn’t find a structured approach I could hand to my boss that would help her understand what she might be getting into and what advantages it might bring to our organisation and the business as a whole. I didn’t find a framework of strategies that could be moulded into EA’s existing structure that would bring immediate benefits. I didn’t find a step-by-step approach to moving to DevOps.

When I left EA it was my intention to create such a framework. When I started work on it though I found myself writing pages and pages of exposition. Every time I presented a concept in the structure of a framework I felt I needed to justify each idea. After all my approach has evolved as the industry and my career has evolved these past twenty years. It’s difficult to sum it up in a couple of pages. After just a couple of weeks of this I decided that I should write the book then the framework.

Now when I publish the framework the book serves as the exposition. If someone wants to understand where an idea came from I can refer them to the book. If someone wants examples, they’re in the book and more can be easily published, supported by the book.

The book has been available for six months now. In that time I’ve sold something like 140 copies. I’ve presented themes from the book to a variety of different audiences, fielded questions, received feedback and reviews and It’s now time to start work on the framework.

The framework will be open source and free. By that I mean I intend to publish it for free and make it available in an online source control system so that others can contribute to it. I don’t yet know what kind of structure I’ll need to put in place to manage contributions (or even if anyone besides me will want to contribute to it).

To get people started I intend to begin the project formally by publishing the structure of the framework. My main inspiration for the structure will be ITIL (specifically the old service management and service delivery I books I studied). I’d then like to start working with a wider group of people on the details. If you’d like to contribute or offer help then please drop me a line: grant@nextgendevops.com

DevOps @SyncHerts Thursday April 2nd

I’m excited to announce I shall be presenting at SyncHerts this Thursday evening. I shall be talking about DevOps in general terms and the framework I use for implementing DevOps and I intend to announce my new project.

The book Next Gen DevOps: Creating The DevOps Organisation has been available for 6 months now and has been almost entirely well received so it’s now time to announce the real reason I wrote the book and what comes next! It’s exciting I hope you’ll join me.

A comparison of monitoring services for an online ad service

I have a bit of an unusual entry for you this week. I’m doing a little work with a small online ad service business on a new project. The challenge has been to choose technology solutions that allow the agency to deploy it’s expertise without needing a large technology team of it’s own and that are completely transparent to our partners and clients. By that I mean I want to provide solutions that provide impartial reporting and if possible API access to the service so no reconciliation is needed and we can minimise middle-man problems. With the wealth of software-as-a-service solutions available at the moment the challenge has actually been to decide which solutions to reject rather than trying to decide what to use.

The latest challenge is availability and performance monitoring. In my last company, Connected Homes, we had a hybrid monitoring solution using Pingdom and Dataloop.IO. Pingdom was a great solution for providing impartial uptime statistics on the external facing sites so that when we achieved 100% in a given month I wasn’t accused of fiddling the figures :).

In the project I’m working on right now I want to go a step further. I want to be able to provide our partners (the sites we publish ads into) and the ad networks and clients (the people who request we run an ad for them) full visibility into our availability and performance.

I’m trying to establish partnership relationships with everyone involved. After all if we all do our job well then everyone benefits, including the visitors to the sites. If any one of us under performs we all suffer.

There seem to be hundreds of monitoring options out there but from talking to friends and doing a little research of my own I’ve settled on five to choose from, they are:

They all offer slightly different services and so before I start comparing them I should specify what I’m particularly interested in:

  1. I’m just looking for a simple HTTPS Get. All our content is hosted on Rackspace’s CDN, Cloudfiles or on a 3rd party ad-server such as Google’s Doubleclick so I have no server monitoring to worry about. I only really need to make 1 GET request per partner because all the variables beyond that are stored in the ad-server itself. I assume that if I get one response back from the ad-server I’ll get a response to any query. If this turns out to be false later I’ll need to update my test but so far it looks true.
  2. Our partners often have global solutions and so I need to monitor availability and performance from different locations around the world. I don’t need to know exact performance in each major city so as long as I can do a test from most continents I’ll consider that sufficient.
  3. I would like (but it’s not completely essential) some level of application performance monitoring. Online ads are relatively simple get requests (that lead to a few more gets) so I’m not looking for real APM but if a particular aspect of the ad-serving process is slow I want the data to know what the problem is.
  4. I need the monitoring solution to provide an API so our partners can pull monitoring data directly into their own monitoring solutions should they choose to.

The business is just starting and they only have a few partners (although several are very large) signed on at the moment so I’ll start by assessing the costs of 3 partners (which equates to 3 ads given each ad is just a different query to the same ad-server). We’ll then scale to 6, 12 and then 36 partners to see how prices scale. I have decided that monitoring every 5 minutes will be sufficient which means each partner will require 8640 tests per month (assuming a 30 day month).

Each service charges differently and offers slightly different features at different price points so we’ll have to make certain assumptions as we go along.

Now that I know the features I need let’s see if any of my requirements cause problems for any of the  monitoring services.

Pingdom

Pingdom list four price ranges on their site, they’re all priced monthly with a discount for annual subscribers. I’ll start by assuming the business wants to pay monthly. Pingdom then allows users paying more to have more checks, more RUM sites (Real-User stats), more page views, more SMS’s and the like.

Pingdom offers to test site performance from multiple regions but only actually offers Europe, the US and Australia, that’s not a bad spread but I know some of our partners have significant interests in the far east so it’s far from ideal. However Pingdom isn’t out of the running yet because it offers Real-User-Monitoring. By adding a little Javascript into my pages I can collect performance data from the actual users of the sites. In some ways that’s better than synthetic test data but it does mean I now need to pay enough that each of my partners can get sufficient RUM sites. This pushes me almost immediately into the higher price bands.

Pingdom offer website performance monitoring.

Pingdom_Free_Perf_test

In the case of simple web applications like mine this provides a nice little performance breakdown similar to those available from within a browser. The advantage here is that the test is performed from one of Pingdom’s test sites.

While Pingdom offer access to their API at all price levels access to that API is by using the main account’s Email address and password. Clearly I can’t have our partners sharing accounts with us so I now need to pay extra for multi-user access to the API. This now means for any size monitoring task I need pay at least £139 per month for the professional access. I would have needed this when we get 12 partners anyway but it’s a shame that I’ll need to pay that price even when we just have our first 3 or 6 partners.

Statuscake

Like Pingdom Statuscake list four price options on their site. With Statuscake the higher price rangers allow tests to be run more often from every minute to every 30 seconds to constant. The higher price brackets allow access to more test locations, more SMS credits and access to additional features which we’ll get to later.

Statuscake offer a wider range of test locations than Pingdom they provide a nifty little map showing the locations available.

Statuscake_map

At the time of writing they have 118 monitoring locations according to their API. Although it should be noted that only 8 of these are available at the lowest price point, it isn’t clear whether you get to choose which 8.

While Statuscake make a point of specifying that their web tests are performed with a webkit they don’t offer a real-user reporting feature like the one Pingdom provide. In my particular case I’m less concerned with real-user reporting than I am testing from locations that are important to my partners. I only mention it here because if I choose Pingdom real-user monitoring is the only way I’ll get adequate test coverage whereas I don’t have that restriction with Statuscake.

API access is with a unique API key. Clearly this isn’t something I want to give out. Statuscake offer a 3rd party access mechanism using api service keys and web hooks to allow users to integrate with other services like Pagerduty. This mechanism should allow me to provide partners to query test test status.

All these features appear to be available at the lowest price point.

Obvious.io

Obvious.io take a different approach to pricing than the other monitoring services in this group. They simple charge $0.002 per check. On the one hand I love this approach, on the other it’s feels very restrictive at scale. Then again I don’t think these monitoring services are really aimed at large scale enterprises.

Obvious.io claim test locations across the USA, Europe and Asia but provide no details. Assuming their infrastructure is hosted in AWS or a similar cloud infrastructure this might be sufficient for my purposes. It’d be nice to have a little more information about where they can monitor from though.

Obvious.io brings a new feature to the table that might actually prove very useful for online advertising. Obvious.io offers visual website monitoring. Obvious.io can compare regions of a website to a snapshot it takes at the time the monitoring is configured and alert if it looks different. This would only be only useful for the static parts of the page but online ads tend to be updated every two weeks so I could set up a monitor for each campaign to ensure the right ads are being served at all times. This could be a great solution for direct campaigns. It’s probably not such a great idea for programmatic campaigns.

Unfortunately Obvious.io makes no mention of an API so it doesn’t look like I can give third parties access to monitoring data. This rules Obvious.io out for me but it might still have a secondary use if I want to set up the visual monitoring for direct ad campaigns.

Monitis

Monitis, the premium version of Monitor.us has a very nifty pricing calculator so rather than providing various  products at fixed prices you can configure exactly what you want and discover what the cost will be. This was very useful for me because it allowed me to tailor the service exactly as I wanted it. The downside is that they don’t allow any very many monitors and sub-accounts before deciding that you’re not allowed to use the calculator anymore and you need to contact them.

Monitis offer a huge number of monitoring locations and as well as specifying them on their site they even provide you a handy guide of the IP address each monitoring location uses: http://www.monitis.com/support/tools/our-ips/. Monitis also offers real-user monitoring, they have standard plugins for WordPress, Joomla or Drupal but for everyone else it just requires pulling in another Javascript from Monitis.

Monitis also offer excellent online help for setting monitors up, in this guide when I was looking into how to set up an HTTPS check I also discovered that this monitor can also check for a text string in the source code for the site. This is nice because you can even check that your whole page is downloaded.

The Monitis service includes a REST API which supports using an API key and secret key or authentication but goes a step further and allows the use of public keys this should mean I can allow 3rd parties to query my monitors and get test results for themselves. The API document also appears excellent at first glance.

Site24x7

Sie24x7 returns to the classic 4 tier pricing model and again each tier increases the number of monitors allowed, the polling frequency and provides access to more features.

Site24x7 don’t seem to specify exactly where their test locations are on the site. They do comment in their FAQ that most pricing options offer 8 locations. Site24x7 offer a nice little set of free tools and playing with the ping and website availability tools tells me that five of their locations are probably USA, Canada, Singapore, India, Australia this alone is quite a nice spread. It’d be nice if they also had somewhere in Europe and Japan. These tools also tell me that my readers in India are not getting the performance they should be.

Site24x7_ping_loctions

Site24x7_website

On the subject of free tools Site24x7 offer so many that I felt they had to be applauded for their work. Along with a ping and website availability check they offer webpage performance analysis, DNS checks, IP resolution, geo-ip checks, port availability, tracerouting, SSL certificate monitoring, heart bleed vulnerability checks, Poodle vulnerability checks, and IPv4 and IPv6 subnet calculators and that’s just the System Administrator tools! They also offer validation tools, content tools and developer tools!

Site24x7 also offer a REST API they even go so far as to comment that one of the use cases they’ve designed their API for is to allow service providers to provide custom monitoring services to their clients. They do this by allowing the generation of multiple API keys. I hope the different API keys can be given different permissions but I couldn’t find a note to that effect in the documentation but given it was designed for this purpose it would seem stupid not to have this feature.

Price

Each of the monitoring services is priced differently and aimed at slightly different use cases. All we can do is consider our specific case and try and find the closest fit. The last piece of this particular puzzle is price. Not being an expert in each of these services I’ve done my best to determine the options I’d need to select in order to get the service level I’d like. For the sake of brevity I won’t detail every choice and option here’s how it breaks down.

Screenshot 2015-03-20 10.32.42

Screenshot 2015-03-20 10.33.14

As expected Obvious.io gets very expensive at scale. The price for Pingdom is artificially high because I included real-user-monitoring on every test. This was necessary because their test locations are so limited.

Summary

Pingdom

I don’t have the option for using real-user monitoring at this stage of the project so that rules Pingdom out straight away. That’s a shame because I’ve had a really good experience with Pingdom in the past. The reality is that the web is global and not offering any monitoring options on the Far East or India just won’t cut it anymore.

Statuscake

Offer a really good fit. The pricing might be artificially low because their £14.99 price point only offers 25 SMS’s and while I don’t expect many failures I can see that being consumed really quickly if we make a mistake implementing a new project. Upgrading to £49.99 per month would then provide unlimited SMS and more test locations and being Statuscake’s price more in-line with some of the other providers in this assessment. However the £14.99 price point is very tempting for the early days of the project.

Obvious.io

The ever-scaling price and lack of an API rule Obvious.io out for the primary use case but I think the visual website monitoring could be useful. It may be a suitable add-on for use later on. However for now it doesn’t make the cut.

Monitis

Monitis were the surprise inclusion for me. I’d never heard of them before, they turned up in a Google search and I included them because they looked to be doing the right sort of thing. It’s a shame their online price calculator only allows relatively small scale because it’s difficult to tell whether prices would keep rising. As it is for the purpose of this assessment I can’t take them any further because I just don’t know what might happen to the price at higher-scale. If I struggle with implementing whoever the eventual winner is I may return to Monitis and give them a call simply because of the quality of their documentation.

Site24x7

I haven’t used Site24x7 before but they were recommended by a friend, ex-colleague and brummie I trust. I wish they provided more information about their test locations and the permissions management options for their service keys. It all sounds good and the pricing is good but I do have some questions unanswered.

Choice

There may well prove to be problems during implementation, either my starting assumptions will be incorrect or API access won’t work as I hope it will so I’m going to choose an order of preference rather than one clear winner at this stage.

  1. Statuscake
  2. Site24x7
  3. Monitis

I hope this has been useful for you. If anyone has any information about any of these providers that they think would benefit readers or help me in my selection and implementation then please leave a comment or send me an email grant@nextgendevops.com

DevOps for Recruiters @TalentWaffle

On Thursday 5th February I gave a very short presentation at a TalentWaffle event organised by Reconverse. TalentWaffle is a meet-up for recruiters in the London tech start-up scene, with guest speakers, solution demos, and the chance to connect & converse. It was a great event, I met a lot of interesting people and was able to share a few pro-tips about hiring people for organisations looking to get into DevOPs. My presentation was about how to engage, recruit and retain engineers.

I only had a 15 minute slot so my content had to be very focused. I didn’t know how au fait the audience would be about DevOps so I actually prepared two presentations. One was an introduction to DevOps with a few basic tips to help recruiters understand what they were getting into if their technology departments started asking for DevOps people. I also prepared a more advanced presentation aimed at those recruiters who were in organisations that were already using DevOps.

The event wasn’t videoed so I thought I’d share my presentation in long form here, that way the three recruiters who were interested in the more advanced content could get to see it :).

So without further ado let me present:

Talentwaffle presentation Talentwaffle presentation (1)

One day I will do a presentation in the style of a choose your own adventure but 15 minutes didn’t really provide me with the opportunity. As such I explained to the audience that the topics in white were the beginners guide content and the topics in blue were the advanced topics. As I have a little more time here I’ll just run through the slides in order.

Talentwaffle presentation (2)

Yes, yes this is the obligatory “What is DevOps slide” but in this case this slide gave me the opportunity to convey a specific message to the recruiters in the room. This message was pushed further by Ken Ward and Matt Buckland later in the evening. Recruiters are in a position of great power and responsibility and unlike Spiderman they don’t have to just whinge and cry about things like an angsty teenager. Recruiters have a huge influence over the attitude and culture of an organisation because they present candidates for interview. If their CTO is asking for DevOps people the recruiters are in a position to find out whether the organisation is really ready fro DevOps or whether the CTO just thinks DevOps is the latest name for System Administrators.

Talentwaffle presentation (3)

This is the first of the more advanced topics. If I’d had the opportunity to present this slide I would have explained to the audience that engineers capable of contributing to a DevOps environment are rare and so they need to be sought out. It’s not enough to release a job spec into the wild and wait for them to turn up. Recruiters need to get on Github and look for the contributors, they need to follow the bloggers and build a network of contacts.

For recruiters in start-up organisations looking for one or two engineers this is impractical. The only alternative if the current employees aren’t able to suggest anyone is to go to the agencies. There are a few recruiters who are making names for themselves in the world of DevOps recruitment these people are making it their business to understand DevOps, the field, the people, the roles and the organisations. If anyone needs recommendations of good external recruiters then please get in touch.

The second major point I wanted to get across with this slide is how to get the attention of those good DevOps people. Every job spec talks about technology as if the particular tools an organisation uses is important. The field of DevOps software is incredibly fragmented and evolving rapidly. This week’s hot software will soon be old news. There are a few packages that are starting to mature and look like good solid options but they could well be eclipsed tomorrow.

Good engineers are far more interested in how engineers collaborate within an organisation to achieve their goals than they are in the tools and technologies used. Nothing is cooler than seeing what happens when motivated and talented software, testing and operations engineers are allowed to focus on a common goal and work together. That’s exciting, that will get engage people.

Talentwaffle presentation (4)

The UK has a strong and vibrant contractor market but some CTO’s seem obsessed with finding permanent DevOps people. I’m asked about it all the time. I’m told that permies have more “skin-in’the-game” how they’re more committed. This doesn’t match with my experience. Good engineers are committed and know they’re only as good as their most recent success. It doesn’t matter whether they are salaried or if they invoice.

To be capable of building, launching, managing and supporting services using DevOps methods and tools an engineer has usually been through the wringer. They’ve worked in environments where the CTO is an ex-developer and the developers leave at 1700 on a Friday having supplied software that won’t deploy at 1600. They’ve been expected to be on-call 24x7x365 for no extra pay and been blamed for live service incidents that weren’t their fault. So when the opportunity comes up to do what they love for £60,000 or £550 per day the choice is fairly straight-forward.

If it’s culturally important to your organisation they you have permanent engineers then grow your own! Encourage your organisation to invest. Hire a couple of good, senior, contract engineers and then bring in young people who haven’t yet learned any bad habits. Hold a hackathon to look for talent, encourage hobbyist engineers looking for their first step into IT and grow your own DevOps people.

Talentwaffle presentation (5)

Engage? Check! Recruit? Check! On to retention! I’ve spoken to a lot of organisations and recruiters since I launched my book last year. I’ve heard a lot about organisations struggling to retain engineers. I’ve seen some of the issues among my friends and former colleagues.

In almost every example the issue comes down to organisations not understanding that without collaboration there is no DevOps. Organisations that attempt to bolt a DevOps team onto their current organisation can only fail and make everyone miserable along the way. Creating a DevOps team to manage deployment automation while software engineers continue to design services with no input from the people responsible for operating or testing them just doesn’t work. Oh sure you’ll get away with it for a while but the approach you used to build software is not the approach you need to build a service. Services live they don’t just launch, they need support and management and optimisation. This isn’t done by presenting different challenges and incentives to different teams of people with different skill-sets and experiences. Online services flourish when talented engineers with different knowledge and experience work together to design, build, launch, manage and support services. Nothing is more inspiring or motivational than being part of team of talented and passionate engineers all focused on the same goals. If your organisation pays well, encourages real collaboration and hires talented people you will have no retention problems.

Talentwaffle presentation (6)

That’s engagement, recruitment and retention out of the way now I wanted to focus on a couple of pro-tips.

I love it when job spec’s list, Chef, Puppet and Ansible. I understand that the organisation is probably trying to convey that they want to use one of those tools and hasn’t decided which one yet. To many engineers that looks like either the organisation is using all of them. If that’s the case that company are a car crash that no-one wants to be a part of. Failing that the recruiter has no idea that all those tools do almost exactly the same job, in which case the organisation may be clueless and no-one wants to be there either.

If you want to make your job appear attractive and stand out from the crowd of DevOps job descriptions describe how your organisation collaborates. Do you eschew the old paradigms of teams based around skill-sets? Do you matrix manage teams around products? Sell people on how they’ll work with their peers to achieve your company’s glorious vision. Remember engineers love solving problems, they love making the world a better place. They don’t care whether they need a spanner or screw driver to solve the problem.

Tell prospective candidates the problems your organisation faces right there in the job spec. Make it clear that you and your organisation are honest, introspective and forthright. Those are all qualities almost universally valued by engineers.

Talentwaffle presentation

I wanted to finish with something that would help recruiters present good candidates to their hiring managers. I wanted to help them identify engineers who are best suited to working in start-ups or newer business.

The main point is that good engineers that can work in collaborative and supportive teams are passionate about what they do and how they do it.

Ask them about something they built a long time ago that’s still running, ask them why it’s still useful. Do they get passionate? Can they laugh about a horrible hack they were forced into that worked so well it’s still running years later?

Ask them about what language they like to code in? Ask them why they care about it. It doesn’t matter if you don’t understand the answer but you’re looking for an answer that conveys that engineers passion for their profession.

Ask them about a time when they worked directly with software and test engineers to achieve something bigger and better than they could do on their own. Do they have the excitement and sense of wonder that comes when you see teamwork actually work.

Talentwaffle presentation (1)

Finally no Next Gen DevOps presentation would be complete with a shameless book pitch, so here it is: PLEASE BUY MY BOOK!

Thanks.

What can DevOps do for security

At the end of last year two security breaches made all the headlines. JPMorgan announced that information about 83 Million accounts had been obtained because a single server lacked two-factor authentication. Sony Pictures fell victim to a Malware attack that allowed attackers to access corporate emails and employee information. Then over Christmas the Playstation Network and Xbox Live were taken down by huge DDOS attacks. Since those attacks I’ve read a few blog posts presenting certain tools organisations can adopt to improve security. The mainstream IT press have focussed on the processes and procedures organisations should have in place in the event of a security breach. There’s a pattern to this reporting and it mirrors the patterns I see in IT security in most organisations. There’s a great deal of focus on technological solutions and a lot of focus on what to do once an incident occurs but very little focus on how organisations should behave day-to-day to improve their security. Unfortunately this situation has necessitated a very long article so if you’re short on time you can skip directly to the conclusion.

The perennial IT security problem

It’s been my experience that IT security is often considered as an additional set of tasks, separate from day-to-day work. In smaller businesses security is just considered another responsibility of the IT engineers in larger firms security is often devolved to security departments while management reiterate that security is everyone’s responsibility and include a couple of paragraphs in the IT systems policy in the employee handbook.

The security department, devoid of any real authority over technical implementation define their role as providing audits and reminding engineering teams of best practices. Engineering teams are then left to implement solutions in isolation while trying to keep on top of all their other work. Development teams are often given little guidance and so are left to do their best to secure the applications they create and the only time they’re given any feedback is after a penetration test is performed or an incident occurs. If I might paraphrase The Incredibles if security is everyone’s responsibility then it’s no-ones. Security is often considered only at the beginning of a project and then ignored until someone decides to run an audit or there’s a security breach. In smaller businesses it is sometimes just ignored altogether in favour of building the business faster. Unfortunately when it’s time to address security again the task is herculean because the number of systems and services has multiplied. In one company I worked at the security department ran an audit on some servers and presented the engineering team with a list of 19,000 recommendations. Clearly this was the output from some automated tool. The various recommendations weren’t prioritised. No attempt was made to rationalise the actions to determine if any tasks needed to be performed on every system. The security team felt they had done their job by highlighting the problems and they moved on, they didn’t even try to work with the engineering teams and their customers to try to get the security work prioritised alongside the engineering team’s other work. Then on a regular basis they’d send an email to the manager of the team to ask how the actions were progressing. This is the very definition of activity rather than progress. A box was ticked and nothing significant was improved. It seems to me that security is another area where DevOps can make a difference. By breaking down silos, fostering collaboration and providing measurable improvements a DevOps approach can turn security from activity into progress. While the ideas I’m going to present below certainly can be useful in organisations not following DevOps practices I think they are almost infinitely harder to implement in a non-Agile, non-iterative, silo’d world.

The security strategy is vital!

Security starts, as all large initiatives should with a strategy. The trick then is to use that strategy to create tactical approaches that can in-turn be used to create action plans with measurable outputs. In my book Next Gen DevOps: Creating the DevOps Organisation I devote an entire chapter to creating strategies and using them to create tactical approaches and detailed plans. I won’t repeat that entire chapter here. What I wanted to do with this article was provide a more specific example, using security, that demonstrates how IT leaders can drive forward the security agenda, liberate improved productivity and make security everyone’s responsibility while at the same time ensuring everyone in the IT organisation understands their responsibilities to the security agenda. I regard a strategy as being: “…a statement that describes how goals will be achieved and success will be determined. It should, for clarity’s sake, include a statement about what those goals are, but the important word is ‘how’.” A typical modern IT business provides ongoing services to it’s customers. It stores some sensitive information about those customers including their name, address, phone number and other personally identifiable information (PII). In addition it will probably store a transaction history and provide some secure purchasing method. This could describe a taxi firm with an online booking app, or a utility provider with some online services or even a games company. A suitable strategy might be expressed thus: “We regard our customers data as important as our own and will treat it as such. We will permit only those accesses that are absolutely essential. We will safeguard this data and will follow the latest best practice guidelines for data storage and systems and network access. While there is no end to security provision we will deem ourselves successful when only those accesses that are absolutely necessary are present and every failed access attempt is audited.” While I admit this strategy is a bit wooly and vague because it’s just an example one it fulfils the basic criteria. It states what our goals are, how we’ll achieve them and how we’ll know when we’ve been successful. The next stage is to get the heads of our department and some senior people together to create the tactical approaches so that our strategy can become plans. In the case of security these tactical approaches are should be guidelines that describe the tools and technologies we’ll use and any configuration options we consider essential. The guidelines may also define any processes we need such as change management or how we’ll handle access requests.

What comes after the strategy?

This is where DevOps can vastly improve our productivity. Rather than considering the domains security needs to be applied to as skill-sets i.e. Data centres, networks, systems and applications DevOps can help us take a product approach. By having our network, software, operations and test engineers collaborate we can look at how our important data is used, how it moves, where it’s stored and who needs access to it and what they need to do with it. By defining our domains this way we don’t risk having tasks fall between the cracks that exist between silos. We also have the opportunity to make decisions that improve security and reduce the workload and time to implement by allowing the engineers from all the disciplines work together. It’s not that this isn’t possible in a traditional environment but negotiating priorities with the half-dozen different managers, building support, trading favours and then trying to bridge the responsibility gaps between each group absolutely kills productivity and often causes larger initiatives to stall indefinitely effectively stopping them completely. It’s then quite likely that certain tasks will require multiple groups to work together and here again DevOps smooths that process too. By focussing on the task and what a successful implementation of that task looks like the organisation is far more likely to actually achieve the desired result rather than having to compromise the implementation. Customers usually provide their data to us using their computer and an application we have created. So we need a secure connection to their computer. We also need a secure application, hosted on a secure system on a secure network to make that connection to their computer. Secure in this case meaning many things but starting with only making or allowing those connections that are absolutely necessary. Getting network, software, system and test teams working together looking at these requirements we can determine the most efficient route forward. It may be obvious to that group that a browser making an SSL connection to a web server requires the least effort and provides an adequate level of security. That then informs the network engineers that a certain type of traffic will need to be allowed for. It also informs the system engineers that certificates will be required and some web server software is needed. It also informs them that certain ports will need to be opened to receive connections from certain protocols. Test engineers now have a whole bunch of things they can test for the presence of and the absence of. This may lead the group to conclude that they need a tool for managing ACLs, that team has the right mix of people to determine if such a tool will be useful and how such a tool can be built, hosted and tested and how that capability can be monitored. Even at this early stage it’s now obvious that guidelines are needed for network connectivity, systems security, SSL connections and web accessible applications. These guidelines provide the tactical approaches that define how security will be provided. These guidelines can mandate certain operating systems and the mechanisms that will be used to ensure they are secure. They can mandate how systems receive internet connections and the firewalls and ACLs that will be in place on the routers and switches between the internet connections and the systems. The guidelines can mandate the Certificate Authorities that will be used and the types of certificates that will be used. They could also mandate particular applications and where to get verified versions of those applications. They might even mandate particular configuration options the group agree are necessary. We now have practical pragmatic guidelines, created by people whose teams will be a part of the actual implementation. They own these guidelines they understand the decisions that led to them and as a result they are responsible for them and for abiding by them. With the guidelines in place projects can now be formulated to provide the solutions necessary. Networks and systems can be designed and built.

From strategy to guidelines to a plan!

The strategy and the guidelines provide sufficient information to allow a project manager or scrum master with sufficient understanding of the product to work with the engineering teams to create plans. If the teams in the organisation are arranged by skill-set rather than product focus there will still be the problem of prioritisation of tasks. It’s still possible for certain teams to delay the project because of their prioritisation choices. At least tasks can’t be missed though. Because the project is focussed on the successful execution of certain tests the senior leadership of the organisation can focus on those tests. That allows prioritisation to be very easily executed. They are either prioritised or not and leadership carries the responsibility for those priority calls. If the operations team are slow to update the configuration management system then the data cannot come in to the organisation securely. There’s no ambiguity there. If the developers chose an implementation method that passes the data insecurely the tests will demonstrate that and the test will not be passed. This gives the organisation a myriad of options, they can create tiger teams, they can hire in temporary staff or they can reprioritise certain work. By using DevOps methods for security implementation each assertion is tested for. Those tests are automated and can execute against the development, test and live environments every minute if so desired. This ensures that no regression bug can break the security mechanism.

Conclusion / TL:DR

As usual the real value of the DevOps approach comes from the automated test capability. Security like any assertion we make in IT needs to be tested. In a world of rapid release cycles it needs to be tested regularly and often. If security is to be everyone’s responsibility then those responsibilities need defined by the people who will take the responsibility and they need the concomitant authority to go along with that responsibility. If that starts with strategy then that responsibility starts with the CTO. That responsibility can filter through the organisation with guidelines, plans and finally implementation steps. By going through each step in order those responsibilities receive further definition at the right level. Senior managers and directors can own the processes for formulating and updating guidelines. Project managers and scrum masters can own the implementation plans and engineers can own the implementation itself. Each group taking and owning appropriate levels of responsibility and authority. Finally by segregating teams and projects by skill-sets we miss out on amazing opportunities to further improve security and productivity. The choices each team makes can have huge consequences on the work of the others. There are also amazing opportunities for each team to help each other for very little additional effort. The whole IT security process can be a good and positive experience for everyone in the organisation rather than an exercise in frustration, negotiation and excuses.

If you’d like to read more you can find my book Next Gen DevOps: Creating The DevOps Organisation on Amazon:

NextGenDevOps_Book_Cover_500

Horrible HR Policies to Nuke in 2015

I was going to present a DevOps perspective on the high profile security problems that have been making the headlines recently but that post needs more work and I just read an amazing article by Liz Ryan that I wanted to share with you and comment on. Liz’s article Horrible HR Policies to Nuke in 2015 is fantastic. I couldn’t agree more with all her points.

I’ll give you a few minutes to read it, it won’t take long.

Are you done? Good wasn’t it? It may surprise you but I wrote about the damaging effect HR policies have on IT in my book Next Gen DevOps: Creating the DevOps Organisation. When the idea of the book was still forming I had a intuitive sense that DevOps would have a bigger impact than just improving products and services. I felt it would improve the lives of the people in our IT organisations. I decided to go back to Management Theory basics and see if those old tried and true management theories supported DevOps. They not only sit well with DevOps they actively encourage it. If you want to know how buy my book :). One of theories that I think supports the case for DevOps was Herzberg’s Motivation-Hygiene Theory.

Herzberg’s research led him to conclude that the factors motivating employees usually stem from the work itself, or effects arising from the work. In contrast, he found that dissatisfaction stems from the work context. He found that achievement, recognition, the work itself, and responsibility and advancement were factors that cause the greatest sense of satisfaction or motivation in employees. The causes of dissatisfaction that de-motivated employees included company policy and administration, relationships with managers and business leadership.

In other words when people can get their job done and be recognised for it they are well satisfied and will become highly motivated. This supports the case for DevOps because DevOps breaks down the barriers between technical teams that prevent progress.

To Liz’s point way back in 1959  Herzberg knew that company policy, administration and the relationships with their managers were the biggest sources for dissatisfaction for employees and hence led to demotivated people.

Enter Liz Ryan’s excellent article. Look at your businesses. Look at the policies, processes and working practices that exist to help people get their work done, improve the product and bring out the best in your people. Then look at those policies that require you to force people to participate that sap the energy and the life from your people such as end-of-year-reviews, employee opinion survey’s and expenses processes.

Take this new year as an opportunity to focus on the tasks at hand, ditch those things that get in your way, that slow your business down, that prevent progress and that enervate your people replace them with product focus, clear success criteria and help your people give their best for you and the business.

Next Gen DevOps Interview with Linuxrecruit’s Tony Chapman

When most people hear that I’ve written a DevOps book they ask me what the book is about, whether it’s a technical book and why I wrote it. When Tony Chapman from LinuxRecruit heard about my book he decided to go one better than just asking me some questions. Tony hired a cameraman and invited me down to his office. Tony and LinuxRecruit are right in the middle of the UK DevOps scene at the moment. Linuxrecuit co-sponsor the London DevOps Exchange #DOXLON alongside my friends Dataloop.IO. #DOXLON is the biggest DevOps group outside of San Francisco.

If you’ve ever wondered what my book Next Gen DevOps: Creating The DevOps Organisation is about and why I wrote it then check out the video on Linuxrecruit’s blog site or just watch it below. If you like what you see and you’d like to buy the book you can find it on Amazon here: http://amzn.to/Zn2SKY