Prior to the burst of the .com bubble the IT industry was obsessed with project management. Our projects always over-ran, they rarely delivered on their promises and almost never met the expectations of the sponsors. We looked to project management to help us improve the situation. The v-method saw a brief resurgence in popularity. Some companies built powerful project management departments. Some businesses tried spreading project management skills throughout their organisations. We tried both of these approaches at AOL and neither particularly worked.

Then after the .com bubble burst in 2000 another solution to managing software projects was proposed and the Agile movement was born. The Agile movement was so successful that I haven’t worked in an organisation using any other method for over a decade now.

The Agile movement was first and foremost a cultural movement albeit one that was augmented with tools. Agile asked that project sponsors work closely with development teams. Agile asked developers to deconstruct their projects and develop and demonstrate smaller pieces of, still valuable, functionality more often. Agile encouraged developers to use this capability to rapidly incorporate changing requirements into their projects.

If I may be so crass as to quote from the Manifesto for Agile Software Development itself:

“We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools…”

Automated build and test tools, continuous deployment and integration tools and story management tools can significantly extend the capability of a development organisation but they are not essential for Agile software development. Those tools would provide value to any organisation using any development or project management process.

Following hot on the heals of the Agile software development movement came automated testing, release management and cloud computing. Meanwhile Facebook gained a billion customers, Microsoft waned, Google waxed and mobile applications took client development in a whole new direction (or should I say directions ;)).

Then came DevOps. DevOps was founded by Patrick Debois and a small group of frustrated developers and system administrators. Their frustration was aimed at how difficult it was for development teams to work with operations teams in their organisations. The pace, culture and methods of both teams seemed to be completely at odds with each other and testing was stuck in the middle. Their idea and hashtag, #devops, captured the imagination of engineers, leaders, luminaries and journalists alike.

The growth of DevOps has had an interesting side effect. There has been an explosion of operations tools and products on the market. Prior to DevOps every time I joined a new organisation I’d have to build another configuration management system and more deployment tools and monitoring solutions. Now the problem I face is which solution to choose. There’s still some significant development work required to make all these tools work together but at least I don’t have to build them all as well.

Just as story tracking tools, burn-down charts and continuous build tools can assist teams doing Agile software development so too do configuration management systems, containers and monitoring tools assist with DevOps.

Successful online services aren’t built overnight. They aren’t the result of one heroic development push. They are built iteratively. They are tested constantly. They aren’t then deployed and forgotten. Their performance in the live environment is studied. Data is collected and analysed. The results of those tests and the data collected is then fed back into the development process and the service is iterated on. Sometimes new features are added. Sometimes performance is improved. Sometimes more data collection is enabled. But always the service lives and grows.

Do you think you can do this at pace if development, testing and operations sit in their own little silos communicating by tickets or email? Do you think bringing operations and testing in on a stand-up or two or inviting them to the odd planning session will make this happen?

The value DevOps brings to an organisation is in fostering collaboration between software development, testing and operations functions. That collaboration comes from cultural changes not just tools adoption. Not just a couple of meetings. Whether you forge that culture by creating multi-skilled teams, as I propose in my book Next Gen DevOps or whether you encourage collaboration with processes, goal-alignment or a matrix management (ugh) it’s the culture of collaboration that’s important.

The reality is that developing modern online services can’t be done by taking development, testing and operations in turns we do them together and succeed or apart and fail.

 

 

If you’ve enjoyed this post you might like to know that my new book Next Gen DevOps: Creating the DevOps Organisation, the indispensable guide to building and managing services in the 21st century will be available on Amazon very soon.

I’ve been doing a bit of self-promotion recently. I’m close to finishing my new book and I’ve starting putting the word around at a couple of conferences and other events. This has given me the opportunity to hang out with some mates of mine who work in recruiting. These guys work at three different agencies each of which are of different sizes, levels of maturity and have quite different clients. They all share the same problem though. Each of these agencies are all placing more people than ever before but they’re also struggling to keep candidates in roles and their rebate levels are rising. If any of you don’t know agencies have to give rebates if the candidates they put forward don’t stay in the role for long enough. The problem is DevOps.

The Stated Expectations
Organisations of all shapes and sizes are approaching these agencies looking for DevOps Engineers.

For now let’s leave aside whether someone can be a DevOps Engineer and whether DevOps is a method or an attitude or a framework or whatever. I think the fact that there are so many “what is DevOps” articles is very telling but let’s stick to the matter at hand.

When organisations ask for DevOps their job descriptions usually talk about continuous integration, automated infrastructure configuration and automated testing. They may mention Puppet or Chef. The cooler ones might mention Ansible or Salt. They’ll probably state that they need experience of LAMP or .NET stacks. They will undoubtably require experience with languages like Ruby or Java. Finally no DevOps job description would be complete without requiring some experience with NoSQL solutions like Redis, MongoDB or Cassandra. To round it all off their will probably be a requirement for AWS, Openstack or VMWare experience.

There’s nothing wrong with this. Most of the people I’ve worked with over the last 15 years would relish the opportunity to build a continuous integration system with these technologies.

The problem comes from what isn’t mentioned in the job descriptions but seems to be expected as a matter of course.

The Implicit requirements
As someone applying for a DevOps role you expect to be developing code for deployment mechanisms and automated infrastructure solutions. You expect to have to configure a lot of third-party tools like source control systems, build systems, containers and repositories. These all have to work with the deployment system, the infrastructure access mechanism and caching solutions and obviously all the security systems that sit between all of those.

You don’t necessarily expect to be on-call 24x7x365. Particularly because that wasn’t mentioned in the job description. You don’t expect to be responsible for everyone else’s applications as well as your own. You don’t expect to be responsible for defining the data retention policy and building the backup & restore mechanism and implementing a suitable testing regimen. You don’t expect to be first line support and product owner of the ticketing system and document management systems. You also don’t understand why you’re supposed to enjoy solving other people’s problems and being interrupted every 5 minutes.

However those are very often the expectations of the leadership that wrote the job description and approved your appointment.

The solution seems straightforward. The hiring organisation could simply analyse the role and responsibilities they really need and put them all in the job description. What might that job description look like? Well you’d have the usual descriptive stuff at the top:

Experienced DevOps Engineer required within a dynamic and innovative software company blah blah blah… Unique opportunity blah blah blah… LAMP stack experience blah blah blah… Agile environment blah blah blah… Working closely with developers blah blah blah…

Then the interesting bit the Key skills and background:

• Ideally you’ll have two or more years of experience in a DevOps role in a fast-paced agile environment
• Working experience with Jenkins or other Continuous Integration tools (e.g. Cruise Control, Go)
• Experience with Amazon AWS / scaling applications beyond single server deployment
• Good network management skills including firewall rule management, AWS security group and AMI management
• Network security skills suitable to classify, manage and protect PII data
• Knowledge of programming languages (Python, Ruby, JavaScript preferred)
• Proven knowledge of server-side technologies such as PHP, NginX, Node, NoSQL and how they are utilised
• Strong storage management skills across both SAN and cloud-based storage solutions
• Experience of CDN and other caching solutions such as Memcache
• Experience of managing data retention solutions and implementing suitable tests
• Knowledge of business continuity best practice and experience implementing appropriate solutions
• Document management and Ticketing systems experience including migrations, upgrades, recovery and optimisation
• Strong incident and crisis management skills
• Change management experience
• Work across multiple work streams and development projects
• Proven experience with Git, ideally you’ll demonstrate your own projects during the interview.

Wow.

Does that seem a bit much?

That has been my team’s role in my last three positions (OK Playfish used SVN not Git but you get the idea). Less than half of that was mentioned in the job descriptions I and my team applied for and all that still neglects to mention 24x7x365 Incident response, crisis management and mitigation for every product the company builds!

The Result
It doesn’t surprise me that people recruited for DevOps roles are leaving them once they realise that the job description only covered half of what was actually expected from them.

There’s been a really interested reaction to this problem. Some recruiters are trying to distinguish System Administration as apart from DevOps. From their point of view a company that asks for DevOps but wants System Administration can save themselves upwards of 20%.

Some organisations have tried to segregate DevOps from System Administration (Engineering etc…). Ask anyone who’s tried to work in those environments and they’ll tell you that more silos aren’t the answer. IT has struggled for years with developer-operations relations simply because they’re distinct silos with different and conflicting goals.

Creating additional infrastructure, DevOps and support silos really isn’t the answer.

One of the principle issues that DevOps was trying to address was communication. We need people with multiple skillets and experiences to work together to build, launch, manage and support internet products. Silos don’t encourage people to work together. They encourage specialisation within the silo but not communication between silos.

If more silos aren’t the answer maybe the answer lies elsewhere

The Solution
Agile gave us the clues to solve this problem back in 2003. The creators of the Agile movement knew that the way to prevent projects overrunning and failing to hit the expectations of the sponsors and stakeholders was to bring those people into the project team. Further they received their requirements in small, discrete, testable units and they released and demonstrated often.

What if we extended those principles to not just development but also operations and testing.

Wait.

What if we extended those requirements to the internet products as a whole. Not just the sterile application sitting in a build environment but the entire product.

The infrastructure that hosts it in every environment: development, build, test and live. The systems and mechanisms that build, deploy and test it. The monitoring and alerting systems that watch it. And the disaster recovery and business continuity systems that attempt to look after it.

Well now we have a product team that combines their different skill-sets, experiences and capabilities in the service of a product. That team can identify the technologies they need to build, launch and manage their products. They can chose the backup solutions they need, they can implement appropriate security, they can develop or hire-in the expertise they need.

Rather than trying to make one team responsible for all the technologies and support challenges of all the products let each team focus on their own products. Let them share the on-call burden with each other. Let them support each other so that when there’s an incident out of hours the team can support their colleague on-call together.

It’s often said, in IT that change is the only constant and yet we’ve had separate development, operations and test teams for over six decades. Perhaps it’s time that changed.

I explore this idea and more in my new book Next Gen DevOps: Creating the DevOps organisation soon to be released on Amazon. If you’d like to know more please contact me grant@nextgendevops.com.

While I was at AOL I took a very different approach to the other Operations teams. They always acted like they provided a generic service to content owners and developers. It felt to me like they wanted to be faceless service providers accountable only to their SLA targets. That approach always seemed inadequate to me. I think my team achieved better results for the people we were working with and the business as a whole when we engaged with them.

At Electronic Arts I spent a fair amount of my time discussing Playfish’s operational strategy with other studios and core service groups. It seemed that these groups wanted to offer more personalised services. This makes perfect sense as some games are free-to-play and their success is dependent on minimising costs and some games are multi-million dollar triple-A titles with very demanding players. None of these groups seemed to know how to offer (or in the studios case receive) that personalised approach. In Playfish this is something we did right from the beginning and as Playfish grew we adapted our strategy to match.

Around this time DevOps and Configuration Management were hot topics on every tech forum. With the advent of Amazon’s Cloud and tools like Heroku and Chef we were seeing more Developers building online services end-to-end. Adrian Cockcroft published his infamous No-Ops tweet. In Playfish we joined a number of DevOps groups and I sent my engineers out to various meets and we even presented at a couple. It became very clear very few people, besides Netflix were using DevOps at scale. Every DevOps conversation seemed to revolve around one tool or another or one service or another with no real discussion about the strategies and processes required for running live services.

So I decided to write a book about how to adopt DevOps strategies in any size organisation. I also describe why DevOps works so well not just for products and services but for engineers and leadership as well.

Next Gen DevOps will be published this summer!