Dune, Arrakis… DevOps

ajnD6Dp_460sDune, by Frank Herbert, is an amazing book and an equally amazing David Lynch movie. It’s also a phenomenal management book and contains some valuable wisdom that we can apply to DevOps.

In this article I’m going to pull out various quotes and thoughts from the characters in Dune and share with you some of the lessons I’ve taken from them.

One of the most powerful lessons I learned from Dune comes about halfway through the book. Just before a pivotal moment in the story, the lead character, Paul Atriedes’, reflects:

“Everything around him moved smoothly in the ancient routine that required no order.

‘Give as few orders as possible,’ his father had told him … once … long ago. ‘Once you’ve given orders on a subject, you must always give orders on that subject.’”

As a leader this advice has been invaluable to me. A team that understands the values of it’s leader and the behaviour he or she expects from them far outperforms a team who are given constant instruction and supervision.

Not only does this hold true for managing teams but also for DevOps. When infrastructure is considered as individual machines that need care and maintenance it encourages us to give constant orders to them. Permit this person to issue this command, archive this log at this time, execute this command etc… Considering infrastructure as a system composed of identical nodes encourages us to create behaviours that the system as a whole conforms to. This is modern configuration management in a nutshell. Any nodes that don’t comply are simply replaced when they display behaviour out of the ordinary. This makes it easier to manage all the different environments the same way making application behaviour more predictable this then reduces the time spent diagnosing why problems occur in one environment and not another which in turn reduces the time needed to implement new features.

What Paul’s father, the Duke Leto Atriedes knew, was that people, whose loyalty he had earned, respected his values and wanted to work according to his values. What I’ve learned is that systems whose behaviour is managed are more reliable and require considerably less maintenance than systems that are poked and prodded at.

Dune even goes so far as to offer us DevOps axioms:

Earlier in the book Paul is being tested by one of his mother’s teachers and she tells him:

“In politics, the tripod is the most unstable of all structures.”

She’s referring to the major structures of government that make up the world of Dune.

However there’s a lesson for us in Technology here too. The three great competing forces in a typical technology department are Development, Operations and Testing. Just as in Dune there can be no stable and productive arrangement of these three groups.

Development are incentivised to change and grow capability. Testing are at the whim of development but are often underfunded and unloved. Operations are incentivised by stability and predictability and often find take a cautious position that plays well to the needs of a test team always looking for more time. If testing receives funding then they form a power block with Development which besieges operations. So there can never be a stable productive accord between these three groups… While they remain three divided groups.

The lesson for us here is to not create these three structures in the first place or to break down the silos that constrain them if we have them. By creating product-centric service teams comprised of engineers from all the essential disciplines the team will have all the skills they need to build, launch, manage and support their service. This new type of team is motivated by the performance of their service not merely it’s infrastructure, it’s code or it’s conformance to predefined criteria.

There’s also some great practical advice for troubleshooting scaling problems. Early in the book Paul quotes the first law of Mentat. In Dune Mentat’s are human computers capable of processing data at incredible rates. The first law of Mental states:

“A process cannot be understood by stopping it. Understanding must move with the flow of the process, must join it and flow with it.”

Consider the data most technical teams have for troubleshooting. System metrics, log entries, transaction checkpoints. All static data indicating a single point in time and in this data we’re supposed to understand how processes are behaving. Now consider what happens when we collect, aggregate and trend this data over time and review states prior to, during and after incidents. We begin to truly understand process behaviour and the behaviour of the various systems those processes operate within. Now to be fair the Application Performance Management tools took this as their goal from the outset but these tools have only been around for a few years now and many organisations are still not investing properly in their monitoring services.

I’d like to leave you with one final message from Dune:

Survival is the ability to swim in strange water.”

The water we swim in these days has become very strange. Applications have become services, services are comprised of micro-services, these services might run on platform services or infrastructure services that are themselves dependant on other services. We need DevOps if we’re going to survive in these strange waters.

Buy Dune on Amazon and while you’re at it you may as well buy my book too.

Featured image 77354890.jpg by Clarita on Morguefile.

DevOps Framework: The last teaser

2015-06-25 13.00.20

This week’s update is another short one we’ve just come back from an awesome holiday in the lake district. I’d had an intense few of weeks of writing and interviewing and then to top it all off I contracted some sort of flu or virus thing. Kylie’s contract had just come to an end and so we decided to take a short break before she started her next one. As you can see we had a great time and the weather was wonderful!

2015-06-24 15.01.59I love the lake district.There’s something about the fresh air and the countryside that really refreshes me. If anyone in Cumbria is considering a DevOps transformation please feel free to get in touch :).

I’ve returned ready to get the framework into a first draft state and published on Github so you can all get your teeth into it. On that note Kylie has finished her initial assessment and she’s provided some initial feedback that I’ll be working on this week.

Finally I’m participating in a panel discussing the business need for DevOps at Computing Summit’s DevOps 2015 event next week, July 8th. I hope I’ll see some of you there If you’re not interested in seeing me speak you should definitely hear what my friend Phil Hendren of Mind Candy fame has to say. While I’ve been leading DevOps initiatives and teams for the last 6 years Phil’s been engineering the solutions that make it happen you can find Phil’s blog here, it’s a great source of practical DevOps advice and tools.

A great example of DevOps in action!

I’m late bringing this to you as I’ve been ill for a few days so please accept my apologies. Last week I read a great piece that makes a really strong case for DevOps. If you think businesses can continue to thrive while they isolate their software, operations, network and test engineers in separate teams and assigned to projects by resource coordinators then you need to read this article.

I won’t write too much more here because I really just want you to read Ms. Macvittie’s article. I do want to close by saying that I’ve heard a lot about the dangers of optimising too early I’ve also seen many examples of what happens when you optimise too late. I think cross-functional product-centric teams can help maintain the balance between feature development and optimisation whether that’s service optimisation as discussed in Ms. Macvittie’s article or refactoring: https://devcentral.f5.com/articles/microservices-and-http2

* Featured image: cotswold-stone-wall.jpg by: thesuccess found on: Morguefile.com

DevOps Transformation Framework Update: Almost there…

I know that some of my readers are impatient to get their hands on the my new project: the Next Gen DevOps Transformation Framework. I’m as eager to release it as they are to play with it but it isn’t quite ready for it’s public debut yet. I have created a project in Github where I will release it but at the moment it just hosts a readme. If want to keep an eye the project you can find it here: https://github.com/grjsmith/NGDO-Transformation-Framework

With the framework I’m attempting something quite challenging. I don’t mean writing the framework, anyone who’s seen DevOps in action and read around the subject a little could create a framework for a DevOps transition. The challenge is presenting it for two audiences with two very different perspectives. I’m trying to create something detailed enough that engineers and managers can argue about it and contribute to it and yet something simple enough that budget holders can understand at-a-glance what they might be getting their organisations into.

Two months ago when I announced I had started work on the framework I mentioned that one of my goals was to create something that would have helped me while I was at EA. I failed to convince the central operations group to adopt DevOps, I was able to convince them of the benefits of DevOps but I couldn’t present them with an approach that they could plan and budget for. The framework successfully presents the scale and complexity of a DevOps transformation but I’m struggling with presenting the evolving benefits of the DevOps approach as the organisation progresses through the framework.

That’s where my secret weapon comes in. My girlfriend Kylie just happens to be a very talented Business Analyst. One of the things Kylie excels at is presenting complex data simply and effectively. Kylie’s working on the framework now and we hope to have the first version ready for release by the end of June.

DevOps: What comes first the culture or the collaboration?

Much has been written on the subject of corporate culture, more has been added since the advent of DevOps. Luminaries such as Patrick Debois, Gene Kim and Jez Humble all talk about DevOps and Culture on a regular basis. Every organisation I’ve worked in has been obsessed by it’s culture. After 20 years in small startups and global enterprises I’ve come to the conclusion that culture isn’t something that can be controlled, measured or managed directly.  This gives me a great excuse to paraphrase one of my heroes the late great Sir Terry Pratchett: “THEN TAKE THE UNIVERSE AND GRIND IT DOWN TO THE FINEST POWDER AND SIEVE IT THROUGH THE FINEST SIEVE AND THEN SHOW ME ONE ATOM OF CULTURE…”

Culture is created as a by-product when we work and play together, the nature of our culture is determined by our purpose and habits. A group that is encouraged to learn from failure and improve as a result will be curious and innovative. A group that only looks inwards for help and is discouraged from interacting with other groups will become defensive and narrow-minded. Groups have a tendency to reach out to other groups when they can speak the same language and have the same goals. Groups tend to become defensive when they perceive they have different or opposing goals to the groups around them, they take on a besieged mentality when they don’t even share terms of reference with the groups around them.

If you want to change a group’s culture you need to change how they work together, how they work with others and the habits they share. If you want a culture of innovation you should expose people to foreign ideas and concepts by creating the opportunity for groups with different skills and perspectives to work together on proposals that will become new projects. If you want a collaborative culture you should create the projects that require people with different skills and perspectives to collaborate.

As far as I can tell, there are currently two successful DevOps models. The directly collaborative cross-functional team model that encourages groups to work together to deliver services and the Service/API model that encourages groups to have rapid, predictable interactions by presenting APIs to each other. If we want collaboration we must break down the silos and allow engineers with different expertise and perspectives to work together. Consider a monitoring/feedback project where software, operations and test engineers work together to delivery better quality tests and monitoring. If we want groups to communicate with each other’s services via API calls we need to ensure all groups have the resources necessary to create good quality API’s. That includes Network, Operations and Test functions as well as focusing on the APIs between services and between clients and services.

The choices we make about how our teams work each day, the things we encourage them to do in the morning (whether we encourage them to check out the quality of the service or just look at their teams project progress) how we encourage them to behave when they make a mistake all contribute to the culture we create. We choose our culture when we decide how our teams work together and what we ask them to focus on and the opportunities we present them to learn, grow and receive recognition. In my experience DevOps happens when the opportunity exists to collaborate.

DevOps accelerates the rate at which the opportunities to collaborate present themselves. When I joined Playfish the Operations team existed as a service provider to the game and platform teams. Operations then built a few tools and shared the code with the games and platform teams and that created a few opportunities for operations and software engineers to work together. As those collaborations yielded more and more interesting and valuable results the level of collaboration increased as did the quality of the service that was provided. This then made more software engineers curious about operations and that yielded more opportunities for collaboration and learning until we had some teams that were almost entirely self-reliant for their regular day-to-day activities that made more space for greater levels of collaboration and so on and so on…

I write a lot more about how to build and foster collaboration in IT organisations in my book Next Gen DevOps: Creating The DevOps Organisation. Available on Amazon now.

*Image credit: Yoel on Morguefile

Next Gen DevOps Transformation Framework Update… Oh dear

Since I started working on the framework I’ve had the idea that I wanted there to be a visualisation of the entire framework.

I have a visual imagination. I’m most comfortable considering the big picture and I wanted to give people, like myself, an at-a-glance view of the framework. I also thought it might help people understand the scale of the framework and help place their organisation within it.

I’ve had four major attempts at this so far and I think I’m getting there. Unfortunately the Scrabble people might sue me if I go with this version :):

NGDOTF Capability Web v0.2

Don’t start projects in the bloody middle!

There must be millions if not billions of words published online about how to start a new project.

I know there are because I read a lot of them before I started writing Next Gen DevOps: Creating The DevOps Organisation.

Most of these articles can be boiled down to two main pieces of advice:

1. Step back from the project and look at the whole domain and then create a structured approach.
2. Dig in a little and get started until you get a sense of the complexity of your project and then step back again and take a second look.

So why does everyone starting a DevOps transformation immediately get cracking on Continuous Delivery?

It’s, arguably, the hardest technical activity in a DevOps transformation. Continuous Delivery in an enterprise with even a couple of years of legacy might involve the orchestration of hundreds of configuration items, the management of thousands of variables and an extremely complex validation process to ensure everything works as it should. There are organisations out there who still aren’t benefiting from reliable continuous build processes who are nevertheless downloading configuration management tools and starting to code the management of application properties. There are still businesses who consider testing to be an entry level role who are trying to get their entire estate to deploy continuously multiple times a day.

Don’t get me wrong I’m all for Configuration Management. If you can get that nailed you unlock a vast amount of potential, it’s just that it’s starting in the middle of the project!

I don’t know where your organisation is or where your strengths and weaknesses lie but before you start building building complex models for divining properties and environment variables you could consider configuration files as artefacts. Manage the creation, modification and deployment of those like you would your applications. You could also ensure you have your deployment tools in version control and treat them like you do your revenue generating services. Manage the development of them with stories and ensure their development keeps pace with the development of the services they need to deploy.

If you really want to push the bleeding edge you could include systems and testing experts in your development projects and ensure that your testing, monitoring and deployment capability is as sophisticated and as capable as your revenue generating services are

Good luck out there and feel free to drop me an email if you need a hand.

Oh and if anyone is wondering about the featured image, that’s my Dad teaching me to climb trees. If the reference is too subtle you start climbing trees at the bottom of the tree where it’s simplest but sometimes hardest to get a good grip :).