/ DevOps

Retrospective: State of DevOps in 2015

DevOps is one of the hottest trends in IT, but it is also one of the most ill-defined trends in the market. While vendors, analysts, consultants and IT practitioners can recognize DevOps when they see it, it’s frequently defined by a collection of tools and practices that are still evolving.

If you ask a DevOps Engineer for the definition of DevOps, you often hear a variety of answers centered around tools and techniques. For some people embracing DevOps it's about managing IT resources with Chef, Puppet, or SaltStack, and for others it is about using tools like Jenkins or Travis CI to automate deployments to cloud based infrastructure.

For most organizations, DevOps was simply about making sure that developers and operations professionals were communicating efficiently. Some companies have dedicated staff focused on DevOps initiatives, while others think of DevOps as a philosophy rather than a job description.

Clearly DevOps was a popular term in 2015, and it's time to put a stronger definition on this movement, as well as some standardized metrics for measuring success.

I came across a report from Gleanster Research, in collaboration with Delphix, and they surveyed 2,381 IT practitioners from across the globe to understand the role of DevOps, its evolving best practices, and the current and planned investments in DevOps initiatives.

This comprehensive survey engaged thousands of IT professionals from North America, France, Germany, the UK and the Netherlands within companies with over 100 employees and active involvement in DevOps, software development, database administration, quality assurance, or product support.

"The goal is not to provide another “me-too” definition of DevOps that is skewed to a technology solution or service offering. We seek to offer the most comprehensive deep dive on the state of DevOps and provide a standardized data-driven definition based on current best practices – a definition that reflects how the most successful organizations that have invested in DevOps approach the concept from the standpoints of people, process, and technology."

Nearly every company surveyed reported ongoing DevOps initiatives or plans to start a DevOps initiative in the next 24 months.

What are the motivations behind the DevOps movement?

  • What drives organizations to start DevOps initiatives?

  • What practices comprise DevOps?

  • How do organizations measure the effectiveness of DevOps in the enterprise?

Existing research has focused more on DevOps as a collection of tools, techniques, and technology. This research focuses on motivations and more practical aspects of how DevOps relates to the overall business.

DevOps is changing at an impressive speed, as the movement is now being influenced by a sea of vendors and conferences, all designed to champion DevOps among companies looking to adopt established best practices.

As with cloud computing and other hot trends in IT, there’s a sense that DevOps means both everything and nothing.

In the midst of all of this activity the IT industry often loses sight of what DevOps offers to organizations struggling with application development, production support, and infrastructure management issues.

  • What is DevOps?

  • What are the benefits it provides?

Answering these simple questions could help to clarify and define one of the fastest growing components of the IT budget. While survey respondents report that they are investing hundreds of millions of dollars in DevOps, how they translate DevOps into investments remained very unique to each organization.

Their findings showcase how the most successful “DevOps Leaders” define the concept. DevOps is about much more than collaboration and agility. “DevOps” defines the transformation IT experiences when cross-functional teams develop and deliver software across the full spectrum of IT systems.

From software architecture and design, to system administration and production support, the term “DevOps” refers to a style of IT management and implementation that places an emphasis on automation and iterative delivery of software while also empowering developers to manage portions of the software delivery process that were previously inaccessible due to specialization within IT.

This is from their 2015 State of DevOps Report:

We’re seeing the “gold rush” phase for DevOps in 2015, and the DevOps hype machine is in overdrive. While this is certainly true of deployment automation, continuous integration, and data management products, the same cannot be said of issue trackers, ITIL-based service management tools, and other systems that predated the emergence of DevOps as a movement in 2009.

DevOps Has Been Generally Ill-Defined.

DevOps and the tools and practices associated with it have transformed IT in just a few years. Continuous deployment, deployment automation, and the self-service provisioning and configuration of hardware, data sources, and networks to support ongoing application development are just a few of the practices that have started to take hold across the entire industry.

What started as a trend for internet-focused companies such as Etsy and Flickr is now approaching the enterprise; daily production releases, agile data management, and testing in production are now commonplace in many organizations, with cloud-based deployments supporting even the most critical of systems.

From the financial industry to healthcare to government, DevOps is being applied to industries once thought too risky for the rapid pace of infrastructure and development work that has come to typify DevOps. Over the past two years DevOps for the enterprise has emerged to support related initiatives such as scaled agile and other processes that place an emphasis on faster time to market and more rapid software releases.


While DevOps grows more popular and has established a foothold in the enterprise, the movement still lacks a strong definition across industries and organizations. Two separate organizations engaged in DevOps may both agree on a common set of tools and techniques, but they often don’t follow the same practices. Some organizations view DevOps as an integral part of the day-to-day activities of the entire IT department. However, organizations with a stronger definition of DevOps tend to empower dedicated teams who are exclusively responsible for rolling out DevOps initiatives, and these are the organizations that see the greatest DevOps success.

What is DevOps Doing?

>“What is DevOps?” is a tough question to answer if DevOps practices vary widely across industries and organizations. We asked DevOps Practitioners and DevOps Leaders to tell us which DevOps initiatives they were working on, and here are the top four.


Key Takeaway:
DevOps practices support rapid iterations, frequent production deployments, and the ability to quickly deliver data required for testing and verification in support of more frequent releases.

Who Drives DevOps?

>Given the concept of “DevOps” and a common perception that it helps break down the “wall of confusion” between development and operations, we hypothesized that established DevOps initiatives would emphasize communication between development and operations – on an equal footing. Surprisingly, the survey results suggest that is not the case.

Wait? Isn’t DevOps about cooperation and communication? It is, and it isn’t. This data very clearly illustrates that DevOps isn’t about two sides of IT collaborating together. DevOps was created as a reaction to slow-moving IT functions that couldn’t keep up with developer-driven innovation and the pace of software delivery in an agile development process.

Developers drive value creation in IT. They are looking for immediate gratification when it comes to data environments, deployments, and self-service infrastructure. DevOps isn’t just about collaboration between Development and Operations for DevOps Leaders; it’s about developers embracing operations functions and deriving scale through automation. DevOps helps developers clearly communicate what they need for faster software delivery.

The Definition is a Moving Target

>How do organizations define DevOps? We gave respondents multiple choice answers to determine how they would generally define DevOps. The hope was one or two areas would clearly emerge among the 2,000 plus survey respondents. But definitions of DevOps vary dramatically, and if anything, it’s become a catch-all that embodies everything from collaboration, to technology, to operations, to agility.

Ultimately DevOps must be defined by measurable outcomes. Some of the definitions collected can drive quantitative success metrics, while others were less suited to it. DevOps Leaders were 12 times more likely than Practitioners to select “uncovering defects in the development lifecycle” as a central part of defining DevOps.

What is interesting about this is that quality issues are a tangible metric for tracking success, whereas collaboration is far more nebulous. As we explore the definition of DevOps we’re looking for ways to link the benefits with measurable success. Themes like automation and collaboration are critical but often difficult to measure.

Under Pressure to Deliver?

Organizations Turn to DevOps

How organizations prioritize value and measure success is key to understanding investments in DevOps. Eighty percent of all respondents indicated they were under strong pressure to reduce the number of defects in development and the number of defects in software delivered to production as early as possible in the development cycle. In fact, these drivers were among the top three for all organizations. Time to market and speed of delivery actually emerged as a secondary concern for all respondents.

For most practitioners the widespread definition of DevOps focuses on agility – the speed and rate at which software can be developed and deployed to production. But the data reveals that, as a practice, organizations turn to DevOps to identify defects before they get to production.


_DevOps isn’t just about delivering software faster and more frequently. Teams engaging in DevOps initiatives mark quality and defect identification as a stronger motivation to invest._

Interestingly, headcount reduction is not a significant pressure for DevOps Leaders. Leaders tend to support dedicated DevOps teams in IT. This highlights another important theme in the data: DevOps is not just about cutting costs. That means your primary justification around investments in DevOps shouldn’t be to work with fewer resources.

DevOps Leaders are focused on producing optimal quality and agility with the right level of resources.

Leaders aren’t pressured to sacrifice quality to save on headcount costs. This also underscores the fact that people are an integral part of DevOps success – technology should be empowering human capital, not replacing it.

The Measurable Benefits of DevOps

Given a clear focus on quality and agility, we wanted to understand how survey respondents were measuring the success of DevOps. Given the dynamic nature of the concept and a wide variety of metrics that could be used to measure success, we split KPI’s between release-related metrics and quality-related metrics, which we thought would be easier for respondents to link to DevOps than metrics related to “overall IT budget,” “agility,” or “cost savings.”

It’s abundantly clear that the economic impact of time is a theme that ripples through all facets of DevOps. This comes as no surprise given the fact that survey respondents indicated that on average 40% of their day is spent re-coding due to bugs. Respondents also indicated it takes an average of 2 hours to reset an environment after a test cycle; reducing this time could have a real and substantial impact on efficiency and effectiveness.

An increased cadence of software releases affects when teams identify and address defects. Forty-eight percent of DevOps Leaders reported that DevOps initiatives resulted in more defects being identified earlier in the software development lifecycle. When defects are identified earlier in the development lifecycle teams spend less time re-coding and more time delivering value.

Full Stack DevOps Initiatives

DevOps emerged from a combination of industry trends. While agile software development has grown into an industry standard over the last decade, cloud computing and infrastructure automation only became practical within the last six to seven years. These two factors, combined with increasing pressure to deliver software faster and more frequently, set the stage for the emergence of DevOps.

When application development teams assert more ownership over infrastructure automation and continuous integration, the IT function must support a full-stack approach to architecture.

In a DevOps-aware organization developers are now responsible for the full lifecycle of development, deployment, and production support.

**These are full-stack initiatives that define DevOps in 2015.**

With this new, “full-stack” approach to software delivery comes new opportunity for delay. DevOps was developed as a reaction to the long lead times required for infrastructure provisioning and integration with bureaucracy-laden ITIL processes.

What are these new sources of delay in the software development lifecycle?


  • Bugs identified late in the development lifecycle
  • Teams delayed while waiting for data environments
  • Teams delayed while waiting for data masking
  • Teams delayed by process for production control

DevOps Leaders are more affected by delays from data environments and data masking.

Data-driven delays are providing greater motivation for DevOps teams to take more proactive steps toward developer-driven agility.

Critical Need: Standard Success Metrics

DevOps still needs universally agreed upon standard KPIs to measure success across industries and organizations. Current approaches to DevOps adoption can be characterized as ad-hoc, either placing an emphasis on following established leaders or on following the advice of an abundance of competing “DevOps vendors.” Unlike all other parts of the IT department, fewer than half of DevOps staff stated that there was a strong definition of DevOps internally.

DevOps in 2015 needed both a strong definition and a set of standard success measures, but how does one measure “improved cooperation between development and operations” or “alignment”?

Based on the survey, two focus areas of Key Performance Indicators emerged.

Software Quality and Release Agility

Development **SOFTWARE QUALITY**

DevOps has a direct effect on software quality, as it allows development teams to integrate and test systems earlier in the software development lifecycle. Prior to the arrival of DevOps, development teams viewed deployment and integration as a separate phase of the software delivery cycle driven by operations. With DevOps Integration and deployment efforts starting earlier, they are driven by developers, and defects related to data environments and deployments can be addressed earlier to avoid costly recoding and delays.

  • Reduced downtime for non-production testing and integration environments
  • Increased availability and timeliness of data environments for development and QA
  • Greater percentage of defects identified earlier in the software development lifecycle
  • Faster identification and turnaround times for defects in production environments

Operations **RELEASE AGILITY**

With an emphasis on infrastructure automation, secure data, and shared responsibility for operations, DevOps addresses inefficiencies in the provisioning and management of IT infrastructure to support both production and nonproduction environments. Automation and lightweight approaches to software releases with less emphasis on ITIL-based process provide IT departments with a roadmap to scale to meet demand as the average size of IT portfolios continues to increase.

  • Reduction in effort required to deliver software to all environments
  • Increase in the frequency of releases and software development lifecycles with a shorter duration
  • More automated approaches to data management and data-security-as-a-service
  • Fewer resources required to support an increased number of applications with more frequent

A Standard Definition of DevOps

The data from the survey revealed an overwhelming need to establish not just a universal definition of DevOps, but a way to measure the success of DevOps initiatives.

DevOps is more than just the close collaboration of two departments (development and operations) within IT, it is more than just managing infrastructure with Chef or Puppet, and DevOps is much more than a specific collection of tools and techniques used to automate deployments and manage infrastructure.

The term “DevOps” refers to the transformation IT experiences when cross-functional teams develop and deliver software across the full spectrum of IT systems. From software architecture and design to system administration and production support, the term “DevOps” refers to a style of IT management and implementation that places an emphasis on automation and iterative delivery of software, while also empowering developers to manage portions of the software delivery process that were previously inaccessible due to specialization within IT.

DevOps tools and practices have one thing in common: they focus on reducing time to market and making it possible to extend the frequent iterations of Agile into infrastructure and data environments. Overall DevOps is inseparable from both agile software development and cloud computing.


  • How quickly an organization can leverage infrastructure and assemble data to support software delivery.
  • DevOps success is measured by the speed, frequency, and quality of software releases.

A Standard Definition of DevOps

"DevOps is a combination of development-focused innovation alongside innovation in infrastructure management and operations. Software releases and the various quality assurance and testing environments supporting a production release play an important part in the success of DevOps initiatives, such as continuous deployment and deployment automation."

Key Findings

  • Large organizations have a propensity for stronger definitions of DevOps, suggesting the emergence of an enterprise-ready DevOps movement.

  • The fact that a higher number of overall employees would correspond to a stronger definition of DevOps is a surprising outcome in this survey.

  • Among DevOps Leaders private clouds such as OpenStack have a much higher adoption rate than public clouds. For larger enterprises, private clouds are making greater inroads than public cloud offerings.

  • More than half of the respondents to this DevOps survey wait a week or longer to refresh non-production environments from the production environment.

  • This is why turnkey solutions such as Secure Data-as-a-Service are rapidly growing in adoption and delivering increased efficiency in enterprise DevOps initiatives.

  • There is a shocking lack of security and access control for critical production data.

  • 72% of DevOps Leaders reported they had no access controls or audits placed on production data.

  • This suggests that more active and successful DevOps initiatives are correlated with a lack of commitment to Secure Data initiatives.

What’s next for DevOps?
  • 39% report that achieving a daily production release cycle is a top priority.

  • About 25% of enterprise respondents have no plans to use public clouds.

Among those with a strong definition of DevOps and a track record of DevOps success:

  • 50% say DevOps is led by Developers
  • 35% indicating it’s a shared responsibility with Operations.

Top Three Reasons for DevOps:

  • Deliver Software Faster (66%)
  • Identify Bugs Earlier (44%)
  • Deliver Software Frequently (43%)

Biggest Challenge to DevOps:

  • 46% of DevOps Leaders say that testing environments are limited due to data issues.


DevOps is changing at an impressive speed, and the movement is now being influenced by a sea of vendors and conferences all designed to champion DevOps. For many companies embracing DevOps practices, 2015 was a year that extended DevOps to areas of the software delivery lifecycle previously untouched by the DevOps movement.

Automating the deployment of often stateless web applications on either containers or virtual machines is now something that organizations embracing DevOps take for granted, yet there is still a large component of the market that has yet to fully embrace automation and cloud based infrastructure.

Companies new to DevOps are facing interesting questions only made possible by the popularity of the DevOps movement. Questions such as “Which cloud provider to use?” or “Which private cloud technology to adopt?” are answered alongside questions about development platforms, operating systems, and (with containers) the definition of a deployable application.

Containerization plays an important role in DevOps as it moves responsibility for deployment
and configuration management toward application developers. Using Docker or Quay coupled with continuous integration and delivery servers has provided a level of automation that can take all of the frustrations out of frequent code deployments.

Enter 2016, and I've just completed a project that combines continuous integration and delivery with a platform that builds applications on a fresh Docker container, caches dependencies (just like Heroku) and outputs the source code to app-name.stackriot.com.

I've spent a lot of time working with virtualization, Docker and automation recently. One of the biggest challenges has been to get a good testing environment internally, without having to pay exorbitant amounts of money to use services like Heroku. I knew there had to be a better, more affordable alternative, and I was determined to find it.

I've used just about all of the CI services available, including Jenkins, Circle CI, Codeship and Travis. I learned that each of these services has their own quirks. For example: hard to install dependencies, Selenium tests, required infrastructure services, build limits, etc. This is why I have grown to love Drone. I can run my test suite in a clean docker image every time, cache dependencies (just like Heroku does), and run a deploy action if the test build succeeds.

I am using Drone for the continuous integration, and Dokku for the PaaS. Drone is built using Go and utilizes Docker. It can be run inside a container itself with very little configuration.

Drone is a Continuous Integration platform built on container technology. Every build is executed inside an ephemeral Docker container, giving me complete control over my build environment with guaranteed isolation.

Drone's integration with Docker means it can support a huge number of languages including PHP, Node, Ruby, Go, and Python, to name a few. Each test will spawn a new container based off of specified images from the Docker Public Registry. You can even make your own to fit your specific application if needed.

Drone also supports various notification mechanisms, including Email, IMs, Gitter and Slack. It also has support for a host of linked services including PostgreSQL, MySQL, MongoDB and Redis. Drone is written in Go, so writing your own service is super easy if necessary.

To make this project even sweeter, I've coupled Drone with Dokku, a simple Heroku like PaaS built on top of Docker. Using the Github flow with this setup allows automatic staging of all feature branches that pass their tests.

With one simple command:
git push dokku master, as long as my build passes in Drone, the code is deployed in a fresh Docker container. Dokku builds the app on a sub-domain of the Dokku host, and it's automatically available at app-name.stackriot.com.

Dokku is a mini-Heroku powered by Docker written in less than 100 lines of Bash. Once it's set up on a host, you can push Heroku-compatible applications to it via Git. They'll build using Heroku buildpacks and then run in isolated containers. The end result is a privately hosted continuous integration platform. It's got all of Heroku's awesome features, and I don't have to pay a premium to use their service.

Seth Bergman

Seth Bergman

I'm a Software Engineer focused on helping companies scale. I love learning about software architecture, containers, open source programming and automation.

Read More