The Disciplined Agile Delivery (DAD) framework explicitly includes architecture-related activities, the role of Architecture Owner, and promotes the philosophy of enterprise awareness. Our experience is that agile enterprise architecture proves to be a key enabler for organizations in the process of adopting a Disciplined DevOps mindset.
In addition to general DevOps strategies , there are several enterprise architecture activities that support DevOps:
- Reuse mindset. An important thing that your enterprise architecture efforts will accomplish is the promotion of a reuse mindset within IT, and throughout your organization in general. Delivery teams with a reuse mindset strive to leverage existing data sources, services, components, frameworks, templates, and many other assets. This reuse mindset is enabled through education, coaching and mentoring by your enterprise architects (who are ideally active members of IT delivery teams in the role of Architecture Owner). It is also enabled by technical roadmaps that indicate the technologies that IT delivery teams should, and shouldn’t, be working with. And of course, having high-quality assets that are easy to discover, to understand, and to apply in the course of providing real value to your stakeholders enables reuse.
- Technical-debt mindset. Your enterprise architecture effort should promote strategies that motivate delivery teams to pay down technical debt when they find it and more important do what they can to avoid it in the first place. Many technical debt strategies are embedded right in the DAD framework, but without a technical-debt mindset this often comes to naught. Enterprise architects, often acting as Architecture Owners on delivery teams, should coach and mentor developers around the issues associated with technical debt. Similarly they should help to educate the senior managers and stakeholders whom they collaborate with in technical debt as well. It requires investment to avoid and remove technical debt, and IT investment decisions are typically in the hands of these people.
- Development guidelines. An important aspect of enterprise architecture is the development of guidelines for addressing common concerns across IT delivery teams. Your organization may develop security guidelines, connectivity guidelines, coding standards, and many others. By following common development guidelines your IT delivery teams produce more consistent solutions which in turn makes them easier to operate and support once in production, thereby supporting your DevOps strategy. A potential drawback of common development guidelines is that developers may feel constrained by them. To counteract this problem the guidelines should be developed and evolved in a collaborative manner with the delivery teams, not imposed from above.
- Technical roadmaps. Your enterprise architecture efforts include the definition, support, and evolution of technical roadmaps that guide the efforts of the rest of the organization (business roadmaps, also important, are the purview of Product Management). This in turn supports the creation of a common and consistent technical infrastructure within your production environments, enabling common DevOps practices such as continuous deployment, automated end-to-end regression testing, and operational monitoring that we discussed in previous blog postings.
An important aspect of your technical roadmap is to capture both the existing IT infrastructure and the future vision for that infrastructure. Your IT infrastructure potentially includes your network, software services, servers, middleware, and data sources to name a few elements. As you can see in the following diagram, when developing your technical infrastructure vision there are two issues to consider:
- Ownership. Does your organization own and operate its own infrastructure or does it outsource some or all of it to external experts? Outsourcing options include traditional strategies such as having another organization (such as HP or IBM) run your data centers to using cloud-technologies hosted by external organizations (such as Amazon or Google). The advantage of owning your own infrastructure is the greater level of control that it provides you, something that is critical when you must guarantee the security and integrity of your IT solutions. Outsourcing potentially offers greater flexibility in managing your IT infrastructure and cost savings from economies of scale. However, outsourcing requires more sophisticated governance and in the case of traditional strategies is a potential bottleneck when the outsourcer cannot respond in a timely manner to your requests.
- Virtualization. Are the elements of your IT infrastructure built to meet the needs of specific solutions or are they softwarized to provide malleability and ease of evolution? With softwarization, also known as software-defined infrastructure (SDI), the elements of your IT infrastructure are fully virtualized. Softwarization includes IT infrastructure models such as a software defined data center (SDDC), software defined storage (SDS), and software defined network (SDN). Softwarization is typically implemented using cloud-based technologies on either side of your firewall. Greater virtualization offers to increase flexibility and programmability of your IT infrastructure, and thereby enabling you to optimize resources. However, the greater flexibility of virtualization can increase the complexity of your testing efforts and make operational incident simulation more difficult.
This ends our series of blog postings about how the Disciplined Agile Delivery (DAD) framework supports DevOps. Well, at least for now. In the near future we will post an article that ties this series together. Please stay tuned.
In the Disciplined Agile Delivery (DAD) framework data management is a Run (operational) activity that focuses on the execution of data-oriented architectures, policies, and processes. Note that the long-term planning efforts around data-oriented aspects of your organization are part of your Enterprise Architecture efforts. Similarly, development of the data-oriented aspects of your organizational eco-system is addressed by solution delivery teams.
Because data management is an important aspect of your Run endeavors it will be affected by your Disciplined DevOps strategy. Our experience is that in addition to the general DevOps strategies describe previously, there are several data management strategies that support DevOps:
- Data and information guidelines. A straightforward way to promote greater consistency in the development and application of data and information sources is to have common guidance that teams will adopt and then follow. This guidance, including both standard policies and guidelines, will need to be defined, supported, and evolved over time in a collaborative and open manner.
- Quality data sources. Your production data sources, including files, databases, and data feeds, should be high quality assets that are easy to work with. When it comes to data sources of record it is particularly important for them to be of high quality so that they are easy to work with and evolve. Unfortunately this is often little more than fanciful thinking in many organizations. With a Disciplined DevOps mindset teams realize that they should be very careful about increasing the technical debt within their data sources, and more importantly invest in the effort to pay down any technical debt that they find.
- IT intelligence. IT intelligence is the creation, support, evolution, and operation of data warehouse (DW)/business intelligence (BI) solutions that support the management and governance of your IT efforts. From a Disciplined DevOps perspective this there are two important aspects of IT intelligence: development intelligence that provides insight into how delivery teams are working and operational intelligence that provides insight into what occurs in production. The automated team dashboards provided by many development platforms are a simple form of development intelligence, a more sophisticated (and useful) strategy is to combine information from various development tools to display it in an integrated dashboard for the team, and more sophisticated yet is to roll up/combine information from different delivery teams into a portfolio management dashboard. Similarly your operations team may have individual dashboards for each solution, they may combine information being generated by individual solutions into an integrated dashboard, and better yet share that information for an IT portfolio management view.
In the next blog posting in this series we will explore DevOps strategies pertaining to enterprise architecture.
In addition to the general release management strategies described previously, the general DevOps strategies, and the construction-focused DevOps strategies (including continuous deployment) there are several other release management strategies that support DevOps:
- Integrated deployment planning. From the point of view of development teams, deployment planning has always required interaction with an organization’s operations and release management staff; in some cases, via liaison specialists within operations often called release engineers. When you adopt a Disciplined DevOps mindset, you quickly realize the need to take a cross-team approach to deployment planning due to the need for operations staff to work with all of your development teams. This isn’t news to operations staff, but it can be a surprise to development teams accustomed to working in their own, siloed environments (luckily this strategy is built into DAD’s Move Closer to a Deployable Release process goal). If your team is not doing this already, you will need to start vying for release slots in the overall organizational deployment schedule.
- Standard development and testing environments based on production. Development teams know that the greater the consistency between their development, testing, and production environments the easier it is to test and deploy. In multi-team environments the implication is that this will result in defacto standardization of many aspects of your environments. Developers may choose different development tools, but aspects of the infrastructure such as operating systems, application servers, middleware, databases and so on will become consistent over time to streamline the overall release process.
- Release service streams. A key tenant of the DAD framework is that every team is unique, and an implication of that is that some teams will need more help than others. Teams will produce different levels of quality, they will have different amounts of automation, they will have different release cadences, and so on. As a result your release management strategy needs to be flexible enough to address these different situations. One way to do so is to offer different server streams, or service levels as it were, to solution delivery teams. For example, you may have a basic release management service stream where release management engineers actively help delivery teams to deploy their solutions into production and even help them to start automating some of their processes. At the other end of the spectrum you may have a continuous delivery service stream for delivery teams that have fully automated their testing and deployment processes and that can be trusted to successfully deploy on their own. And of course you could have several other service streams between those two extremes. The advantage of this approach is that it is very flexible albeit at the cost of slightly greater scheduling complexity.
- Release blackout periods. Some organizations have periods of time where they choose not to release new functionality into production unless it is absolutely critical. These blackout periods typically occur during high-volumes of business transactions. For example, many North American retail companies will have blackout periods between mid-November and early January for the holiday sales seasons. Many organizations will have blackout periods near the end of their fiscal years to enable them to focus on the process required to close out the year.
- Shared practices. Although this is really a process improvement issue, it’s worthwhile to point out that whoever is involved with release management should actively strive to share effective practices between teams. Sharing learnings across teams is an important aspect of enterprise awareness.
In the next blog posting in this series we will explore data management strategies that support a DevOps mindset.
In this blog posting we describe four general release management strategies that support DevOps. These strategies, from least effective to most effective, are:
- Release windows (slow cadence). A release window is a period of time during which one or more teams may release into production. A release slot is subset within that release window (and may be the entire window) during which a team may deploy their solution into production. For example, your organization may have a policy that production releases occur between 1am and 5am on Saturday evenings (the release window) and that up to four releases may occur during that window (the release slots). In lean terms, a release slot is effectively a Kanban card that allows a team to deploy. Release windows tend to align with periods where system usage is lower, although in the modern world of global 24/7 operations these periods have all but disappeared. With a slow cadence approach to this strategy the release windows occur far apart, as seldom as once a week or even longer. The advantages of this approach are that it provides a consistent release cadence to business stakeholders and it provides consistent release date targets for delivery teams. The primary disadvantage with slow cadence release windows is that they become bottlenecks for release management processes that need to support multiple teams. There are only so many release slots available during each window and this number can be easily exceeded, forcing teams to aim for a future release window. This problem becomes exacerbated when teams start to move to a continuous delivery strategy.
- Release train. The idea with a release train is that every team involved with that “train” has the same release cadence – for example this train releases once a quarter, or once a month, or even once a week. This strategy is commonly used in large programs, or teams of teams, where the individual teams are each working on part of a larger whole. Having the common drumbeat of a release train provides a consistent release schedule for stakeholders and serves as a rallying point for development teams. The train metaphor works quite well in practice. If your team misses the release date, if you miss the train, then the train goes on without you and you need to wait for space on the next on. Dependencies are also respected. For example, if several components need to ship together they must all go on the same train (similar to a family taking a trip together). The primary disadvantage is that development teams are constrained to a common release schedule, making it difficult to support lean or continuous delivery strategies. A potential disadvantage is that release trains may also suffer from the bottleneck problems of slow cadence release windows.
- Release windows (quick cadence). To support continuous deployment, particularly across many delivery teams, you will need a large number of release slots. The implication is that you will also likely need more release windows more often. The advantage of quick cadence release windows is that they are less likely to suffer from the bottleneck challenges associated with slow cadence release windows and release trains.
- Continuous release availability. With this approach delivery teams are allowed to release their solutions into production whenever they need to. In many ways this is simply an extension of the release window strategy to be 24/7. This is the only strategy that truly supports continuous delivery. To make it work a host of DevOps practices are required, such as fully automated deployment, fully automated regression testing, feature toggles, self-recovering components, and many others are required.
Our experience is that most enterprises today employ a slow cadence release window approach although are starting to evolve into the quick cadence version of this strategy. This is usually motivated by the adoption of agile techniques by solution delivery teams and more often than not by continuous delivery practices. We also see large programs take a release train approach, a strategy pioneered in the 1990s by large software companies such as Microsoft and Rational that sold software suites comprised of many products that needed to be shipped together. In recent years the OpenUP and SAFe frameworks have popularized the release train strategy. The strategy of continuous release availability is commonly used in advanced DevOps organizations such as Etsy and Amazon.
An important point to be made is that there are several options available to you, each of which has its advantages and disadvantages. No single approach is perfect, and no single approach works in all situations. You not only need to have choices, it’s good to have choices.
The next blog posting in this series continues with more release management DevOps strategies.
In this blog posting we describe two issues for organizing your release management strategy: How to scope release management and how to organize the team.
There are two fundamental issues to consider when scoping your release management efforts:
- Paradigm support. Will your release management process focus on supporting one paradigm, such as agile/lean teams or will it provide a more holistic strategy to also support agile/lean teams, traditional teams, iterative teams, and even ad-hoc teams? Many people who are currently writing about release management tend to focus on a single paradigm, although they may not explicitly state this, but the reality is that most enterprise-class organizations need multi-paradigm support in practice.
- Domain support. Will your release management process focus on IT-related issues or will it address the full range of business-related release issues? IT-related release issues include deploying new software and hardware into production. Business release issues may include marketing campaigns, training your sales force, and organizing external support mechanisms for end users to name a few. This is particularly important for commercial solutions being produced for the end customer of your organization.
These two issues lead us to the following quadrant chart depicting the potential scope for release management:
From a Disciplined DevOps point of view we of course promote a Holistic Enterprise scoping strategy. Whatever scoping strategy you choose your release management strategy will need to be able to support the scaling situations faced by your delivery teams. This includes teams of various sizes, different levels of geographic distribution, dealing with different levels of domain and technical complexity, teams that are organizationally distributed, and teams in compliance situations.
There are three strategies to consider for organizing your release management team:
- Operations led. In many small to medium-sized organizations release management is one of many activities that are performed by the operations team. With this approach a “release team”, in some cases an individual, is put together to release a solution on a project-by-project basis. Although there is often a hand-off point from the development team to the operations team, the operations team may require several members of the development team to be actively involved with the deployment. The advantages of having operations manage releases are that they are very familiar with the current state of your production environment and they know what other releases are happening in parallel (if any) and thereby have an integrated view of the overall situation. The primary disadvantage is that they will not know the intricacies of the new release of the solution, hence the need to include development team members.
- Separate release team. Larger organizations will often have an explicit release management team, often a subgroup of their operations department. The advantages of a separate team include the ability to grow expertise in release management, familiarity with your production environment, and the ability to manage releases in an integrated manner. The disadvantages are the lack of familiarity with solution(s) being released and the potential to inject overhead into the overall release process.
- Delivery team led. This approach is common in very small organizations that do not have separate operations teams and in organizations delivery teams have adopted at very disciplined approach to DevOps that supports the practice of continuous deployment. The advantages of a delivery team approach are that that team is very familiar with how the solution is built and they are given greater flexibility to deploy as needed into production. The disadvantages are that there is a risk of deployment collisions in multi-team environments and integration problems in production between disparate systems. Luckily these disadvantages can be addressed via a combination of development-oriented DevOps practices and the following release management practices.
There is no such thing as a “best practice”, except perhaps from a marketing point of view. All practices (and strategies) are contextual in nature. In some situations a practice works incredibly well and it other situations a practice can be the kiss-of-death. Two of the philosophies behind the Disciplined Agile Delivery (DAD) framework are that choice is good and that you should understand the trade-offs of the options available to you. In short, practices are contextual in nature and you should strive to adopt the ones that work best for the situation that you find yourself in.
Let’s work through an example. A hot button for many software developers are strategies around cost estimation, typically used for budgeting and forecasting purposes. In the DAD framework initial estimation is addressed by the Develop Initial Release Plan process goal, the goal diagram for which is shown below. As you can see with the Estimating Strategy factor there are several options available to you. We’re not saying that these are all of the potential estimating options available, but we are saying that this is a fairly good representative range. The arrow on the left-hand side of the strategies list indicates that the strategies towards the top of the list are generally more effective for initial planning than the strategies towards the bottom of the list. Your mileage may vary of course.
The following table summarizes the trade-offs involved with two of the estimating strategies, in the case formal point counting (such as function points) and an educated guess by an experienced individual. One of the reasons why the DAD book is so thick is that much of it is tables like this communicating the trade-offs of the hundreds of practices and strategies encapsulated by the framework. As you can see, there are advantages and disadvantages to both strategies. You can also see that there are situations where each strategy potentially makes sense. Although you may not like a given strategy, I personally abhor formal point counting, you should still respect the fact that the strategy is viable in certain situations (perhaps not yours).
|Formal point counting||
|Educated guess by experienced individual||
There are several reasons why DAD’s goal-driven approach is important:
- It makes your choices explicit. If your team wants to truly own your process then it first needs to know what choices are available to it.
- It makes it clear that practices are contextual in nature. No practice is perfect for all situations, every single one has advantages and disadvantages. Are you choosing the ones that are most appropriate for your situation?
- You can have more coherent discussions of the trade-offs that you’re making. We have endless religious battles in the IT industry around process-related topics. This often happens because people choose to focus on the benefits of their favorite practices and to downplay the disadvantages (or worse yet are oblivious to them). To help your team move to more effective practices it’s important to recognize the trade-offs involved with each, to then discuss them rationally, and then decide to adopt the strategy that is most viable given your situation. Note that this doesn’t necessarily mean that you’re going to adopt the best practice from the start, but that instead you’ll adopt the best one that you can right now. Later on, perhaps as the result of a retrospective, you’ll decide to make improvements to your approach (in the case of process factors where the strategies are ranked by effectiveness, your team may choose to adopt a strategy higher in the list).
- Improves the effectiveness of retrospectives. During a retrospective it is fairly straightforward to identify potential problems that you’re facing, what isn’t as easy is identifying potential solutions. You can improve retrospectives by having these process goal diagrams available to you to suggest potential strategies that you should considering adopting.
- You can avoid reinventing existing practices. Many very smart and very experienced people have found ways to deal with the same challenges that you’re facing today. Furthermore, many of them have shared these strategies publicly. If you don’t know that these strategies exist you are at risk of wasting time reinventing them, time that could be better spent adding real value to your organization.
- It enables scaling. Teams in different situation will make different process decisions. Although teams at scale, perhaps they are large teams or geographically distributed, will follow many of the same practices as small co-located teams they will also adopt a few strategies that make sense for them given their situation. DAD’s process goals provide the insight that teams need to understand how they can address the challenges associated with the scaling factors that they face.
For a more detailed discussion about the challenges around “best practices”, you may find the article Questioning Best Practices to be an interesting read. The New Deal for Software Development provides some interesting insights as well.
In this blog posting, part of our continuing series on DevOps, we explore solution support strategies. There are several solution support (help desk) strategies, which can be combined, that you may choose to adopt. These options are:
- Online information. A very common “self serve” support strategy is to develop and maintain online assets such as frequently asked questions (FAQ) pages, training videos, and user manuals to name a few. This enables end users to potentially support themselves, although suffers from the TAGRI (They Ain’t Gonna Read It) syndrome.
- Online discussion forums. Many organizations choose to implement internal discussion forums so that their end users can help each other in learning how to use their systems. This is effectively a collaborative self-serve support option for end users. The primary advantage is that your “power users”, or in some cases members of the development team, will come to the aid of other users who are struggling with an issue. A potential disadvantage is that you risk your discussion forum becoming a complaints forum if problems aren’t addressed in a timely manner.
- Asynchronous support. With asynchronous support strategies an end user will put in a request for help and then sometime later somebody gets back to them with help (hopefully). Common ways to implement asynchronous support include implementing a standard support email or a support request page/screen. It is common in many organizations to put a service level agreement (SLA) in place putting limits on how long people will need to wait for help.
- Synchronous support. With synchronous support strategies end users are put in contact with support people (who may even be one of the application developers) in real-time. This is often done via online chat software, video conferencing, or telephone calls. The key advantage of synchronous support is responsiveness. However, synchronous support can be expensive to operate and potentially frustrating for end users, particularly when the support desk function is outsourced to people following scripts.
- Support alerts. With this strategy your solution itself detects serious problems affecting end users, such as a data source or a service/component being unavailable. When such an event occurs, and the solution isn’t able to swiftly recover, the end user is informed of the problem and presented with a “Would you like help?” option. If yes, they are put in direct contact with an appropriate support person who then helps them in real-time. This is part of your solution’s self-recovery process.
- Developer-led support. This strategy has development teams performing the support services for their own solutions and was described previously in DevOps Strategies: Development.
In the next installment in our DevOps series we will describe release management strategies.