Disciplined Agilists are Enterprise Aware

Enterprise awareness is one of the key aspects of the Disciplined Agile Delivery (DAD) framework.  The observation is that DAD teams work within your organization’s enterprise ecosystem, as do all other teams.  There are often existing systems currently in production and minimally your solution shouldn’t impact them.  Better yet your solution will hopefully leverage existing functionality and data available in production.   You will often have other teams working in parallel to your team, and you may wish to take advantage of a portion of what they’re doing and vice versa.  Your organization may be working towards business or technical visions which your team should contribute to.  A governance strategy exists which hopefully enhances what your team is doing.

What it Means to be Enterprise Aware

Enterprise awareness is an important aspect of self discipline because as a professional you should strive to do what’s right for your organization and not just what’s interesting for you. Teams developing in isolation may choose to build something from scratch, or use different development tools, or create different data sources, when perfectly good ones that have been successfully installed, tested, configured, and fine-tuned already exist within the organization.  Disciplined agile professionals will:

• Work closely with enterprise professionals.  This includes working closely with enterprise technical architects and reuse engineers to leverage and enhance the existing and “to be” technical infrastructure; enterprise business architects and portfolio managers to fit into the overall business ecosystem; senior managers who should be governing the various teams appropriately; operations staff to support your organization’s overall development and operations (DevOps) efforts; data administrators to access and improve existing data sources; IT development support people to understand and follow enterprise IT guidance; and business experts who share their market insights, sales forecasts, service forecasts, and other important concerns.  In other words, DAD teams should adopt what Mark refers to as a “whole enterprise” mindset.
• Adopt and follow enterprise guidance.  Your organization may have, or hopes to one day have, a range of standards and guidelines (guidance) that it wants delivery teams to adopt and follow.  This may include guidance for coding, user interface development, security, and data conventions to name a few.  Following common guidance increases the consistency and maintainability of your solutions, and thus your overall quality.
• Leverage enterprise assets. There may be many enterprise assets, or at least there should be, which you can use and evolve.  DAD teams strive to work to a common infrastructure; for example, they use the enterprise-approved technologies and data sources whenever possible, and better yet they work to the “to be” vision for your infrastructure.  If your organization uses a disciplined architecture-centric approach to building enterprise software, there will be a growing library of service-based components to reuse and improve upon for the benefit of all current and future solutions.  To do this DAD teams will collaborate with enterprise professionals throughout the lifecycle and particularly during Inception during envisioning efforts.   Figure 1 summarizes the Inception phase goal Align with Enterprise Direction which summarizes the strategies you may choose to follow.  Read Disciplined Agilists Take a Goal-Driven Approach for more information on DAD’s goal-driven strategy.

Figure 1. Inception Goal: Align with Enterprise Direction.

• Enhance your organizational ecosystem. The solution being delivered by a DAD team should minimally fit into the existing organizational ecosystem – the business processes and systems supporting them – it should better yet enhance that ecosystem.  To do this, the first step is to leverage existing enterprise assets wherever possible as described above, often working with enterprise architects to do so. In addition to the enterprise architects DAD teams will also work with operations and support staff closely throughout the lifecycle to ensure that they understand the current state and direction of the organizational ecosystem.  DAD teams will often be supported by an additional independent test team that will perform production integration testing (amongst other things) to ensure that your solution works within the target production environment which it will face at deployment time.  Furthermore, experienced DAD teams will even fix problems that they run into via proven refactoring techniques.  Figure 2 summarizes the general goal Leverage and Enhance Existing Infrastructure which summarizes strategies for how DAD teams may accomplish this.

Figure 2. General Goal: Leverage and Enhance Existing Infrastructure.

• Adopt a DevOps Culture. DAD teams will work with operations and support staff closely throughout the lifecycle, particularly the closer you get to releasing into production.  DevOps culture and strategies are baked right into DAD, a topic for a future blog posting.
• Share learnings.  DAD teams are learning oriented, and one way to learn is to hear about the experiences of others.  The implication is that DAD teams must also be prepared to share their own learnings with other teams.  To do this organizations might choose to support agile discussion forums, informal presentations, training sessions delivered by senior team members, and internal conferences to name a few strategies.
• Adopt appropriate governance strategies.  Effective governance strategies should enhance that which is being governed. An appropriate approach to governing agile delivery projects, and we suspect other types of efforts, is based on motivating and then enabling people to do what is right for your organization. What is right will of course vary, but this typically includes motivating teams to take advantage of, and to evolve, existing corporate assets following common guidelines to increase consistency, and working towards a shared vision for your organization. Appropriate governance is based on trust and collaboration. Appropriate governance strategies should enhance the ability of DAD teams to deliver business value to their stakeholders in a cost effective and timely manner.  Unfortunately many existing IT governance strategies are based on a command-and-control, bureaucratic approach which often proves ineffective in practice. The DAD book explores appropriate governance, the impact of traditional governance strategies, and how to adopt an appropriate governance strategy in detail.  The article Adopting Agile Governance Requires Discipline also provides greater insight.
• Open and honest monitoring. Although agile approaches are based on trust, smart governance strategies are based on a “trust but verify and then guide” mindset.  An important aspect of appropriate governance is the monitoring of project teams through various means.  One strategy is for anyone interested in the current status of a DAD project team to attend their daily coordination meeting and listen in, a strategy promoted by the Scrum community.  Although it’s a great strategy we highly recommend, it unfortunately doesn’t scale very well because the senior managers responsible for governance are often busy people with many efforts to govern, not just your team.  Hence the need for more sophisticated strategies such as an “development intelligence” approach supported via automated dashboards.  More on this in a future blog posting too.

Why is Enterprise Awareness Important?

Enterprise awareness is important for several reasons.  First, you can reduce overall delivery time and cost by leveraging existing assets.  In other words, DAD teams can spent less time reinventing the wheel and more time producing real value for their stakeholders.  Second, by working closely with enterprise professionals DAD teams can get going in the right direction easily.  Third, it increases the chance that your delivery team will help to optimize the organizational whole, and not just the ”solution part” that it is tasked to work on.  As the lean software development movement aptly shows this increases team effectiveness by reducing time to market.

Challenges to Enterprise Awareness

Unfortunately there are two main challenges to supporting enterprise awareness on agile teams.  First is the cultural challenge within the agile community that some ”agile purists” perceive this as unecessary overhead.  Reasons for this misunderstanding include a lack of understanding of the overall enterprise picture or some agilists who have previous experiences with enterprise professionals who struggle to work in an agile manner.  This points to the second challenge that enterprise professionals often don’t understand how to work effectively with agile teams.  Sometimes this is because the agile teams they’ve been working with until now haven’t been sufficiently disciplined to work with them effectively, but more often than not it’s because they still choose to follow older, more traditional approaches to their craft (they may find my articles about Agile Enterprise Architecture, Agile Enterprise Administration, and even The Enterprise Unified Process to be illuminating).

These challenges are cultural in nature, and thus difficult to overcome.  Agilists and enterprise professionals need to respect one another and strive to learn more about what the other group is trying to accomplish.  They must strive to work with one another and thus learn from each other.  Not only is this possible it is highly desirable.

In summary, DAD teams and more importantly DAD practitioners are enterprise aware.  They recognize that enterprise aware strategies improve their ability to provide value to their stakeholders both within the scope of a solution as well as at the organizational level.  To coin an environmental cliché: Disciplined agilists act locally and think globally.

Material for this blog posting was modified from Discplined Agile Delivery: A Practitioner’s Guide to Agile Software Delivery in the Enterprise by Scott W. Ambler and Mark Lines (IBM Press, 2012)

Disciplined Agilists Take a Goal-Driven Approach

In this posting I explore the goal-driven aspect of the Disciplined Agile Delivery (DAD) process decision framework.   This goal-driven approach enables DAD to avoid being prescriptive and thereby be more flexible and easier to scale than other agile methods.  For example, where Scrum prescribes a value-driven Product Backlog approach to managing requirements DAD instead says that during construction you have the goal of addressing changing stakeholder needs.  DAD also indicates that there are several issues surrounding that goal that you need to consider, and there are several techniques/practices that you should consider adopting to do so.  DAD goes further and describes the advantages and disadvantages of each technique and in what situations it is best suited for.  Yes, Scrum’s Product Backlog approach is one way to address changing stakeholder needs but it isn’t the only option nor is it the best option in most situations.

I start by describing how to visualize goals.  I then summarize the goals called out by the DAD framework, a topic we’ve written about in the past so I only cover this topic briefly here.  I end with a summary of the advantages and disadvantages of a goal-driven approach over the more prescriptive approaches of older agile methods.

Visualizing Goals

In the DAD book we described goals in a non-visual manner using tables which explored the advantages and disadvantages of the techniques associated with an issue.  Since we wrote the book both Mark and I have spent a lot of time helping people to understand what a goals-driven approach entails and we’ve found that many people respond well to visual representations of a goal.  Yes, the process decision tables are very important but a visual overview helps to provide context for the detailed information.

In the second half of 2012 we began developing a way to represent goals in a visual manner using what we call a goals diagram.  A goals diagram, the notation for which is summarized in Figure 1, is in effect a form of decision tree.  In Figure 1 you see that a goal is indicated using a rounded rectangle and the issues pertaining to a goal with normal rectangles.  Goals will have one or more issues that you need to consider addressing, with most goals having four or five issues although some have eight or nine.  Each issue is then addressed by two or more techniques/practices.   Because there may be many techniques to choose from, we indicate “default” techniques in bolded italics.  These defaults are good starting points for teams new to agile.  Some issues you may choose not to address, so an option of “None” will be indicated in these cases.  Sometimes options are “ordered”, which is indicated by a upwards pointing arrow to the left of the list of techniques.  What we mean by this is that the techniques appearing at the top of the list are more desirable from the point of view of agile and lean thinking and the less desirable techniques are at the bottom of the stack.  Your team of course should strive to adopt the most effective techniques they are capable of performing given the context of the situation that they face.  In Figure 1 the first issue has an ordered set of options whereas the second issue does not.  Typically when the options are ordered you will only choose one of them whereas you MIGHT choose several options in unordered situations.

Figure 1. The notation of goal diagrams.

Let’s work through some examples.  Figure 2 depicts the goal diagram for Explore Initial Scope, a goal that you should address at the beginning of a project during the Inception phase (remember, DAD promotes a full delivery lifecycle, not just a construction lifecycle).  Where some agile methods will simply advise you to populate your product backlog with some initial user stories the goal diagram of Figure 2 makes it clear that you might want to be a bit more sophisticated in your approach.  What level of detail should you capture, if any (a light specification approach of writing up some index cards and a few whiteboard sketches is just one option you should consider)?  What view types should you consider (user stories are one approach to usage modeling, but shouldn’t you consider other views to explore the data or the UI)?  Notice how we suggest that you likely want to default to capturing usage in some way, basic domain concepts (e.g. via a high-level conceptual diagram) in some way, and non-functional requirements in some way.  There are different strategies you may want to consider for going about modeling.  You should also start thinking about your approach to managing your work.  In DAD we make it clear that agile teams do more than just implement new requirements, hence our recommendation to default to a work item stack over Scrum’s simplistic Product Backlog strategy.  Finally Figure 2 makes it clear that when you’re exploring the initial scope of your effort that you should capture non-functional requirements - such as reliability, availability, and security requirements (among many) - in some manner.

Figure 2. Exploring the initial scope.

Figure 3 depicts one of the goals that you should address during the construction phase, in this case Address Changing Stakeholder Needs.  This is an iteresting example for two reasons.  First, it captures the key decisions surrounding the second of the 12 principles  of the Agile Manifesto, DAD actually suggests that we extend the Agile Manifesto to reflect our learning over the past decade+, that of welcoming changing requirements.  Second, it is the only goal with an issue that overlaps with that of another goal, in this case we indicate that your Work Item Management Strategy is an important issue to consider for both this goal and Explore Initial Scope (see Figure 2).

Figure 3 makes the issues surrounding how to address changing stakeholder needs very explicit.  How are you going to prioritize changes?  A business value approach is one option, the approach popularized by Scrum, but I’ve found that the risk-value approach promoted by Unified Process (UP) to be a more robust strategy that leads to greater chance of agile project success.  There’s advantages and disadvantages to each technique so you’ll want to choose the one best for you.  When are you going to accept the change?  During the current iteration as Extreme Programming (XP) suggests or a future iteration as Scrum suggests?  Do changes come directly from stakeholders or via a proxy such as a product owner or business analyst?  How will your team elicit changes (via modeling, demos, …)?

Figure 3. Addressing changing stakeholder needs.

The advantage of visualizing goals as I’ve shown in Figures 2 and 3 is that it makes it very clear what process-related decisions you need to make and what options you have available to you. The disadvantage of this sort of diagram is that they get fairly big at times, as you can see.  This effectively prevents us from taking the diagrams one step further to indicate the trade-offs associated with each technique and as a result you’ll still need the text tables we included in the DAD book for that.

The Goals of DAD

In the previous section I indicated that there are many goals called out by the DAD framework,.  Figure 4 summarizes these goals, which have evolved slightly from what we published in the book (we refactored a few to make them more consumable).  Notice how each of the three phases (Inception, Construction, and Transition) are described by specific goals.  Also notice how some goals, such as Grow Team Members and Address Risk, are applicable throughout the entire lifecycle.

Figure 4. Goals throughout the lifecycle.

Mark Lines’ post Being Goal-Driven Requires Discipline explores the history of the figure above if you’re interested in how the goals have evolved since the DAD book was published.

The Advantage of Goals Over Prescription

First and foremost, DAD is a process decision framework.   One what that it achieves this through it’s goal-driven approach that guides people through the process-related decisions that they need to make to tailor and scale agile strategies to address the context of the situation that they face.  My experience is that there are several fundamental advantages to taking a goal driven approach to agile solution delivery.  A goal-driven approach:

1. Supports process tailoring. I think that Figures 2 and 3 make it very clear how DAD enables people to make intelligent process decisions.  I think that this is a huge improvement over previous process frameworks, particularly IBM’s Rational Unified Process (RUP) that provided a lot of great process advice (regardless of what some agilists may claim) but struggled to provide consumable process tailoring advice.
2. Enables effective scaling.  DAD provides a foundation from which to scale agile approaches.  An important part of scaling agile is to tailor your strategy to reflect the realities of the scaling factors which you face.  For example, consider your approach to exploring the initial scope of your effort (the goal captured in Figure 2).  A large team or a geographically distributed team will make different tailoring decisions than a small co-located team.  A team in a regulatory environment will make different decisions, particularly around amount of detail, than teams in non-regulatory environments.  These are just three of several scaling factors (more on this in a future blog posting, although you may postings in my agility at scale blog to be of interest).
3. Makes your process options very clear.  Figure 4, in combination with the more detailed goals diagrams (such as in Figures 2 and 3) make it very clear what you need to consider when tailoring an agile solution delivery process to meet the unique needs of the situation faced by your team.
4. Takes the guesswork out of extending agile methods.  Although it makes for wonderful marketing rhetoric, it’s disingenuous for people to claim that simple methods such as Scrum can be tailored to meet your actual needs.  Yes, I suppose this claim is true but how do you do so?  Shouldn’t you start with a full delivery lifecycle, not just a construction lifecycle? Shouldn’t the framework cover a wider range of issues, such as leadership and requirements management as Scrum does, technical issues as XP does, modeling and documentation as Agile Modeling does, and many other issues?  In short, shouldn’t it be a hybrid?   Finally, shouldn’t you be given some context-sensitive advice for tailoring the details, as we do with the goal-driven approach described here?
5. Makes it clear what risks you’re taking on.  By making your process decision options clear, and by describing the trade-offs associated with those options, DAD makes it very clear what risks you’re taking on.  Want to write a detailed requirement specification up front (yes, in a very small number of situations this is in fact a viable option for agile teams) then DAD is going to make it very clear what risks you’ve just taken on by doing so.  DAD also makes it clear when this decision is appropriate, so if you’re not in this situation then it  is likely time to rethink your approach.  Although we cannot prevent challenges such as a Water-Scrum-Fall approach where a heavy approach is taken to Inception and Transition and an agile/Scrum approach to Construction we can certainly make it very clear what the impact is of the decisions that led you to that approach.  Since the DAD book came out in June 2012 I’ve spoken with several people who have used the decision tables in it to argue against inappropriate process decisions on their projects.  In many situations the argument “that isn’t agile” falls on deaf ears, whereas “that will take longer and here’s why”, “that will be more expensive and here’s why”, “that will result in lower stakholder value and here’s why” will be listened to.
6. It hints at an agile maturity model.  Recently for the Cutter Consortium I wrote an article for their December 2012 Cutter IT Journal issue about how DAD and CMMI potentially fit together.  In that article I suggested that in the case of issues where the options are ordered there is a clearly an indication of agile maturity or sophistication.  Having said that I have no desire to wade into the agile maturity model morass, but I think it’s an important observation nonetheless.

So far we’ve identified two disadvantages to DAD’s goal-driven approach when working with customer organizations.  First, it makes the complexities of solution delivery explicit.  Although some of us want to believe that the simplistic strategies of other agile methods will get the job done we inherently know that software development, or more accurately solution delivery, is in fact a complex endeavor in practice.  Second, some people just want to be told what to do and actually prefer a prescriptive approach.  DAD mitigates this problem a bit by suggesting default starting points (shown in italized bold text in the goal diagrams) but even this can be overwhelming for some people.  Interestingly, when we were writing the book two of our 30+ reviewers were adamantly against giving people choices because they felt it was better to adopt a more prescriptive approach as we see in older agile methods.

I hope that this blog posting has given you some food for thought that you can leverage on your next agile project.  Got Discipline?

A Full Agile Delivery Lifecycle

One of the key aspects of the Disciplined Agile Delivery (DAD) process decision framework is that it promotes a full, end-to-end, solution delivery lifecycle.  The figure below overviews a high-level view of the DAD lifecycle.  It is a three-phase lifecycle, more on this in a minute, where you incrementally build a consumable solution over time.  We start with this high-level view so that we can explore several important concepts before diving into details.

First, the DAD lifecycle explicitly calls out three phases:

1. Inception. During this phase project initiation activities occur.  Although “phase” tends to be a swear word within the agile community, the reality is that the vast majority of teams do some up front work at the beginning of a project.  While some people will mistakenly refer to this effort as Sprint/Iteration 0 it is easy to observe that on average this effort takes longer than a single iteration (the 2009 Agile Project Initiation survey  found the average agile team spends 3.9 weeks in Inception whereas the most common iteration/sprint length is two weeks).  So in DAD’s Inception phase we do some very lightweight visioning activities to properly frame the project.  It takes discipline to keep Inception short.
2. Construction.  During this phase a DAD team will produce a potentially consumable solution on an incremental basis.  They may do so via a set of iterations (Sprints in Scrum parlance) or do so via a lean, continuous flow approach (different versions of the lifecycle are discussed later).  The team applies a hybrid of practices from Scrum, XP, Agile Modeling, Agile Data, and other methods to deliver the solution.
3. Transition.  DAD recognizes that for sophisticated enterprise agile projects deploying the solution to the stakeholders is often not a trivial exercise.  DAD teams, as well as the enterprise overall, will streamline their deployment processes so that over time this phase become shorters and ideally disappears from adopting continuous deployment strategies.  It takes discipline to evolve Transition from a phase to an activity.

Granted, there is more to the agile SDLC than just those phases.  First, there are pre-project aspects to portfolio management where potential projects or products are identified, priortized, and sufficiently funded to start an Inception phase effort.  After Transition a solution is operated and supported in production (or the marketplace in the case of commercial products).  Eventually, potentially after decades of use, a solution is retired (taken out of operation).   If we were to look at things from the point of view of IT, there are also cross-product/project concerns at the enterprise level such as enterprise architecture, portfolio management, strategic reuse, and more (for more details, see Enterprise Unified Process).  Having said this, these aspects of the overall lifecycle are not covered by DAD although do have an affect on how DAD teams work.

Earlier I said that the DAD framework does not prescribe a single lifecycle, unlike other agile methods such as Scrum.  In the DAD book we focused on two versions of the lifecycle, the basic/agile and the advanced/lean versions, although since the first version came out we’ve also described a continuous delivery version.  The point is that every team is in a unique situation, so for the DAD framework to be effective it must be flexible enough to support several versions of a lifecycle.  Let’s explore three such versions of the lifecycle.

The Agile/Basic DAD Lifecycle: Extending Scrum

The figure below presents a more detailed view of what we call the Agile/Basic DAD lifecycle which extends Scrum’s construction lifecycle.  In addition to this being a more detailed view of the lifecycle, there are several interesting aspects to this lifecycle:

1. It’s iteration based.  Like many agile methods, including both Scrum and XP, the solution is built incrementally in a time-boxed manner.  These timeboxes are called iterations (what Scrum calls sprints).
2. It uses non-Scrum terminology.  Although the lifecycle is Scrum-based we chose to use non-branded terminology in DAD, in the case of this diagram the term iteration instead of sprint.   The terminology doesn’t really matter, so if you’re more comfortable with Scrum terminology use that instead.
3. It shows inputs from outside the delivery lifecycle.  Although the overview diagram above showed only the delivery lifecycle, the detailed diagram below shows that something occurs before the project before Inception and that agile teams often get new requirements (in the form of change requests and defect reports) coming in from production.  These inputs provide important context for the overall delivery lifecycle.
4. There is a work item list, not a product backlog.  DAD has a greater scope than Scrum, and when you take this greater scope into account you begin to realize you need a more robust change management approach than Scrum’s product backlog.  Work items include requirements, defects, and other non-functionality oriented work such as training, vacations, and assisting other teams.  All of this work needs to be prioritized somehow, not just implementation of requirements.  For more on this, read Agile Best Practice: Prioritized Requirements.
5. In includes explicit milestones.  Along the bottom of the lifecycle diagram there is an indication of suggested light-weight milestones that DAD teams should strive to meet.  Such milestones are an important aspect of agile governance.

We call this the basic/agile lifecycle because it’s likely where you’re going to start with DAD.  Common scenarios for adopting this version of the lifecycle include situations where you’re extending Scrum to be sufficient for your needs or you’re transitioning from RUP to a disciplined agile approach.

The Advanced/Lean DAD Lifecycle

The figure below depicts what we call the Advanced/Lean DAD lifecycle.  There are several interesting features to this lifecycle:

1. It supports a continuous flow of delivery.  In this lifecycle the solution is deployed as often, and whenever, it makes sense to do so.  Work is pulled into the team when there is capacity to do it, not on the regular heartbeat of an iteration.
2. Practices are on their own cadences.  With iterations/sprints many practices (detailed planning, retrospectives, demos, detailed modeling, and so on) are effectively put on the same cadence, that of the iteration.  With a lean approach the observation is that you should do something when it makes sense to do it, not when the calendar indicates that you’re scheduled to do it.
3. It has a work item pool.  All work items are not created equal.  Although you may choose to prioritize some work in the “standard” manner, either a value-driven approach as Scrum suggests or a risk-value driven approach as both DAD and RUP suggests, but other work may fit this strategy.  Some work, particularly that resulting from legislation, is date driven.  Some work must be expedited, such as fixing a severity one production problem.  So, a work item pool and not a prioritized stack makes a bit more sense when you recognize these realities.

We call this the advanced/lean lifecycle because it’s something we see teams evolve to over time.  They often start with the basic lifecycle described earlier but as they learn, as they improve the way that they work, the lifecycle they follow becomes leaner.

The Continuous Delivery DAD Lifecycle

The figure below depicts a Continuous Delivery version of the DAD lifecycle.  It is basically a leaner version of the previous lifecycle where the product is shipped into production or the marketplace on a very regular basis.  This could be often as daily, although weekly or monthly is quite common too.

Why Support Several Lifecycles?

This is a good question.  First, one lifecycle clearly does not fit all.  Teams find themselves in a unique situation: team members are unique individuals with their own skills and preferences for working, let alone the scaling/tailoring factors such as team size, geographic distribution, domain complexity, organizational culture, and so on which vary by team.  Because teams find themselves in a wide variety of situations shouldn’t a framework such as DAD support several lifecycles?  Furthermore, just from the raging debates on various agile discussion forums, in agile user groups, at agile conferences, and even within organizations themselves it’s very easy to empirically observe that agile teams are in fact following different types of lifecycles.

Second, we were uncomfortable with the idea of prescribing a single lifecycle.  We wanted to avoid prescription in the DAD framework to begin with, for all the rhetoric about the evils of prescription within the Scrum community it’s clear that Scrum is in fact quite prescriptive in practice.  We’ve seen many teams get into trouble trying to follow agile methods such as Scrum to the letter of the law in an environment where “Scrum out of the box” really isn’t a good fit.

Third, we’re firm believers in process improvement.  If you are in fact improving your process as you learn, is it not realistic that your lifecycle will evolve over time?  Of course it will.  We’ve seen teams start with something close to the basic/agile lifecycle that evolve it to the advanced/lean or continuous delivery lifecycles over time.
Process improvement

The Downside of Supporting Several Lifecycles

There is clearly a downside to explicitly supporting several lifecycles.   By doing so we explicitly admit that DAD teams will be follow a unique tailoring of the process that best fits their situation, a concept that can be ant-thetical in organizations still clinging to the notion of repeatable processes (DAD promotes repeatable results over repeatable processes).   It also makes it clear that enterprise professionals, such as enterprise architects or data management groups, need to be sufficiently flexible to support several delivery lifecycles.  Instead of suboptimizing around such enterprise processes (i.e. forcing all project teams to conform to a single data management strategy) you instead want to build enterprise teams that are sufficiently skilled and flexible to support delivery teams in a manner which best suits the delivery teams.  Finally, it’s clear that governance needs to be results based, not artifact based.  The good news is that DAD builds effective governance right into the framework.

Roles in Disciplined Agile Delivery

The Disciplined Agile Delivery (DAD) framework suggests a robust set of roles for agile solution delivery.  These roles are overviewed in the following figure:

Primary Roles

Primary roles are commonly found regardless of the level of scale.  There are five primary roles:

1. Stakeholder.  A stakeholder is someone who is materially impacted by the outcome of the solution.  In this regard, the stakeholder is clearly more than an end-user: A stakeholder could be a direct user, indirect user, manager of users, senior manager, operations staff member, the “gold owner” who funds the project, support (help desk) staff member, auditors, your program/portfolio manager, developers working on other systems that integrate or interact with the one under development, maintenance professionals potentially affected by the development and/or deployment of a software project.   DAD teams will ideally work together with their stakeholders daily throughout the project.
2. Team Member.  The role of team member focuses on producing the actual solution for stakeholders.  Team members will perform testing, analysis, architecture, design, programming, planning, estimation, and many more activities as appropriate throughout the project.  Note that not every team member will have every single one of these skills, at least not yet, but they will have a subset of them and they will strive to gain more skills over time.  Team members are sometimes described by core agile methods as “developers” or simply as programmers.  However, in DAD we recognize that not every team member necessarily writes code.   Team members will identify tasks, estimate tasks, “sign-up” for tasks, perform the tasks, and track their status towards completion.
3. Team Lead.  On agile projects the role of the traditional project manager changes substantially, and in fact the term project manager is now unfortunately frowned upon.  The agile community has focused on project or team leadership over team management, observing that the best “managers” prioritize leadership over technical management issues such as planning and estimation.  An important aspect of self-organizing teams is that the team lead facilitates or guides the team in performing technical management activities instead of taking on these responsibilities him or herself.  The team lead is a servant-leader to the team, creating and maintaining the conditions that allow the team to be successful.  The team lead is also an agile coach, helping to keep the team focused on delivering work items and fulfilling their iteration goals and commitments that they have made to the product owner.  He or she acts as a true leader, facilitating communication, empowering them to self-optimize their processes, ensuring that the team has the resources that it needs, and removes any impediments to the team (issue resolution) in a timely manner.  When teams are self organizing, effective leadership is crucial to your success.
4. Product Owner.  In a system with hundreds or thousands of requirements it is often difficult to get answers to questions regarding the requirements.  The product owner is the one individual on the team who speaks as the “one voice of the customer”.  He or she represents the needs and desires of the stakeholder community to the agile delivery team.  As such, he or she clarifies any details regarding the solution and is also responsible for maintaining a prioritized list of work items that the team will implement to deliver the solution.  While the product owner may not be able to answer all questions, it is their responsibility to track down the answer in a timely manner so that the team can stay focused on their tasks.  Having a product owner working closely with the team to answer any question about work items as they are being implemented substantially reduces the need for requirements, testing, and design documentation.  You will of course still have need for deliverable documentation such as operations manuals, support  manuals, and user guides to name a few. Each DAD team, or subteam in the case of large programmes organized into a team of teams, has a single product owner.  A secondary goal for a product owner is to represent the work of the agile team to the stakeholder community.  This includes arranging demonstrations of the solution as it evolves and communicating project status to key stakeholders.
5. Architecture Owner.  Architecture is a key source of project risk and someone needs to be responsible for ensuring the team mitigates this risk.  As a result the DAD framework explicitly includes Agile Modeling’s role of architecture owner. The architecture owner is the person who owns the architecture decisions for the team and who facilitates the creation and evolution of the overall solution design. The person in the role of team lead will often also be in the role of architecture owner on small teams.  This isn’t always the case, particularly at scale, but it is very common for smaller agile teams.  Although the architecture owner is typically the senior developer on the team – and sometimes may be known as the technical architect, software architect, or solution architect – it should be noted that this is not a hierarchical position into which other team members report.  He or she is just like any other team member and is expected to sign-up and deliver work related to tasks like any other team member.  Architecture owners should have a technical background and a solid understanding of the business domain.

Secondary Roles

Secondary roles are typically introduced, often on a temporary basis, to address scaling issues.  There are five secondary roles:

1. Specialist.  Although most agile team members are generalizing specialists, sometimes, particularly at scale, specialists are required.  For example, on large teams or in complex domains one or more agile business analysts may join the team to help you to explore the requirements for what you’re building.  On very large teams a program manager may be required to coordinate the team leads on various subteams.  You will also see specialists on DAD teams when generalizing specialists aren’t available – when your organization is new to agile it may be staffed primarily with specialists who haven’t yet made the transition to generalizing specialists.
2. Domain Expert.  The product owner represents a wide range of stakeholders, not just end users, so it isn’t reasonable to expect them to be experts in every nuance in your domain, something that is particularly true with complex domains. The product owner will sometimes bring in domain experts to work with the team, for example, a tax expert to explain the details of a requirement or the sponsoring executive to explain the vision for the project.
3. Technical Expert.  Sometimes the team needs the help of technical experts, such as a build master to set up their build scripts, an agile database administrator to help design and test their database, a user experience (UX) expert to help design a usable interface, or a security expert to provide advice around writing a secure system. Technical experts are brought in on an as-needed, temporary basis to help the team overcome a difficult problem and to transfer their skills to one or more developers on the team.  Technical experts are often working on other teams that are responsible for enterprise-level technical concerns or are simply specialists on loan to your team from other delivery teams.
4. Independent Tester.Although the majority of the testing is done by the people on the DAD team themselves, some DAD teams are supported by an independent test team working in parallel that validates their work throughout the lifecycle.  This independent test team is typically needed for agility at scale situations within complex domains, using complex technology, or addressing regulatory compliance issues.
5. Integrator.For large DAD teams which have been organized into a team of subteams, the subteams are typically responsible for one or more subsystems or features. The larger the overall team, generally the larger and more complicated the system being built. In these situations, the overall team may require one or more people in the role of integrator responsible for building the entire system from its various subsystems.  On smaller teams or in simpler situations the Architecture Owner is typically responsible for ensuing integration, a responsibility that is picked up by the integrator(s) for more complex environments.  Integrators often work closely with the independent test team, if there is one, to perform system integration testing regularly throughout the project.  This integrator role is typically only needed at scale for complex technical solutions.

Why So Many Roles?

This is a common question that we get from people familiar with Scrum.  Scrum has three roles – ScrumMaster, Product Owner, and Team Member - so why does DAD need ten?  The primary issue is one of scope.  Scrum mostly focuses on leadership and change management aspects during Construction and therefore has roles which reflect this.  DAD on the other hand explicitly focuses on the entire delivery lifecycle and all aspects of solution delivery, including the technical aspects that Scrum leaves out.  So, with a larger scope comes more roles.  For example, because DAD encompasses agile architecture issues it includes an Architecture Owner role.  Scrum doesn’t address architecture so doesn’t include such a role.

Some Important Thoughts About Roles

On a DAD project any given person will be in one or more roles, an individual can change their role(s) over time, and any given role will have zero or more people performing it at any given time.  For example, Peter may be in the role of team member and architecture owner right now but step into the role of team lead next month when Carol, the existing team lead, goes on vacation.

Roles are not positions, nor are they meant to be.  For example, Jane may be in the role of stakeholder for your project but have the position of Chief Financial Officer (CFO) within your organization. In fact, although there may be hundreds of stakeholders of your project none of them is likely to have a position of “stakeholder.”

Agile deemphasizes specialized roles and considers all team members equal – everyone pitches in to deliver a working solution regardless of their job description. An implication of this is that with the exception of stakeholder everyone is effectively in the role of team member.

Traditional roles, such as business analyst and project manager, do not appear in DAD.  The goals which people in traditional roles try to achieve, for example in the case of a business analyst to understand and communicate the stakeholder needs/intent for the solution, are still addressed in DAD but in different ways by different roles.  There isn’t a perfect one-to-one match between any given traditional role and a DAD role, but as you will find in the Disciplined Agile Delivery book there are reasonable transition strategies.  The critical thing for traditionalists to understand is that because the underlying paradigm and strategy has changed, they too must change to reflect the DAD approach.

Material for this blog posting was modified from Chapter 4 of Disciplined Agile Delivery: A Practitioner’s Guide to Agile Software Delivery in the Enterprise (IBM Press, 2012)

Advanced DAD workshop in Zurich, April 8-9 2013

I will be teaching a two-day Advanced DAD workshop in Zurich on April 8&9th.  The workshop is hands-on and is meant for intermediate-level people.  People new to agile can still attend of course but may find it a bit overwhelming.  I’m always happy to stay late and help anyone struggling, and happier yet to do so over a beer.  Anyway, click here for more information.

Adopting Agile Governance Requires Discipline

Governance establishes chains of responsibil­ity, authority and communication in support of the overall enterprise’s goals and strategy. It also establishes measurements, policies, standards and control mechanisms to enable people to carry out their roles and responsibilities effectively. You do this by balancing risk versus return on investment (ROI), setting in place effective processes and practices, defining the direction and goals for the department, and defining the roles that people play with and within the department.

Governance and management are two different things. Governance looks at a team from the outside, treating it as a system that needs to have the appropriate structure and processes in place to provide a stream of value. Management, on the other hand, occurs inside the team and ensures that the structure and processes are implemented effectively.  The Disciplined Agile Delivery (DAD) process framework characterizes governance as an element of enterprise awareness from the team’s point of view because governance looks at the team from the outside.

It is easier to avoid your traditional governance and tell management that “agile is different” than it is to work with your governors to adapt your governance to properly guide the delivery of your agile teams.  As we described in the book every organization has a necessary degree of governance and there are ways to make it especially effective on agile initiatives.  It takes discipline to work with your governors to help them understand how disciplined agile teams operate and then discipline to accept and conform to the resulting governance process.

Our experience is that the most effective way to govern agile teams is to focus on collaborative strategies that strive to enable and motivate team members implicitly. For example, the traditional approach to motivating a team to provide good ROI would be to force them to develop and commit to an “accurate” project budget, and then periodically review their spending to ensure they’re on track. An agile approach would be to ask the team to provide a ranged estimate of what they believe the cost will be so as to set expectations about future funding requirements.  Then the team works in priority order as defined by their stakeholders, visibly providing real business value through the incremental delivery of a potentially consumable solution.  Costs are tracked via the team’s burn rate (the fully burdened cost of the people on the team plus any capital outlays for equipment or facilities) and value is tracked by the stakeholders’ continuing satisfaction (hopefully) with what the team is delivering for that cost.  In short, a traditional approach often measures financial progress against a budget whereas an agile approach seeks to maximize stakeholder value for their investment by always working on the most valuable functionality at the time.

The DAD framework includes several important agile governance strategies:

• Adopting a risk-value driven lifecycle
• Explicit, light-weight milestone reviews
• Agile enterprise teams that work closely with agile teams
• Regular coordination meetings (daily standups in Scrum)
• Iteration/sprint demos
• All-hands demos
• Follow enterprise guidelines (coding standards, UI standards, data conventions, …)
• Retrospectives, and better yet measured improvement
• Increased stakeholder visibility
• Development intelligences (BI for IT)
• Aligning agile team governance with other governance (operations, security, data, …) strategies
• Agile measurement/metrics programs
• Active risk mitigation
• Robust role definitions

Many of the strategies described above are “standard” agile governance strategies, and a few are unique to DAD.  It requires discipline to adopt and then execute on effective governance strategies, particularly in organizations where you already have a strong traditional governance program in place.

Disciplined Agile Delivery on LinkedIn

We recently started a discussion group on LinkedIn called, you guessed it, Disciplined Agile Delivery (DAD).  You’re welcome to join and get involved in the conversation.

Strategies for Verifying Non-Functional Requirements

Early in the lifecycle, during the Inception phase, disciplined agile teams will invest some time in initial requirements envisioning and initial architecture envisioning. One of the issues to be considered as part of requirements envisioning is to identify non-functional requirement (NFRs), also called quality of service (QoS) or simply quality requirements. The NFRs will drive many of your technical decisions that you make when envisioning your initial architectural strategy. These NFRs should be captured someone, in a previous blog I explored the options available to you, and implemented during Construction. It isn’t sufficient to simply implement the NFRs, you must also validate that you have done so appropriately. In this blog posting I overview a collection of agile strategies that you can apply to validate NFRs.

A mainstay of agile validation is the philosophy of whole team testing. The basic idea is that the team itself is responsible for validating its own work, they don’t simply write some code and then throw it over the wall to testers to validate. For organizations new to agile this means that testers sit side-by-side with developers, working together and learning from one another in a collaborative manner. Eventually people become generalizing specialists, T-skilled people, who have sufficient testing skills (and other skills).

Minimally your developers should be performing regression testing to the best of their ability, adopting a continuous integration (CI) strategy in which the regression test suite(s) are run automatically many times a day.  Advanced agile teams will take a test-driven development (TDD) approach where a single test is written just before sufficient production code which fulfills that test.  Regardless of when tests are written by the development team, either before or after the writing of the production code, some tests will validate functional requirements and some will validate non-functional requirements.

Whole team testing is great in theory, and it is strategy that I wholeheartedly recommend, but in some situations it proves insufficient.  It is wonderful to strive to have teams with sufficient skills to get the job done, but sometimes the situation is too complex to allow that.  There are some types of NFRs which require significant expertise to address properly: NFRs pertaining to security, usability, and reliability for example.  To validate these types of requirements, worse yet even to identify them, requires skill and sometimes even specialized (read expensive) tooling.  It would be a stretch to assume that all of your delivery teams will have this expertise and access to these tools.

Recognizing that whole team testing may not sufficiently address validating NFRs many organizations will supplement their whole team testing efforts with parallel independent testing  .  With this approach a delivery team makes their working builds available to a test team on a regular basis, minimally at the end of each iteration, and the testers perform the types of testing on it that the delivery team is either unable or unlikely to perform.  Knowing that some classes of NFRs may be missed by the team, independent test teams will look for those types of defects.  They will also perform pre-production system integration testing and exploratory testing to name a few.  Parallel independent testing is also common in regulatory compliance environments.

From a verification point of view some agile teams will perform either formal or informal reviews.  Experienced agilists prefer to avoid reviews due to their inherently long feedback cycle, which increases the average cost of addressing found defects, in favor of non-solo development strategies such as pair programming and modeling with others.  The challenge with non-solo strategies is that managers unfamiliar with agile techniques, or perhaps the real problem is that they’re still overly influenced by disproved traditional theories of yesteryear, believe that non-solo strategies reduce team productivity.  When done right non-solo strategies increase overall productivity, but the political battle required to convince management to allow your team to succeed often isn’t worth the trouble.

Another strategy for validating NFRs code analysis, both dynamic and static.  There is a range of analysis tools available to you that can address NFR types such as security, performance, and more.  These tools will not only identify potential problems with your code many of them will also provide summaries of what they found, metrics that you can leverage in your automated project dashboards.   This strategy of leveraging tool-generated metrics such as this is a technique which IBM calls Development Intelligence and is highly suggested as an enabler of agile governance in the DAD framework. Disciplined agile teams will include invocation of code analysis tools from you CI scripts to support continuous validation throughout the lifecycle.

Your least effective validation option is end-of-lifecycle testing, in the traditional development world this would be referred to as a testing phase.  The problem with this strategy is that you in effect push significant risk, and significant costs, to the end of the lifecycle.  It has been known for several decades know that the average cost of fixing defects rises the longer it takes you to identify them, motivating you to adopt the more agile forms of testing that I described earlier.  Having said that I still run into organizations in the process of adopting agile techniques that haven’t really made embraced agile, as a result still leave most of their testing effort to the least effective time to do such work.  If you find yourself in that situation you will need to validate NFRs in addition to functional requirements.

To summarize, you have many options for validating NFRs on agile delivery teams.  The secret is to pick the right one(s) for the situation that you find yourself in.  The DAD framework helps to guide you through these important process decisions, describing your options and the trade-offs associated with each one.  For a more detailed discussion of agile validation techniques you may find my article Agile Testing and Quality Strategies to be of value.