Developing agility

Posted on June 28, 2010 by


At Calvis we often talk about ourselves as “agile” and using “agile methods”, but these terms, as with almost all business shorthand, risk becoming vacuous linguistic space-fillers. However, agility is at the core of who we are for a reason, so I want to explore that reason and refill the semantics of the language.

When we think of agility, a common image is that of a cat (big or of the domestic variety), perhaps performing a seemingly impossible leap, perching on a narrowest of ledges, or navigating a treacherous chase or escape at high speed. These all capture the key elements of agility: being quick and nimble.

In an organizational context, agility is about the same things. Responding quickly to the obstacles that are put in our path and accurately manoeuvring around or through them without losing time.

This of course sounds fantastic, but it is in the “how” that we must linger. Take, for example, a pivotal aspect of a recent project we have been completed. The proposal involved a significant element of bespoke software development and our proposed approach involved using our agile method.

The method involves two core facets that underpin its agility:

  • an iterative approach that splits the overall development into three discrete stages with clear functionally specified deliverables out of each; and
  • a weekly cycle of visualization reviews with end users where the latest version of the developing software together with current designs is reviewed (visually – i.e. by looking at the real or mocked-up system rather than specification and papers) and feedback provided into the development iteration.

This approach clearly requires an additional set of meetings (weekly) with users and some of the project team, and adds a project management burden. So where are the benefits?

Some are fairly obvious:

  • weekly visualizations engage the end users earlier in the process and more immersively, leading to more useful feedback, greater user ownership and a closer fit between what is delivered and what the users expect will be delivered; and
  • breaking larger projects into smaller parts based on separation of functionality has been a well-trodden path for software development and improves the project manager’s ability to monitor progress and so manage any exceptions.

However, these benefits really only scratch the surface. To understand why our agile approach really pays dividends, we need to explode some commonly held myths about software development projects.

MYTH 1: A successful project is one where you understand the requirements so well that you can (even if you don’t) provide a detailed functional specification before you start writing code.

Even today, a remarkable number of project practitioners subscribe mindlessly to this myth. This myth was part of the belief system that actually gave birth to structured project management methods.

The problem with this myth is that it is partly true. It is a theory in direct response to the observation that projects that went badly wrong also had poorly specified outcomes. The assumption that these two observations are correlated has some veracity, but is not the whole story.

When structured methods were first introduced they all followed what has become known as the “traditional waterfall approach”. This approach has essentially five parts: (i) initiation; (ii) planning or design; (iii) execution or production; (iv) monitoring and control; and (v) completion.

The waterfall approach works very well for small, well specified projects, but for bigger projects it starts to break down. Some of the first work to investigate this was done by NASA and they created a model called the “Cone of Uncertainty”.

The principle is that at the outset, there is so much uncertainty that any estimates can be a factor of 4 out either way. As the project progresses, the uncertainty diminishes until it reaches zero – on the day the project is finished.

This model is a giant step forward in project thinking because it implicitly recognizes that it is impossible to pre-plan a project to succeed on time and within budget: it is only possible to either achieve that or not, because you can only know when you’ve finished. This is not only because the effort required to completely specify a software project very quickly outgrows the value the project will deliver, but also because the business value proposition is very likely to change during the project – either because the business itself achieves a better understanding of what is needed or because market conditions or other external factors change the business case (and external factors out of the business’ control account for over 30% of what makes a business succeed).

MYTH 2: A comprehensive project plan up front will avoid cost or time overruns

That isn’t to say planning doesn’t help of course – it is vital and necessary. Rather it is to say that we need to plan on the assumption that we are uncertain of what will be involved. NASA’s solution was simply to monitor the statistical factors to apply at each stage based on the average of their projects and then build that into the plans for subsequent projects.

However, agile methods (formerly known as “light-weight methods”) started to look at how the development method itself can be changed to accommodate this degree of uncertainty and to make time spent on activities more productive.

MYTH 3: It is necessary to complete all of the architecture and design up-front to avoid costly redesign later on

To improve productivity we need to ensure that each activity is scheduled to be as late as it can be without affecting the critical path. This approach, better known from its application in manufacturing, is called just-in-time (or “JIT”).

In manufacturing, JIT saves time and money by removing the need to store and manage parts that have been made in advance by making things and delivering them immediately precisely when they are needed. This also means that if the requirements change there is no “stock” that needs to be relocated or disposed of.

In software development, this translates to only undertaking tasks like architecture and design, specification, and development just as they are required. This differs fundamentally to the waterfall approach that says do all the design up-front, then all the reviewing, then all the development. In the agile method, design takes place immediately before development and review is undertaken during the development.

In practice, as has been shown particularly in the realms of “extreme programming”, it is more efficient to only design for the immediate requirements (perhaps the current 4-8 week iteration). This is because to do more involves second guessing the cone-of-uncertainty and that means you will end of either (i) getting the design wrong because there was an uncertainty that undermined the design, or (ii) you will over-specify the design to accommodate as much of the uncertainty as possible, and in doing so increase the cost of development and implementation.

MYTH 4: To estimate a project accurately we need a detailed functional specification

The overall size of an agile project is not estimated based on detailed functional specification or design (which, as we have discovered, are elusive at the beginning of a project in any event) but on much more tangible and real factors, like business objectives, and experience-centric assessments of complexity.

It still isn’t a science, but this is much more akin to how more established engineering industries estimate. When a new road is being built, the estimate isn’t put together by counting the number of 20mm aggregate stones that will be required, but by multiplying the length of each segment of the road by a known cost-per-mile based on conditions. It won’t be perfect every time, but it will be a good estimate and it will save a lot of up-front time on the project and therefore bring forward the realization of the benefits by the business.

Our agile method, therefore, means that design decisions are only taken when they are needed during the project. This means they can benefit from all of the learning that has gone on during the project leading to faster, more accurate decisions – agile decisions, you might say!

By focussing the development on tiny work packages that are reviewed weekly within an iterative framework, we are able to deliver projects that are as cost effective and fast as possible, whilst being in a continual position to assess whether the time and cost of the project continues to represent good business value.

Agile projects, far from being unpredictable or out of control, they are in fact the most controlled and predictable projects because they avoiding wasting time traditionally lost in poor communication or poor specification. Agility in this context is perhaps more like a gymnast than a cat: certainly quick and nimble but also perfectly prepared, planned and executed.