Coaching: How Not What

Yesterday evening I was playing a game of 5-a-side football (soccer to my American readers) with some friends. One of the players on my team was a particularly competent player who was also very vocal in offering advice to his team mates on what to do.

For example, when a player from the opposing team was approaching me with the ball he would offer sage advice such as "Don't let them get past you!". This was rather stating the obvious, as I was quite clearly not planning on letting the opponent go past me and take a clear shot on goal.

In this particular situation my challenge was not in knowing what I should do, but in having the skill and experience necessary to carry out the task. No amount of shouted commands were suddenly going to increase my skill to a level beyond that which I currently possess.

This got me thinking about how we coach people in programming and agile practices. How often do we say to someone something like "that needs to be refactored" or "you need to keep your functions small"? Or, we might say to an agile team "don't overcommit yourselves this sprint". All of these are instructions to do something, made with the assumption that the people receiving them have the skill and experience necessary to carry out the actions.

What if, like me on the football field, the recipients of these instructions knows what they should do, but not how to do it. Clearly we are not being successful coaches in these cases. We need to focus much more on the how rather than the what. Advice offered should be enabling, providing the recipient with a way of learning and gaining experience.

For example, "if you pull this bit of code out into a separate function then this loop becomes much simpler" is much more helpful than "that needs to be refactored".

As coaches we need to be mindful of how we communicate in order to improve the people under our guidance. It's very easy to let it slip and just become another player shouting advice to those who haven't yet gained the skill necessary to implement that advice.

Making Agile Go Fast

Just recently I was speaking with a potential customer who had concerns about the velocity of their Scrum teams. They had recently (9 months ago) switched to an agile approach and had expected to get much more delivered than they had.

Now they were looking in to scaling up the number of Scrum teams on the project to try and increase velocity and deliver more quickly. In my experience of many agile projects, increasing the number of people on the project, especially one that is struggling, never has the desired effect of making things go faster. The advice to this customer was to therefore look at making their existing teams and process more effective rather than increasing the number of people on the project.

This got me thinking about what makes agile projects go slow. Across a number of projects that I have experienced, I’ve identified four common traits: two related to people and process and two related to technical concerns.

The Wrong People

Agile is all about using the right people. In fact this is true for any project or process, but for agile it is even more important. Use the right people and the agile process will work. Use the wrong ones and you will struggle.

The people who you don’t want on your agile project are those who are not disciplined enough to follow sound engineering practices and those who try to build complex frameworks/toolkits and who conduct personal science projects on your codebase. If you want to go fast then it’s vital that you have a clean, simple and well-refactored codebase that is backed up by a set of quality tests. Team members who don’t support this vision slow down the whole team.

To get back on track, remove those devs who aren’t following good engineering discipline and those who keep making things more complex. Bring in experienced agile developers to pair with the remaining team, giving them the responsibility of ensuring quality engineering practices are followed.

Putting The Process First

Another common mistake of companies starting out with agile projects is that they assume the process will make them go faster. Sorry to disappoint, but this is just not true. What makes agile teams go fast is creating open channels of communication and self organising and empowered teams.

If an organisation can’t create an excellent environment for communicating then no agile process will help. Agile helps teams go faster by eliminating the unnecessary documentation and waterfall steps. However, this information transfer must still happen and it happens through closer communication and collaboration.

Sadly, many organisations create agile teams and give them a process to follow but then insist on telling them how to organise, what tools and techniques to use, how to work and exactly what to do. This isn’t agile, it’s just another way of organising a traditional team. If you want agile to go fast then the team need total control over how they work and what they do.

To get back on track from here, relax on the process. Work on increasing communications. Give the agile teams the chance to decide how they will organise and work. Give them control and they will go faster. Agile is not a process, it’s about letting people get the job done.

Using The Wrong Technologies

So many agile projects struggle because they are using the wrong technologies. When this is the case it’s usually because the technology approach has been foisted on the team by an ‘ivory-tower’ architect or some senior manager who was wooed by the sales consultant from a large vendor. In other cases it may be that the team are forced into using a ‘corporate standard’ technology set.

In many cases the team will repeatedly identify the technology as a major factor in retrospectives. However, more often than not they will be told that this is not something that can be changed and that they have to stick with it. Why throw away hundreds of thousands or even millions on staff costs when backtracking on some poor technology choices would make so much difference.

Again, giving the teams the authority to make technology decisions is essential. Allowing technology choices to be reversed if they prove to be hindering delivery is also an important part of being agile.

Making The Architecture Too Complex

Many software systems are designed, or become, far too complex. In an agile environment this is a guaranteed way of making things go slower. Software must be kept as simple as possible so that it is easy to continually change and enhance.

If the architecture of an agile system has become so complex that it hinders development then it must be revisited and greatly simplified. Layers must be removed, interfaces reduced and code and technologies radically refactored to make everything simpler. Once that is done then the project must work all out on stopping it from getting more complex again in the future.

Only by using the right people, correctly empowered and communicating can the team have the ability to go fast and be truly agile. Only by making the correct technology choices, reversing bad technology choices and keeping everything simple can the ability to turned into real agile delivery.

Pairing is Wearing

In this post back in August 2011 I was talking about the many benefits of Pair Programming. Having now been full-time pair programming for almost an entire year, I wanted to mention one of the downsides that I have discovered.

A Holiday

I've just finished my holiday. It wasn't the best one ever, mainly due to camping in the UK in June: which meant heavy rain and strong winds. Still, we had a good time and it was very relaxing.

As the week progressed and I had a chance to relax, unwind and get some lie-ins I started to discover that my mind was starting to become more alert. I was feeling less tired. Great ideas were starting to flow again. It was at this point I realised just how weary I had been at the start of the holiday. I had known I was feeling pretty knackered, just not quite how much.

This then got me thinking as the the possible cause of this tiredness and how it could gradually build up over time without me being fully aware of it. I looked back at what I've been doing for the past year...

• I've been travelling a lot to get to/from work - but less than my previous job.
• I've been learning lots of new stuff - but I'm a compulsive learner, so nothing new here.
• We've put at least 3 major releases live - but I've put plenty of software live before.
• We've been pairing full time - this is new!

Is Pairing More Tiring?

The conclusions that I have had to draw is that working in a paired environment is more mentally tiring that working alone or individually within a team. There's more of a requirement to stay focused when working as a pair - it's particularly noticable when one member of the pair is feeling unwell or tired. Also while pairing you are multitasking more - writing code, thinking about solutions but also communicating your thoughts and maintaining a running dialogue. All of this extra work requires extra mental energy and thus increases fatigue.

So, how can we pair over a period of time without burning out? Here's some thoughts that I hope to experiment with over the coming months:

• Take more frequent and longer breaks than you would do while working individually. The productivity gain of working in pairs easily allows this to be supported
• Do a lot more walking at lunchtime!
• Try to provide team members regular tasks that they can undertake outside of a pair to give their mental energy drain a rest. Stuff that doesn't require extensive multitasking.

Anyone else got any thoughts or ideas?

Done and Dusted

In a recent retrospective our team dug out our Definition of Done and gave it a good review. We ended up removing a number of unnecessary items. This got me thinking about what is Done?

In previous places that I have worked the Definition of Done was a long check-list of things that had to have been completed and verified. Is this the best that a philosophy that claims to value "individuals and interactions over processes and tools" can come up with? This got me thinking...

Isn’t Done actually just the state when the team agree that the story is complete? Why should it need to be any more than that?

Done: The state when all team members agree that the story is complete

We could expand this slightly to define the roles that the different team members play in this definition:

• Developers agree that the code is complete, tested and refactored to an acceptable standard
• QAs agree that the acceptance criteria for the story has been met, that the functionality works and that they have not found any defects introduced by the new feature
• BAs and POs agree that the feature meets the business objectives for the story
• (we could also add points for DBAs, Ops etc, as needed)

So, if we accept that the team can decide for themselves when something is Done, why do we end up with long checklists, often including a whole range of metrics? It could be because 'management' aren’t able to release their need to monitor everything and produce nice reports. However, it’s more likely a matter of trust.

Teams are just not trusted to self-organise and make good decisions. If the team say that a story is Done then for many managers there is an immediate suspicion that the team is not telling the truth and is trying to cut corners. They then require that there is concrete proof that this is not the case. This is despite the fact that a group of professionals all happen to agree that it is Done. It just doesn’t make sense.

If the team has never released a story as Done when it clearly wasn’t then why doubt they will suddenly stop doing this in the future? Show trust in the team that is delivering completed work.

If the team are marking stories as Done when they clearly aren’t then there is a much more serious problem with the team and they probably either need replacing, mentoring or a firm guiding hand. Adding a requirement to satisfy a long checklist and provide associated metrics will never improve a bad team.

As with many things agile, it’s not about the process: it’s about changing attitudes; it’s about trusting teams to deliver the best solutions possible; it’s about not bogging the teams down with unnecessary controls.

Increasing Agile Capacity

A scenario: you have a successful agile project. It’s made up of a number of small teams, each working on a separate component of the application suite. This has been working extremely well. You are capacity planning for the coming six months and you identify that one of these teams is likely to become a bottleneck. There are just too many stories that fit into their component area. How do you deal with this?

The above is something that looks likely on one of the projects that I am working on. I’ve been putting in some thinking on how this might pan out. This post documents my thoughts.

Traditional Reaction

There are typically two standard reactions to above problem: grow the team that is likely to be the bottleneck, or add an additional team to work on the component. In my experience neither of these works. Adding more people is rarely ever a good way to make a project go faster!

Growing the team

The best agile teams I have worked on have had one thing in common. They have all had between 4 and 6 devs, a QA, and a Scrum Master. As soon as the team gets any bigger than this I’ve found that the following occurs:

• The performance gain of adding each additional dev is less each time.
• It becomes much more difficult to gain consensus.
• Team meetings become more long-winded and less productive.
• Communication becomes more difficult.
• It becomes difficult to maintain team values.
• Code cruft (technical debt) increases.

Thus, adding additional devs and QAs to a team, either permanently or by borrowing from another team, just doesn’t work. Typically I’ve even found that the team goes slower as it has to spend more time cleaning up after itself.

Adding temporary devs (perhaps for a day or two) is, I’ve found, often worse than adding a permanent dev as there is less sense of ownership and shared team values. Working with good developers makes this less so, but I still believe it occurs more frequently than we like to admit.

Adding more teams

Having multiple teams working on a single component’s code base just doesn’t work well. I worked on one project where we had four teams working on the same code. We were tripping over each other all the time. Particular problems I have seen working in this way:

• Communications between the teams becomes a big overhead.
• More difficult to gain consensus on major technical decisions.
• Scheduling work to avoid multiple pairs changing the same code is very difficult.
• Very difficult to maintain shared values across multiple teams.
• One team’s nice code is another team’s cruft.
• Code cruft increases because one team is less willing to refactor another team’s code.
• Different approaches to achieving the same implementation become more prevalent.
• One team undoes another team’s work because they disagree with how it was done.
• Important aspects like performance tend to slip between the cracks.
• Productivity of the combined teams is far less than the sum of the productivity of each team working independently.

Multiple teams working on the same code base is a recipe for disaster. It results in more overhead, lower code quality and more cruft. Also, you get far less productivity gain that you would had you allowed each team to work independently on their own component.

Other Approaches

So, given that there is more work than the measured velocity of the team shows they can achieve in the time available. And, given that we don’t want to grow the team or add more teams. How do we solve the problem?

The aim is to keep small teams who can each work on an area of code that they have sole ownership of. This will give them the incentive to ensure that this component is clean, simple, maintainable and has a low level of cruft. It will also maximise the productivity of each individual team by minimising communication overhead.

I think a combination of four specific solutions are the best approach. These being: reducing waste, simplifying requirements, moving complexity and splitting components.

Reduce Waste

A good agile team should already be pretty good at reducing their waste. Retrospectives that generate actions which are then implemented go a long way towards this. A good Scrum Master who removes impediments is vital. The team will be provided with the best tools for the job.

Still, there is likely to be further waste that can be eliminated to allow the team to go faster. The first set comes from the category of unnecessary process waste. This includes things like:

• Having to attend full meetings where there is very little need for members from the team to be present.
• Long-winded design sessions that could be cut down with a bit more decisiveness.
• Unnecessarily revisiting architecture and design decisions over and over again.
• Re-enacting the same debate repeatedly because consensus wasn’t achieved.

The other set comes from giving the bottlenecked team stories to play that aren’t essential to what they are trying to deliver. This can be things like:

• Helping ops with tracking down a problem
• Adding additional logging of messages to an external system
• Trying to reproduce an infrequent defect
• Updating a build pipeline

In an ideal world, all of the above should be done by the team. However, if that team has a high volume of work to get through then these tasks can easily be divested out to teams that are perhaps not running at full capacity, allowing the team that is to maximise their throughput of important functional stories.

Simplify Requirements

Often performance of teams can be greatly hampered by having requirements that are either too complex or not clearly thought out. Ensure the backlog is groomed to contain only the most essential stories. Make sure these have been paired down to the absolute minimum set of requirements needed to fulfill the business needs.

The worst scenario is having a heavily loaded team working on features that are overly complex and time consuming. I’ve seen many cases where teams have worked on a fairly complex story only to later find that the clients uses just the core 80% of the feature, which could have been built in 20% of the time. The extra 80% of the time spent delivering that last 20% of non-essential requirement is just waste.

Analysts and the business need to scrutinise the backlog, squeezing every unnecessary piece of work from the stories. If the business doesn’t need some small aspect right now then create a separate story and play it later when the team has less critical path work.

Move complexity

In most large software systems it is possible to locate complexity in different components. For example table paging could be implemented in the server code or in javascript on the browser. If complexity in one place is causing capacity problems then it may be possible to move elements of this complexity to another team that has more spare capacity.

This may not always be the most technically pure solution, but it may be the best trade-off in order to deliver a product on time and budget. The complexity can always be relocated in the future.

As an example, consider a story where one component has 8 points of work do to delivery a feature and it’s two clients each have 2 points of work to consume the feature. Total 12 points. However, it might be possible to split the work differently, so that the core component implements something much simpler for 3 points, while the two clients take on some complexity and implement 5 points each. Total 13 points. So, the two approaches are roughly equivalent in size. The second might result in some duplication on behalf of two teams, but this allows the team with a large backlog to complete more stories. A pragmatic trade-off.

Split Components

If a team has multiple components, then the logical option is to relocate one or more of those components to another team. What if they only have one component?

Ideally, buy reducing waste, simplifying requirements, and moving complexity it should be possible to shrink the size of the backlog for a component. Hopefully the smaller backlog should be sufficiently sized that the team can complete it. What if they still can’t?

To me this seems to be a problem of decomposition. A component that is so large and complex that a team of six devs can’t maintain and enhance it is a component that is too big. In this case it should be split into two or more smaller components, with the current team keeping one part and a different team taking the other part(s).

Making the split is always technically challenging, but provided it is made across a clear interface boundary then my experience is that it is usually possible divide a component up into smaller constituent parts.

This is probably the only way that I would recommend going about adding more people to a project - introduce a new team (or repurpose an existing team) and give them their own code base and component(s) to work on.

Learnings From A Scala Project

I’ve recently been working as an associate for Equal Experts. I’ve been part of a successful project to build a digital marketing back-end solution. The project started out life as a JVM project with no particular language choice. We started out with Java and some Groovy. However, the nature of the domain was one that leant itself to functional transformations of data so we started pulling in some Scala code. In the end we were fully implementing in Scala. This post describes the learning gained on this project.

The Project

The project team consisted of five experienced, full-time developers and one part-time developer/scrum master. One of the developers (me) was proficient in Scala development (18 months, part-time) while the others were all new to the language.

The initial project configuration (sprint zero) was based on a previous Java/Groovy project undertaken by Equal Experts. The initial tooling, build and library stack was taken directly from this project with some examples moved from Groovy to Scala. The initial project stack included:

• Gradle (groovy based build system) with the Scala plugin bolted in
• IntelliJ IDEA with the Scala plugin
• Jersey for RESTful web service support
• Jackson for Json/Domain Object mapping
• Spring Framework
• MongoDB with the Java driver

Tooling

The biggest hurdle faced by the project was tooling, specifically tool support for the Scala language. While Scala tooling has come on a huge amount over the last year, it is still far from perfect. The project faced two specific challenges: build tool support and IDE support.

Build Tool Support

The Gradle build tool is primarily a build tool for Java and Groovy. It supports Scala via a plugin. Similar plugins exist for other build tools like Maven and Buildr.

What we experienced on this project was that this was not the ideal scenario for building Scala projects. Build times for loading the Scala compiler were rather slow and support for incremental compilation was not always as good as we would have liked. More often than not we just had to undertake a clean and build each time. This really slowed down test-driven development.

Currently (in my opinion) the only really workable build solution for Scala is the Simple Build Tool, also known as SBT (https://github.com/harrah/xsbt/wiki). This is a Scala specific build tool and has great support for fast and incremental Scala compilation plus incremental testing.

With hindsight it would probably have been better to switch over from Gradle to SBT as soon as the decision was made to use Scala for the primary implementation language.

IDE Support

The Scala plugin for IntellJ IDEA is one of the best editors for Scala. In particular its code analysis and highlighting has come on in leaps and bounds over the last few months. However, the support for compilation and running tests within the IDE is still less than ideal. Even on core i7 laptops we were experiencing long build times due to slow compiler startup. Using the Fast Scala Compiler (FSC) made a difference, but we found it somewhat unstable.

What was more annoying was that after failed compilations in IDEA, we often found that incremental compilation would no longer work and that a full rebuild of the project was required. This was VERY slow and really hampered development.

One proven way of doing Scala development within IDEA is to use the SBT plugin and run the compilation and tests within SBT inside an IDEA window. This has proven very successful on other Scala projects I have been involved with and again with hindsight we should have investigated this further.

Some developers on the team also found the lack of refactoring options, in IDEA, when working with Scala somewhat limiting. This is a major challenge for the IDE vendors given the nature of the language and its ability to nest function and method definitions and support implicits. Personally this doesn’t bother me quite as much as my early days were spent writing C and C++ code in vi!

Libraries

Scala has excellent integration with the JVM and can make good use of the extensive range of Java libraries and frameworks. However, what we experienced in a number of cases was that integrating with Java libraries required us to implement more code than we would have liked to bridge between the Java and Scala world. Also, much of this code was less than idiomatic Scala.

What we found as the project progressed was that it became increasingly useful to switch out the Java libraries and replace them with Scala written alternatives. For example, very early we stopped using the MongoDB Java driver and introduced the Scala driver, called Casbah. Then we added the Salat library for mapping Scala case classes to and from MongoDB. This greatly simplified our persistence layer and allowed us to use much less and more idiomatic Scala code.

Aside: If you are working with MongoDB then I strongly recommend Scala. The excellent work by Brendan McAdams of 10gen and the team at Novus has created a set of Scala libraries for MongoDB that I think are unrivaled in any other language. The power of these libraries, while maintaining simplicity and ease of use, is amazing.

We never got a chance to swap out some of the other major libraries that the original project structure was built on. However with more hindsight we should perhaps have made the effort to make these swaps as they would have resulted in much less and much cleaner code. Some specific swaps that we could have done:

• Jersey for Unfiltered or Bowler
• Jackson for sjson or lift-json
• Spring for just plain Scala code

Scala projects work well with Java libraries. However when a project is being fully implemented in Scala then it’s much better to swap these Java libraries for their Scala implemented alternatives. This results in much less and more idiomatic Scala code being required.

Team Members

Having highly experienced team members makes a big difference on any project and this one was no exception. The other big advantage on this project was that all of the developers were skilled in multiple languages, even though most had not known Scala before. One thing I have found over many years is that developers who know multiple languages can learn new ones much more easily than those who only know one. I’d certainly recommend recruiting polygot developers for new Scala projects.

It also helped having at least one experienced Scala developer on the team. The knowledge of the Scala eco-system was helpful as was the ability to more quickly solve any challenging Scala type errors. However, the main benefit was knowledge transfer. All of the team members learned the Scala language, libraries and techniques much more quickly by pairing with someone experienced in the language.

Additionally, as the experienced Scala developer I found it very valuable working with good developers who were still learning the language. I found that explaining concepts really clarified my understanding of the language and how to pass this on to others. This will be of great benefit for future projects.

Any new project that is planning on using Scala should always bring on at least one developer who already has experience in the language. It creates a better product and helps the rest of the team become proficient in the language much more quickly.

Object-Oriented/Functional Mix

We found Scala’s ability to mix both object-oriented and functional approaches very valuable. All team members came from an object-oriented background and some also had experience with functional approaches. Initially development started mainly using object-oriented techniques, but leaning towards immutable data wherever possible.

Functional approaches were then introduced gradually where this was really obvious, such as implementing transformations across diverse data sets using map and filter functions. As the team got more familiar with the functional programming techniques more functionality was refactored into functional style code. We found this made the code smaller, easier to test and more reliable.

Perhaps the only real disadvantage of moving to more functional leaning Scala code is that some code became less readable for external developer with less exposure to the functional style. It became necessary to explain some of the approaches taken to external developers: something that would probably have not been necessary with more procedural style code. This is worth bearing in mind for projects were there will be a handover to a team not skilled in Scala and functional programming techniques.

Conclusions

The project was very successful. We delivered on time and more features than the customer originally expected. I don’t think we would have achieved this if the implementation was in Java, so from that point the smaller, more functional code base produced by developing in Scala was a real winner. Using Scala with MongoDB was a major benefit. Tools was the real let-down on the project. Some problems could be alleviated by moving from Java libraries and solutions to their Scala equivalents. However, there is still some way to go before Scala tools (expecially IDEs) match the speed an power of their Java equivalents.

Technical Investment vs Technical Debt

A subject that has been bothering me for a while is the concept of Technical Debt. It’s an interesting analogy, but I think it has become increasingly misused and misunderstood in the agile world. In this post I want to focus on the real meaning of Technical Debt and describe an alternative approach that worked on a current project.

What is Technical Debt?

In its simplest form, Technical Debt is work that you really should be doing but which you put off until later in order to meet some short-term delivery goal - it’s a compromise! For example, I might know that my JSON to Domain Object mapping approach is sub-optimal bit I decide to live with that for a while in order to complete a set of urgent features required for an important customer demo.

The key part of Technical Debt is that you MUST pay it back at some point in the future; and the sooner you do the better. To continue my example: as soon as the customer demo is complete, the fixing of the mapping approach should be prioritised as an urgent task.

How is it Misused?

One of the main problems I have encountered on many projects is that they misuse Technical Debt. The two most common misuses are:

• Being far too willing to accept Technical Debt; and
• Allowing Technical Debt to exist for too long (i.e. not paying it back quick enough)

Both of the above burden a project and, in my experience, they are allowed to happen because people don’t understand the true implications of Technical Debt correctly. There seems to be a general miscomprehension that Technical Debt can be captured as a story in the backlog that can be played at some time in the future much like any other story.

What I believe that most projects miss is the fact that having a Technical Debt story in the backlog not only requires that the debt is paid back in the future, but that its mere presence incurs an ongoing interest payment on the project. This interest payment comes in many different forms, some of which include:

• increased difficulty understanding the code base
• unnecessary complexity in the code
• more difficult and time consuming refactoring
• needing to write more code than necessary to add new features
• slower build and test times

All of the above (and other forms) result in one major impact: reduced team velocity. So, any Technical Debt, besides being a story on the backlog that has to be played in the future, also reduces velocity. Too much of it and velocity grinds to a halt.

Cumulative Impact

Projects that tend to allow Technical Debt to be readily created and to persist also tend to allow multiple pieces of Technical Debt to be present at the same time. This has a cumulative effect in that features are built on one piece of debt and then new debt grows on these features and so on. After a while the ongoing interest payments increase exponentially and the effort required to repay each piece of Technical Debt increases significantly.

At this point a project tends to come to a halt completely because the interest payments of having the Technical Debts stop the project adding any new features, while the effort required to pay back the debt has become too huge. The project either fails or needs a reboot.

Is There An Alternative?

From time to time, any project must accept some compromise in order to meet some critical goal. I see no problem with this. However, what projects need to do is to ensure that these compromises are kept as small as possible and have the smallest possible payback cost. So, how do we ensure this? The solution I would propose is what I have coined as ‘Technical Investment’.

Using the debt/investment analogy for personal finances, we can see two approaches: I can borrow a load of money to buy what I need now and then have to deal with interest payments until I can pay the debt back; or, I can put some money away into an investment so that when I need to buy something I can do so without having to go into debt. I temporarily reduce the size of my investment to make the purchase but then I quickly replenish it and grow it again ready for another future purchase. Hopefully my investment also grows over time so I can buy more things or reduce my investment portfolio by less each time.

So, how does this Technical Investment work in agile software development? What we do is we dedicate a small portion of the team during each iteration to refactoring, cleaning, rewriting, upgrading libraries and so on. This may be in any area of the code, not just that directly touched by the current stories in play. We invest in the code base a small amount each iteration. Perhaps we might deliver one small story or feature less each sprint, but as the project progresses we avoid the large reductions of velocity that are inherent in a Technical Debt driven model.

So, Does It Work?

On a recent project we adopted the above approach (although somewhat informally, which is why I wanted to document it more formally in this post). We had a team of five developers working on weekly sprints. Each day we created two pairs to work on features, while the fifth team member was left free to invest in the code base. The lone person changed each day or so. What did we find?

Firstly we found that the need to incur Technical Debt (or draw down from our investment, to follow the analogy) was significantly reduced. Quite often the areas where we would have had to make a compromise to meet a sprint goal were already addressed and improved before making the compromise became necessary. We were able to preempt many areas of improvement before they became an issue.

Secondly, we found that in cases where a compromise was made (where we drew down from our investment) that we were able to quickly fix the compromise and get back to adding to our investment much sooner. We also avoided the problem of any cumulative compromises, because payback was smaller, which made correcting individual cases much simpler when we decided to accept them.

By working in this way we were able to have a continually improving code base, while still maintaining a high velocity of new features. The code base continually improved in quality and clarity throughout the project even though features were continually being added.

But Shouldn’t This Happen Anyway?

This is an interesting question. Yes, a good agile team should be continually refactoring and improving code as part of every story. However in practice, the pressure to complete features often reduces the amount of improvement that is done. It’s also often difficult to justify changes to code in an area that no-one is currently working on just because it needs some love: which may benefit some unknown work in the future.

Having a concept called ‘Technical Investment’ also gives a concrete thing that can be talked about with Product Owners. We can discuss the investments made during reviews and present them in a positive way, something that I have found much more difficult to do with debt.

I suspect that it may also be possible to measure and visualise the investment in a similar way to the measuring of velocity. This is certainly something I’d like to explore in the future as being able to show how investing in the code reduces future reductions in velocity is a great way to justify the practice.

Conclusion

Keeping the code base clean and building up good quality through continual Technical Investment on each iteration has many advantages over the use of Technical Debt to capture and deal with compromises. When applied, the need to compromise becomes less and the effort involved in fixing each compromise is significantly reduced. The chances of building up cumulative, velocity zapping compromises also lessens significantly. If you have a longer-running or larger project then continual Technical Investment is the only sure way to make sure the project doesn’t grind to a halt due to debt interest as some time in the future.

The Power of Pairs

In my current contract we have a policy of full-time pair programming. It has been working incredibly well. I thought I'd share some of my observations as to why this might be the case.

Previous experience of pairing

I've used and worked with pairs in previous roles with various degrees of success. However, none of those previous roles adopted the full-time pairing model. I've always seen some benefit in pairing, especially on complex or critical pieces of code: two minds working together on these sorts of problems is invaluable. My previous roles have, however, always gone back to singletons for the more mundane and less critical coding tasks.

The one thing I have never seen previously is any of the purported productivity boosts from pair working. This could be because I've only used pairs on for complex tasks, which would take a long time anyway. But surely I should have seen some benefit in the past? Why didn't I?

Our pairing model

The team that I am working with at the moment is made up of five developers. Each of us comes with experience of many successfully completed projects. We all have different language and domain backgrounds, but there seems to be a common thirst from every member of the team for learning and improving. I think this is perhaps one of the key reasons why our pairing model works so well.

Each day we swap the pairs. Sometimes if we are deep in a task we might keep the same pair for two days, but never longer. The dynamic of the team is therefore always changing. It also encourages up to keep tasks small so that we can generally complete them in a single pairing session.

Hold on, don't we have an odd number of people? Yes, and this I think works very well. Each day we pair up and one person gets to work on their own for the day. We try to ensure that each person gets to be the singleton regularly. This single person picks up simple technical debt tasks, carries out investigative spike or just refactors and tidies up the code base. They are requested not to pick up any complex development. Working in pairs is incredibly demanding of concentration and being the singleton for a day also helps keep people fresh.

The benefits

So, why does this pairing model work so well, and why have we created a team that appears to be incredibly productive? Here's my thoughts…

The right team members - All of the developers on the team are good developers and more importantly they are open and keen to learn and improve. We have a number of different language and platform backgrounds on the team, but rather than being a problem it is just more useful information to share between us. The team members have created an environment of sharing and constant learning.

Willingness to question and compromise - Every member of the team has the ability and confidence to question the approach being taken by the other developer in their pair. While I think experience helps in this, you really want even junior team members to have this confidence to question. Linked to this is the fact that each team member is open to new ideas and is either prepared change their thinking or, even if they disagree, come to some form of compromise to allow the pair to continue moving forward. Developers who argue, fail to question or who are not prepared to learn and adapt will kill a pair's productivity.

Shared knowledge - By rotating the pairs on a daily basis, knowledge of the application, language and development approaches is spread throughout the team. This just wouldn't happen if pairs were fixed or people working alone.

The right tools and environment - The team have created a development and build environment that they are comfortable with and that is productive. The team are empowered to make any changes that allow them to work better. These could be changes to the build process or introduction of a new library the cuts the amount of or simplifies code.

A focus on quality - All the members of the team have the same view of the importance of quality and testing. There us a mutual agreement of what constitutes good quality and all team members strive for this goal. Teams where there are differing notions of quality between individual members just don't work as well - even in a non-pairing environment.

The ability to technically invest - By having one team member constantly assigned to cleaning up technical debt, refactoring and improving the code base it allows the pairs to spend all of their time investing in new features. The code base is kept clean and this makes it much easier to introduce new approaches or libraries that further improve the code.

Conclusion

So, would I adopt full-time pair programming again in the future? Definitely! However, to get the real benefit you need the right team of people and you need to create the right environment and culture for them to work in and give them the right set of tools. I'd also try to always have an odd number of developers so that pairs could be rotated and one person be free to clear down any technical debt so that the pairs can invest.

Selecting The Correct Test Scenarios and Data

Yesterday I happened to be reviewing some integration tests with my excellent tester colleague. These tests were defined using a set of highly simplified synthetic data selected for the purpose of making the tests pass. We both agreed that tests should actually be written using a mixture of realistic and synthetic data selected for the purpose of trying to make the tests fail! As developers we are often too keen to write tests that prove that our code is good rather than writing tests to find the problems in our code.

This exercise bought to mind a situation I had a few years ago with some code provided by an offshore development team. It was a very simple piece of code and came with some tests to show that it worked. However, it exhibited a single fundamental flaw that the synthetic data designed to make the tests pass would never have found. I've recalled the problem below (somewhat simplified), and I've update the code from Java/JUnit to Scala and Cucumber (because quite frankly I prefer these languages!).

The Requirement

The specific requirement for the system being built was that it would collect a user's date of birth during the sign-in process. If the user was under 17 then they would not be allowed to use the system. If they were 17 or over then they would be taken to the main menu. The particular piece of code in question was the age validation algorithm.

The Supplied Solution

The initial solution to this problem consisted of the following piece of code:

class AgeValidation {
  private val AuthorisedAge = 17

  def isAuthorisedAge(user: User) = {
    val cal = Calendar.getInstance
    val currentYear = cal.get(Calendar.YEAR);

    cal.setTime(user.dateOfBirth)
    val birthYear = cal.get(Calendar.YEAR);

    (currentYear - birthYear) >= AuthorisedAge
  }
}

Tests were supplied that covered two specific scenarios:

Feature: The application should ensure only users over 17 are allowed access
    As a Application Security
    I want only users aged over 17 to access the system

    Scenario: User aged over 17 can access the system
        Given a user aged over 17
        When the age check is applied
        Then they are authorised to continue

    Scenario: User aged under 17 cannot access the system
        Given a user aged under 17
        When the age check is applied
        Then they are not authorised to continue

And the implementation of these tests was as follows:

class AgeValidationSteps extends ScalaDsl with EN with ShouldMatchers {
  private var user: User = _
  private var valid: Boolean = _
  private val dateFormatter = new SimpleDateFormat("dd/MM/yyyy");

  Given("""^a user aged over 17$""") {
    user = new User("test", dateFormatter.parse("01/01/1970"))
  }

  Given("""^a user aged under 17$""") {
    user = new User("test", dateFormatter.parse("01/01/2010"))
  }

  When("""^the age check is applied$""") {
    val validator = new AgeValidation
    valid = validator.isAuthorisedAge(user)
  }

  Then("""^they are authorised to continue$""") {
    valid should be (true)
  }

  Then("""^they are not authorised to continue$""") {
    valid should be (false)
  }
}

Extra credit if you can spot the problem with this implementation and why the synthetic test data would have never found it before you read on!

The Problem With This Code

Did you spot it? Here's a hint: what if my 17th birthday is not until tomorrow? Yes, that's right, the code has a fundamental defect in that anyone who is 17 in the current year but who has not yet had their birthday is granted access to the system even though they are still 16. And yes, this was the actual algorithm submitted to us by the offshore developers!

Quite clearly, the synthetic test data used in the supplied tests was selected to ensure the tests passed, but not to find the problems in the code. Let's correct the feature spec to base it on values that actually might find something:

Feature: The application should ensure only users over 17 are allowed access
    As a Application Security
    I want only users aged over 17 to access the system

    Scenario: User who was 17 last year can access the system
        Given a user whose 17th birthday was last year
        When the age check is applied
        Then they are authorised to continue

    Scenario: User who is 17 next year cannot access the system
        Given a user whose 17th birthday is next year
        When the age check is applied
        Then they are not authorised to continue

    Scenario: User who was 17 yesterday can access the system
        Given a user whose 17th birthday was yesterday
        When the age check is applied
        Then they are authorised to continue

    Scenario: User who is 17 today can access the system
        Given a user whose 17th birthday is today
        When the age check is applied
        Then they are authorised to continue

    Scenario: User who is 17 tomorrow cannot access the system
        Given a user whose 17th birthday is tomorrow
        When the age check is applied
        Then they are not authorised to continue

And the associated test steps:

class AgeValidationSteps extends ScalaDsl with EN with ShouldMatchers {

  private var user: User = _
  private var valid: Boolean = _
  private val dateFormatter = new SimpleDateFormat("dd/MM/yyyy");

  Given("""^a user whose 17th birthday was last year$""") {
    val cal = Calendar.getInstance
    cal.add(Calendar.YEAR, -18)
    user = new User("test", cal.getTime)
  }

  Given("""^a user whose 17th birthday is next year$""") {
    val cal = Calendar.getInstance
    cal.add(Calendar.YEAR, -16)
    user = new User("test", cal.getTime)
  }

  Given("""^a user whose 17th birthday was yesterday$""") {
    val cal = Calendar.getInstance
    cal.add(Calendar.YEAR, -17)
    cal.add(Calendar.DATE, -1)
    user = new User("test", cal.getTime)
  }

  Given("""^a user whose 17th birthday is today$""") {
    val cal = Calendar.getInstance
    cal.add(Calendar.YEAR, -17)
    user = new User("test", cal.getTime)
  }

  Given("""^a user whose 17th birthday is tomorrow$""") {
    val cal = Calendar.getInstance
    cal.add(Calendar.YEAR, -17)
    cal.add(Calendar.DATE, 1)
    user = new User("test", cal.getTime)
  }

  When("""^the age check is applied$""") {
    val validator = new AgeValidation
    valid = validator.isAuthorisedAge(user)
  }

  Then("""^they are authorised to continue$""") {
    valid should be (true)
  }

  Then("""^they are not authorised to continue$""") {
    valid should be (false)
  }
}

Run this set of tests against the existing code and the final scenario 'User who is 17 tomorrow cannot access the system' fails. Our tests have actually done something useful: they have proved that our code doesn't work. Test data selected with the specific goal of trying to find problems with the code has done its job. I'll leave it as an exercise for the reader to fix the implementation so that it all works correctly!

Further Improvements

Even with our improved set of test data there are still some significant improvements that can be done to these tests by selecting some additional scenarios and test data values that might find problems:

• Testing the algorithm works with data boundary cases such as first/last day of the year, leap years and so on
• What happens when the user enters a date of birth in the future, does the code still work?
• What about if the person is over 100? Are we dealing with 4 digit or 2 digit years?
• Is this an international system? If so, then what about timezones? A person in a timezone east of us may already by 17, but may not be 17 in our timezone.

Apply The Same Rules To Our Tests

We can also apply the same thinking to our tests. For example, if we run our improved tests against the original code but do so on the first day of the year then they will all pass. The reason being that subtracting one day from the current date would push the birth date into the previous year. We therefore need to think not only about how our synthetic test data might find problems in our code by also how they may cause problems in our tests. In this case, solving this problem is just a case of improving our code to allow our tests to define the current date rather than using whatever the current system date is:

class AgeValidation(baseDate: () => Date) {
  ...
}
object AgeValidation {
  def apply = new AgeValidation(() => new Date)
  def apply(baseDate: String) = new AgeValidation(() => new SimpleDateFormat("dd/MM/yyyy").parse(baseDate))
}

// And in the tests
val validator = AgeValidation("15/07/2010")

Our test code now has the ability to control exactly what current date is used as a basis for the comparison, so it can experiment with a wider range of possible scenarios.

Does This Change Our TDD Approach

All of the above got me thinking about how this impacts on the Test-Driven Development approach. In this model we first write a test to validate the expected behaviour of our code. We then write minimal code so that the test fails. Next we write the code to make the test pass. Finally we refactor to make the code better without breaking the tests. Most developers will approach this by picking some synthetic values that are designed to prove that the 'happy path' through the code works when they have written it. Perhaps we should be focusing more on TDD where the tests we start out writing are for the scenarios that are more likely to identify problems - that way the code we write to make these pass has to deal with these cases. Certainly something I will be experimenting with in my next TDD session.

My New Appreciation of Testing

Recently I have had the benefit of working with a really good tester on an agile project. This person has consistently challenged both my terminology and approach to testing. After working with him I have come to a completely new conceptual model of automated testing (checking) vs manual (and exploratory) testing, how they are very different and how they complement each other. This is the story of what I have learnt.

A Developer Model of Testing

I've been writing automated tests as part of my development process for as long as I can remember. These used to be tests written after the code to check that it worked, then I started doing TDD and using the tests to drive the design while validating that it also worked as expected. Recently I've also added BDD into the mix as a way of driving and testing requirements. However, all this time my mental model of testing has been somewhat fixed and, as I now realise, focused solely from a development perspective. Let me try to describe this view.

As a developer, my goal is simple, write some code such that:

• given a know starting state and;
• applying a known sequence of operations;
• with a defined set of input values;
• then I get an expected and valid output state

My typical approach would be to write tests that:

1. Set up some fixture code representing a state that I want to start from
2. Define a test that applies an operation with valid input values
3. Define some assertions that should be true about the state after the operation has completed
4. Implement and refactor the code until the assertions pass

I'd then go on to create some additional variations of the above test with alternative applications of the operation, adjusting the implementation code so that each test's assertions pass. Typically the alternative tests would cover a mix of:

• Apply an operation with different valid input values
• Apply an operation with invalid input values
• Apply an operation with boundary input values
• Apply an operation with non-supplied input values
• Apply an operation with potentially harmful input values
• Apply an operation forcing different failure cases in underlying code, subsystems or boundaries
• Apply operations involving concurrency or synchronisation

As a conscientious developer I'd be trying to write automated tests that cover the widest range of scenarios so that I could be confident my code would work as expected in all situations. When describing my progress in a stand-up I'd be using phrases such as:

• "The code is done and is working because all the tests pass"
• "The tests show we've fixed all the defects"
• "We've made the change and the tests show that we haven't broken anything"

From a developer point of view the functionality would then be handed over to a QA team member to check and perhaps for them to write some higher level test against to ensure the code meets the functional requirement (i.e. given a good set of data and a good set of inputs then the correct result is observed). Code complete, story done, into production, next please!

The Awkward Tester

The above was pretty much my approach to testing and is pretty typical of that taken by most developers I think. However, when working with this excellent tester on our team he started to question the statements that we made about our code. For example:

• The code is done and working because all the tests pass: "Your tests show that your code makes the tests pass, but they don't prove that it works"
• The tests show we've fixed all the defects: "The tests actually just show that you've fixed the specific identified defect case(s), all the other unknown ones are still there"
• We've made the change and the tests show that we haven't broken anything: "No, the tests show that you haven't broken the paths covered by the tests, there's probably other things that are broken though"

My initial reaction was to just laugh this off, put the tester down as an awkward, pedantic sod and carry on as before. However, based on his well deserved reputation and previous work record, I trusted and respected the skills of this particular person. This sowed some seeds of doubt in my mind: was it just the language us developers were using that he had problems with, or was he really talking about some fundamental wider view of testing that we, as developers, were all missing?

My New Developer Model of Testing

After some serious contemplation, introspection (and a number of very long lunchtime walks) I have come to realise that what I have been considering as 'comprehensive' automated testing is really just a small part of of a much wider testing world. By focusing so much on these automated test suites I had become blind to the wider implications of my code quality. Our excellent tester friend calls what we do automated 'checks', not tests, and I think he might be right! What we are building are pieces of non-deployed code that check that our real code does what we are expecting it to do in a certain set of scenarios. Testing is so much more. Let me explain my current thinking...

In automated checks we first set up fixtures that represents known starting states for our application, component or class. For all but the most trivial examples these states will only be a small subset of all of the possible states that my application could be in. My checks then pick an operation or set of operations (i.e. feature) to evaluate against this state. Again, for anything but the most trivial single function cases the operations we pick and the order they are called will represent just a small subset of all possible operations and combinations of those operations.

Next we look at input values and their combinations and the same subset rules apply. This is also true for failure scenarios that we might check. Finally, we assert that our code leaves the application in a consistent state, with those states again being just a small subset of the possible states my application could be in. We then clear down our test fixtures and move onto the next test. This can be clearly shown in the following diagram:

Now, our tester has a different view. He is trying to identify possible places where my application might break or exhibit a defect. He is therefore using his skills, knowledge and experience to pick combinations of specific states, operations, input values and failure scenarios that are both likely to occur and most likely to find the defects in the application. These may be inside the subsets used by the automated checks, but can also be outside, from the wider set of possibilities.

Additionally, the tester has another weapon in their arsenal in that they can use the output state of one set of tests as the input state into the next - something that it's not so easy to do in automated tests that might be run in a different order or individually. We can see the tester's approach to testing in the following diagram:

It is clear to me now that the two approaches are both necessary and complementary. Checking has far less value if testing is not also applied. Testing would also be far more expansive and time consuming if checking was not in place to find and eliminate a certain percentage of defects at development time.

The Role of the Tester

So, what is the role of the tester? Most developers would tend to assume that the tester's job is to prove that their code works (and I will even admit that this was probably the way I used to think). Go on admit it developers, how many times have we cursed the tester under our breath when they have found a critical flaw in out code a few days/hours before it's due to be released? Come on, we know we've all done it!

Turns out we were wrong! The role of a tester is to prove that our code doesn't work and to prove it before the code gets into production and is broken by a real user or customer. Why should a tester be trying to prove that my code works? Isn't that why I've invested so much time and effort into building my automated suite of checks - to give me confidence that my code works under the situations that I have identified. What I really want to to find out is where my code doesn't work so that I can identify the omissions from my suite of checks and fix the broken code paths. This is what a good tester should be doing for me.

First and foremost a good tester should be using his skills to find scenarios whereby my code fails and leaves the application in a broken and inconsistent state. This should be the Holy Grail for a tester and they should be rewarded for finding these problems. Who wants a production outage caused by a defect in their code? Not me, that's for sure. Next they should be finding cases where the application fails non-cleanly but is not broken and can continue to function. If they can do either of these things then they are worth their weight in salary/day rate! A tester that can only find the places where the code fails cleanly or does not fail at all is really not of much value: my automated checks already proved this! I know I will certainly be valuing my testers much more than I used to. And you should too.

So how as developers do we aid out able testers in finding the defects that are in our code? We have to help them understand how the bits of the system fit together; we have to explain the approaches we have used to determine our automated checking scenarios; we have to help them understand where the failure points might be and; we have make sure they know the expected usage patterns that we have coded to. Given all this information or excellent tester should be able to find the gaps, the failure scenarios and the unexpected usages that expose the defects that our automated checks do not. When they find them we should be glad, and keen to find out how they managed to get our lovingly created code to break. We then fix the problem, add checks for those paths that we missed and learn the lessons so that we don't repeat the same defect creating pattern in any future code we write.

The Stand-up Revisited

So, given this new understanding of testing, testers and automated checks, what should the progress descriptions in the stand-up really be:

• "The code implementation is complete and my checks for the main scenarios all pass. Mr Tester, can you find the defects please?"
• "I've fixed the specific path(s) which I identified as being a cause of the reported defect and added automated check. We need to test to find the other paths that might be causing the same problem."
• "The existing checks all run against the new code. We need to test to see which other paths or aspects of features have been broken by these changes."

See the difference? No assumptions that just because an automated suite of checks has completed successfully that the code either works, is free of defects or that its addition has not broken the application somewhere. The automated checks can reduce the probability of bugs or issues escaping the development process and can give me more confidence in my code. However, some defects and problems are still likely to be present and my application will still break under certain conditions that my checks don't cover. I now say: "...over to you Mr. Tester I'll explain what I've done and you can help me find where it is broken!".

Sprinting with BDD

After some initial experiments into doing BDD using Cucumber our team has decided to adopt a full BDD and TDD approach to our current sprint. It's slightly experimental at the moment as we work out the best ways of working. Once we are done we will retrospect and see if we want to adopt the approach going forward. Initial feelings from our first couple of days are that it is going well. Here's what we have done so far.....

Sprint Planning

After deciding to give BDD and TDD a try for the sprint, the first thing was the sprint planning. Fortunately for this sprint we have a story that is pretty well understood, but sufficiently challenging to be of value in evaluating the approach. The story basically consists of taking some existing data, retrieving additional data from a new remote system and delivering the combination of the two to a remote destination (basic decoration of existing content).

Our sprint planning consisted of two parts:

1. a high level technical discussion of the story and agreement of the general architectural structure and approach
2. the writing of Cucumber 'Feature' files containing all of the acceptance scenarios for the story

These two parts went well and we created a good set of acceptance scenarios, but we then found ourself struggling to get our heads around how to go about implementing them. Our reasoning was that rather than writing out individual technical tasks, we would just create one card for each scenario. These scenario cards would then form our sprint backlog.

However, the first scenario was a 'happy path' one and we were thinking that in order to implement it we would have to build the entire solution; the remaining scenarios would then just be tiny additions. This didn't feel right in that we would then have one scenario stuck as 'in progress' for days and then all of them moving within a few hours right at the end of the sprint. Our next thought was to therefore break down this scenario into a number of technical tasks. As it turns out, this wasn't the correct approach and it soon became clear that by doing this we were missing the benefits of BDD and TDD: we were defining the detailed solution in the planning session, rather than letting it emerge from the implementation of the tests.

We decided to take a break and think over our approach. It was then that it came to us that we were still thinking in a non-TDD mindset. To pass the first scenario, we didn't need to implement the whole solution, just enough to make the BDD test pass. As we then moved on to the next scenario we could implement more of the functionality, add depth and so on. With this in mind we decided to start the sprint and inspect and adapt as we went.

Setup and Configuration

The first step in the sprint was to add Cucumber support to our project. Given that we are using BDD for acceptance tests, we decided to add this support into our functional test suite. Unit level TDD will continue to be done using our existing testing tools (JUnit, Hamcrest and mockito).

Our functional tests are written in Java and are built and executed using Maven. We therefore added a dependency to cuke4duke, which runs Cucumber on top of the JRuby platform. This setup included adding a dependency on the maven-cuke4duke-plugin, which runs the Cucumber 'Features' within a maven test phase.

Aside: Why did we pick Cucumber (which is a Ruby framework) when we are doing a Java project? There were a number of reasons, but the main ones were:

• We liked the HTML report format
• A number of the team are already using Cucumber on external projects (in Ruby and Scala) so there was an existing familiarity
• It seems to keep out of the way and be less invasive than many of the Java based alternatives

With this set up, we added the Features file that we built during spring planning into the appropriate features directory and created step definitions with pending methods for each of the Given/When/Then cases.

The First Scenario

The first scenario was to set up some existing content and some additional content, decorate the existing content with the additional content and then verify that the content delivered at the destination had been correctly decorated. The steps for Cucumber were written accordingly and the assertions added the 'Then' case. Now it was time to implement.

This was where our TDD thinking really kicked in. Previously we would have started building the whole solution. Instead we just implemented the elements of the destination required to allow us to verify the decorated content. Given that we were only testing with one piece of content we just hardcoded to return a fixed set of additional data. Test passed.

Now, we know that this is not the full solution and that we have plenty more work to do, but we have a green on our test report. We have 8 scenarios to complete and the story is not done until they all pass, so what's wrong with this approach? Our next scenario tests with a different set of content and additional data, so this is broken by the hardcoding approach, so we can then work on the next step of delivering different sets of additional data to the destination. In this case we will still probably hardcode the data that is delivered based on the content, but progress will be made. Later scenarios will then add the calls to the new remote system to get actual data. By the end of the scenarios we will have built the entire solution driven solely by tests.

Areas of Uncertainty

Even though things are going well, we still have a number of areas of uncertainty that we need to resolve by then end of the sprint. These include:

• Whether we can successfully complete the story in this way
• Whether this approach impacts our productivity
• Whether the team members are able to adapt to working in this TDD style
• Whether the business will identify with the inherent value of this approach
• Whether we can get the BAs and the POs to start thinking of the stories as BDD scenarios

We hope to resolve these as the sprint progresses. I do think that initially productivity will be reduced due to the time it takes to learn a new approach and to become proficient in it. However, while productivity may be reduced for a short time, I believe that quality will go up due to the test-driven nature of the work and that we will be more certain of building the correct solution. I it my hope that the team members will see the value of this approach and will opt to continue with it for a few more sprints to allow them all time to become proficient in the approach. We will then be able to evaluate its success or failure more objectively.

To help the business understand the value of this approach (assuming of course that it is successful) I will plan to include a quick overview of the BDD approach in the Sprint Review next week. I will give a short presentation on the approach we took, show the scenarios that we created during planning, talk through the approach we took to implement them and show the Cucumber report that is produced. It will be interesting to see the reactions.

I'll post a follow-up at the end of the sprint to let you know how we got on.

The Best Way to Apply BDD

I am currently in the process of rewriting a simple e-commerce application for one of my customers. The new version is implemented using Scala and the excellent Lift framework and is a rewrite of a six year old struts/JSP version. The core application is largely identical but I am adding a wide range of new and improved administration functions.

Typically I build a simple, well understood application such as this using ordinary TDD principles to create unit tests and drive out the code. I'd also usually add some fairly basic UI tests using something like Selenium or even just HtmlUnit to validate the front-end. However, I like to use these sorts of projects to explore better ways of doing things. I've therefore decided to use a Behaviour Driven Development approach using Aslak Hellesøy's excellent Cucumber framework. (I'll post more details on installing and using Cucumber and cuke4duke with sbt once I get it all figured out!)

In this post I want to explore some of my thinking about exactly how and where to use BDD within the project and the places where I want to integrate the Cucumber features and scenarios.

What To Use BDD For

I am a strong advocate of the TDD approach, and I intend to follow this within this development for the low level details (i.e. unit and component tests). I'm therefore looking at using BDD as a way of capturing the higher-level business requirements and acceptance criteria. My aim is to use BDD and Cucumber to ensure that the application I am building actually does what the business expects it to. Lower level tests will still be written using ScalaTest to ensure the code that I am writing does what I expect it to do and also to drive out the architectural and implementation details.

Where To Integrate BDD

Given this positioning for BDD, where I have been struggling is the correct level in my application to target the Cucumber step definitions (step definitions are the code that actually executes the scenario and conditions described by the behaviour). The following sections describe the different options I have considered and advantages and disadvantages of each:

BDD at the User Interface - In this approach the step definitions are implemented to execute against the running web application. This is usually achieved using a technology such as Selenium or HtmlUnit. Each scenario loads web pages, clicks links, fills and submits forms and makes assertions about the HTML documents and associated resources that are returned.

Advanages:

• The behaviour is asserted across the complete application.
• The BDD approach can be used to drive the development of the UI as well as the application logic.

Disadvanages:

• You can only execute the feature steps against a built and deployed application - which may make development slower.
• A web-based UI is usually the least stable part of a web application and thus feature steps may be broken as the UI is tweaked, making it appear that behaviour is being lost. Step definitions may become quite fragile.
• Using BDD at this level may not fit well when user experience or visual designers are being used as this often results in rapid changing of the UI
• Generally testing at this level requires a lot of boilerplate code to be written - which makes wringing BDD step definitions more complex.
• Testing of web-based user interface may be better suited to a suite of UI tests that can focus just on verifying the UI display and interaction.
  

BDD at the Web Protocol - In this approach the step definitions are written to directly exercise the web protocol used to interface to the application. For example, the steps make HTTP calls to the application and assert that the returned results are correct and contain the expected content.

Advanages:

• Does not need to worry about the complexities of web pages, javascript and so on that are perhaps better tested in a dedicated suite of UI tests.
• Very well suited to RESTful or other web services that return structured data instead of HTML documents.

Disadvanages:

• You can still only execute the feature steps against a built and deployed application
• Building complex sequences of requests to simulate thinks such Ajax may be more complex than testing against the UI.
• Asserting against returned HTML content may be more complex than using assertions in a framework like Selenium at the UI layer.
• Building and testing at this layer requires that you also support a suite of UI tests.
• For a web application, the interface at the HTTP layer may change quite frequently, requiring frequent changes to the step definitions. This will be particularly common in an Ajax heavy application.

BDD at the Controllers - In this approach, we ignore the UI and view parts of the application. Instead we wire up the whole application from the controllers/snippets layer down. Step definitions are then written that invoke the controllers/snippets with pre-defined requests and assert that the response action and data that would be used to generate a view is correct.

Advanages:

• Features can be tested as part of the test phase of a build - no need to deploy.
• Testing at this level tends to be more stable and less fragile as the application emerges.
• It is usually much simpler to invoke and assert at a code level rather than a UI level

Disadvanages:

• A good set of testing features is required by the chosen web framework, so that requests and responses can be easily simulated and asserted against.
• Tests at this level tend to spend a significant portion of logic dedicated to the interactions with the UI rather than asserting against business rules.
• A suite of UI tests is required in order to test the UI and these will also generally exercise the same controllers/snippets in order to drive the UI.

BDD at the Services - In this final approach we test behaviour at the level directly below the controller/snippet. This is usually testing against service level code, but may also require writing some step definitions against domain level objects. As this level we are much more interested in testing the business behaviour of the application rather than the behaviour of how the application interacts with the user.

Advanages:

• Features can be tested as part of the test phase of a build - no need to deploy.
• Code at this level will usually be more stable than at the higher levels, thus making the tests less brittle.
• Step definitions at this level will usually be simpler to write and maintain than those written against the UI or controllers.
• We are testing against actual business rules rather than how a user might interact with the application.
• We require less testing specific tools and framework support.

Disadvanages:

• Testing at this level exercises less of the application flow than the tests at higher levels.
• A suite of tests is additionally required to validate the UI and the controllers to ensure that the user interaction with the system calls the services with the correct data in the correct order and correctly displays the results.
• The business is more likely expecting the behaviour of how they interact with the application to be verified.
• Developers must be VERY disciplined to ensure that no business rules, logic or behaviour is implemented in the controllers layer.

So, Where From Here?

After looking at all the possible places to use BDD and Cucumber, I have to draw the conclusion that there is no outright winner. It feels to me that the best mix is a combination of step definitions that test behaviour at the service layer (to verify business rules and behaviour) plus another set that test behaviour of the UI. By combining both of these approaches it should be possible to cover the application sufficiently to have confidence that its behaviour is correct.

This then leads me on to think about the best way to create the feature descriptions of the behaviour. Do we create a single description of the desired behaviour and then write two step definitions - one for the service layer and one for the UI? Or do we write a feature definition describing the required behaviour for the business services and then have an alternative behaviour specific for the user interaction with the system?

I'm still unsure of the correct answers to these questions. Anyone out there got any thoughts or experience? In the mean time I'm going to build my application trying both approaches and see which works out best.

Why Releases Should be Automated and Environments Equivalent

Last weekend I was involved in a fairly major release for one of my customers. Ultimately we were unable to complete the release due to a number of factors (some within our control and others outside). However, the release was still deemed successful as we were able to cleanly abort and rollback to the previous working version with an absolutely minimal service outage. The release did however highlight a number of places where the release process could be improved.
So, where did this release go wrong. There were three main problems that were encountered:

• A switch failure in one of the data centres that temporarily rendered some servers inaccessible.
• Mistakes in the manual actions in the release plan.
• A data inconsistency between two different data centres.

Unexpected Events During a Release

Murphy's Law: Anything that can go wrong, will go wrong.
Unfortunately, no matter how well you plan, there will always be things that you can't anticipate. In this particular release we had a hardware switch fail right in the middle of the release. An erroneous configuration in the data centre caused this failure to render a number of our servers unreachable. We were therefore mid-release with no way to go forward and an arduous rollback process to restore us back to a working state.
Fortunately due to diligent disaster recovery work undertaken previously we were able to fail the entire site over to the alternative data centre. A pragmatic decision was eventually taken that we couldn't wait on the chance that we MIGHT be able to complete the release so we rolled back. This was well planned and went smoothly enough that within 30 minutes of a new switch coming online we were back up and running on the original software versions. Nice work.
However, even flushed with the success of our miraculous escape from the clutches of Murphy, we have to consider what might have been. Our rollback was a largely manual process and although tested in the test environment, our production is just different enough as to add a level of uncertainty to the rollback process. There's also the greater chance of errors in manual steps when under pressure of a live situation.
So how could we be more certain of our rollback plan?

Untested Manual Release Plans

Our release had a very detailed plan that had been well reviewed before hand. A variation of the plan had been executed in test but, as I already said, the production environment is just different enough as to require a different (and slightly more complex) plan. Additionally, this release had some additional complexities requiring a number of extra manual steps.
What we discovered during the release was that some of the steps specific to production were in fact not quite correct. Couple this with the chance of making errors when executing manual steps and we have a number of places with a high potential for failure.
While the plan did run pretty smoothly, how could we reduce the chances of errors and mistakes?

Problems That Can't be Found Before Production

The final problem encountered on the release was one of database consistency. Database in two different data centres were thought to be consistent. It turns out they were not! With hindsight, we should have checked this and have been continually monitoring these over time. We live and learn.
What should really have happened is that we detected the possibility of inconsistent databases before we got to production. Unfortunately all the lower environments only have a single data centre model for the database in question so there was no way that anything could ever get out of sync in those cases. Hence, no way to detect the problem before reaching production.
How can we find these sorts of problems earlier?

Automate Releases, Equivalent Environments

Fortunately, all of the above problems can be fixed with two solutions. I wanted to say SIMPLE solutions, but unfortunately they are not particularly simple and require a fair amount of work to get right. These solutions are:

1. Automate the release and rollback pipelines.
2. Ensure Integration and Test environments are identical equivalent to Production.

Automated Releases
In a modern computing environment there is really no need to include manual steps in a release. Even when releases have lots of complexity it should be possible to script all the required steps. An automated script can be tested, fixed and re-tested many times in order to ensure it is correct before running it against a production environment. This is much better than a manual process that is difficult to test and open to human error.
Don't get me wrong, creating automated releases is hard, but the benefit of smoother, faster and more frequent releases to production gives significant enough business benefit to make the effort worthwhile.
Equivalent Environments
A manufacturer building a product wouldn't create a prototype and then go straight into mass production with a design that varied significantly from that prototype. Companies building software shouldn't do so either. The purpose of Continuous Integration is to find problems early when they are quicker and cheaper to fix. In order to work well, the lower environments need to match production in terms of applications, server structure, configuration and so on.
Additionally, if we are to build automated deployment plans then we need production like environments to develop and test them against. An unproven automated plan is of no more value than a fully manual one.

Conclusions

By automating releases and having equivalent environments we give ourself the best possible chance of undertaking regular, successful and stress-free releases. When Murphy does come along, knowing we have an automated way to get back to a working system saves a whole lot of worry and stress. Yes, it's hard to get to this point, but the long term benefit to a business from doing so is worth every penny and every hour spent.

When Points Become Hours

I was recently working on a Story Pointing exercise for one of my customers who is running a Scrum based project using User Stories (for those not familiar with User Stories, Scrum and Story Pointing, please take a look here and here). This exercise took the form of a Planning Poker session (for info on Planning Poker look here). This all works great in theory, but this project had fallen into one particular trap.....

The notion of a Story Point is that it is supposed to be a measure of story size and complexity relative to other stories in the same backlog. So, if I start with a story that I estimate as 2 points then a story that seems twice as big and complex would be 5 points (on a scale 1, 2, 3, 5, 8, 13, 20, 40, 100), while a story half the size and complexity would be 1 point. It shouldn’t be necessary to know exactly how the story will be implemented in order to gauge its size and complexity relative to other stories.

However, this particular project had over time reached a stage whereby a Story Point had become associated with a particular number of hours development and test effort. This is a dangerous situation to get in to, but very easy to achieve - especially on long running projects like this one where a large amount of historic data had been collected supporting this association.

Two specific problems that occur when Story Points become measures of number of hours of implementation effort are:

Solutionising During Story Pointing

People by nature are generally much more comfortable thinking in terms of tasks and how long they will take as opposed to relative size and complexity. Therefore, when there is a link between the two they tend to switch to the more comfortable approach of asking “how long would it take me to do this piece of work?”. Once they have this figured out they then convert this value back to Story Points.

The problem with this is that to answer the ‘how long’ question you have to know what the solution will be. Therefore you have to work out how you would implement the story then break it down into tasks and then estimate these tasks - usually all done in your head in the space of a few minutes. Chances are some tasks will be missed - meaning the estimate is wrong. Also likely is that when a team later comes to implement the story they will pick a different solution (or the best solution might have changed) and therefore come up with a completely different estimate. 

Any Story Points derived in this way are therefore meaningless unless the people estimating the story immediately go out an implement it using the exact solution used during the pointing process.

Unwillingness to Commit

The other problem with linking Story Points and implementation hours is that people then feel that giving a Story Point estimate is a commitment to complete the story in the associated number of hours. If people feel that this commitment will be enforced then they either become unwilling to commit to a Story Point estimate until they have every last detail nailed down - hence nothing gets sized or sizing takes forever. Alternatively they inflate their point estimates by a significant (usually random) amount to account for uncertainty and unknowns and thus the sizes are no longer meaningful. 

So, when a project gets into this situation, how does it get out of it? Unfortunately, the only real way to overcome the association between Story Points and hours is to repoint the entire backlog. Start with a medium size/complexity story that is pretty well understood (but which has not been implemented or estimated) and  pick an arbitrary size somewhere in the middle of the point range chart (say 5 or 8). This ensures there is no starting link between hours and size. Then estimate all other stories relative to this starting story. This could be a major undertaking for a project with a huge backlog - but it's essential in order to move forward.

And, how do you avoid getting into the same situation again? That is fortunately easier: don’t track hours against stories. A Story Point is a measure of size relative to other stories. Hours are a way of measuring the progress of a team within a sprint. There’s no link between the two and trying to put one in place is always a bad idea.