Making Agile Go Fast

Just recently I was speaking with a potential customer who had concerns about the velocity of their Scrum teams. They had recently (9 months ago) switched to an agile approach and had expected to get much more delivered than they had.

Now they were looking in to scaling up the number of Scrum teams on the project to try and increase velocity and deliver more quickly. In my experience of many agile projects, increasing the number of people on the project, especially one that is struggling, never has the desired effect of making things go faster. The advice to this customer was to therefore look at making their existing teams and process more effective rather than increasing the number of people on the project.

This got me thinking about what makes agile projects go slow. Across a number of projects that I have experienced, I’ve identified four common traits: two related to people and process and two related to technical concerns.

The Wrong People

Agile is all about using the right people. In fact this is true for any project or process, but for agile it is even more important. Use the right people and the agile process will work. Use the wrong ones and you will struggle.

The people who you don’t want on your agile project are those who are not disciplined enough to follow sound engineering practices and those who try to build complex frameworks/toolkits and who conduct personal science projects on your codebase. If you want to go fast then it’s vital that you have a clean, simple and well-refactored codebase that is backed up by a set of quality tests. Team members who don’t support this vision slow down the whole team.

To get back on track, remove those devs who aren’t following good engineering discipline and those who keep making things more complex. Bring in experienced agile developers to pair with the remaining team, giving them the responsibility of ensuring quality engineering practices are followed.

Putting The Process First

Another common mistake of companies starting out with agile projects is that they assume the process will make them go faster. Sorry to disappoint, but this is just not true. What makes agile teams go fast is creating open channels of communication and self organising and empowered teams.

If an organisation can’t create an excellent environment for communicating then no agile process will help. Agile helps teams go faster by eliminating the unnecessary documentation and waterfall steps. However, this information transfer must still happen and it happens through closer communication and collaboration.

Sadly, many organisations create agile teams and give them a process to follow but then insist on telling them how to organise, what tools and techniques to use, how to work and exactly what to do. This isn’t agile, it’s just another way of organising a traditional team. If you want agile to go fast then the team need total control over how they work and what they do.

To get back on track from here, relax on the process. Work on increasing communications. Give the agile teams the chance to decide how they will organise and work. Give them control and they will go faster. Agile is not a process, it’s about letting people get the job done.

Using The Wrong Technologies

So many agile projects struggle because they are using the wrong technologies. When this is the case it’s usually because the technology approach has been foisted on the team by an ‘ivory-tower’ architect or some senior manager who was wooed by the sales consultant from a large vendor. In other cases it may be that the team are forced into using a ‘corporate standard’ technology set.

In many cases the team will repeatedly identify the technology as a major factor in retrospectives. However, more often than not they will be told that this is not something that can be changed and that they have to stick with it. Why throw away hundreds of thousands or even millions on staff costs when backtracking on some poor technology choices would make so much difference.

Again, giving the teams the authority to make technology decisions is essential. Allowing technology choices to be reversed if they prove to be hindering delivery is also an important part of being agile.

Making The Architecture Too Complex

Many software systems are designed, or become, far too complex. In an agile environment this is a guaranteed way of making things go slower. Software must be kept as simple as possible so that it is easy to continually change and enhance.

If the architecture of an agile system has become so complex that it hinders development then it must be revisited and greatly simplified. Layers must be removed, interfaces reduced and code and technologies radically refactored to make everything simpler. Once that is done then the project must work all out on stopping it from getting more complex again in the future.

Only by using the right people, correctly empowered and communicating can the team have the ability to go fast and be truly agile. Only by making the correct technology choices, reversing bad technology choices and keeping everything simple can the ability to turned into real agile delivery.

Done and Dusted

In a recent retrospective our team dug out our Definition of Done and gave it a good review. We ended up removing a number of unnecessary items. This got me thinking about what is Done?

In previous places that I have worked the Definition of Done was a long check-list of things that had to have been completed and verified. Is this the best that a philosophy that claims to value "individuals and interactions over processes and tools" can come up with? This got me thinking...

Isn’t Done actually just the state when the team agree that the story is complete? Why should it need to be any more than that?

Done: The state when all team members agree that the story is complete

We could expand this slightly to define the roles that the different team members play in this definition:

• Developers agree that the code is complete, tested and refactored to an acceptable standard
• QAs agree that the acceptance criteria for the story has been met, that the functionality works and that they have not found any defects introduced by the new feature
• BAs and POs agree that the feature meets the business objectives for the story
• (we could also add points for DBAs, Ops etc, as needed)

So, if we accept that the team can decide for themselves when something is Done, why do we end up with long checklists, often including a whole range of metrics? It could be because 'management' aren’t able to release their need to monitor everything and produce nice reports. However, it’s more likely a matter of trust.

Teams are just not trusted to self-organise and make good decisions. If the team say that a story is Done then for many managers there is an immediate suspicion that the team is not telling the truth and is trying to cut corners. They then require that there is concrete proof that this is not the case. This is despite the fact that a group of professionals all happen to agree that it is Done. It just doesn’t make sense.

If the team has never released a story as Done when it clearly wasn’t then why doubt they will suddenly stop doing this in the future? Show trust in the team that is delivering completed work.

If the team are marking stories as Done when they clearly aren’t then there is a much more serious problem with the team and they probably either need replacing, mentoring or a firm guiding hand. Adding a requirement to satisfy a long checklist and provide associated metrics will never improve a bad team.

As with many things agile, it’s not about the process: it’s about changing attitudes; it’s about trusting teams to deliver the best solutions possible; it’s about not bogging the teams down with unnecessary controls.

Learnings From A Scala Project

I’ve recently been working as an associate for Equal Experts. I’ve been part of a successful project to build a digital marketing back-end solution. The project started out life as a JVM project with no particular language choice. We started out with Java and some Groovy. However, the nature of the domain was one that leant itself to functional transformations of data so we started pulling in some Scala code. In the end we were fully implementing in Scala. This post describes the learning gained on this project.

The Project

The project team consisted of five experienced, full-time developers and one part-time developer/scrum master. One of the developers (me) was proficient in Scala development (18 months, part-time) while the others were all new to the language.

The initial project configuration (sprint zero) was based on a previous Java/Groovy project undertaken by Equal Experts. The initial tooling, build and library stack was taken directly from this project with some examples moved from Groovy to Scala. The initial project stack included:

• Gradle (groovy based build system) with the Scala plugin bolted in
• IntelliJ IDEA with the Scala plugin
• Jersey for RESTful web service support
• Jackson for Json/Domain Object mapping
• Spring Framework
• MongoDB with the Java driver

Tooling

The biggest hurdle faced by the project was tooling, specifically tool support for the Scala language. While Scala tooling has come on a huge amount over the last year, it is still far from perfect. The project faced two specific challenges: build tool support and IDE support.

Build Tool Support

The Gradle build tool is primarily a build tool for Java and Groovy. It supports Scala via a plugin. Similar plugins exist for other build tools like Maven and Buildr.

What we experienced on this project was that this was not the ideal scenario for building Scala projects. Build times for loading the Scala compiler were rather slow and support for incremental compilation was not always as good as we would have liked. More often than not we just had to undertake a clean and build each time. This really slowed down test-driven development.

Currently (in my opinion) the only really workable build solution for Scala is the Simple Build Tool, also known as SBT (https://github.com/harrah/xsbt/wiki). This is a Scala specific build tool and has great support for fast and incremental Scala compilation plus incremental testing.

With hindsight it would probably have been better to switch over from Gradle to SBT as soon as the decision was made to use Scala for the primary implementation language.

IDE Support

The Scala plugin for IntellJ IDEA is one of the best editors for Scala. In particular its code analysis and highlighting has come on in leaps and bounds over the last few months. However, the support for compilation and running tests within the IDE is still less than ideal. Even on core i7 laptops we were experiencing long build times due to slow compiler startup. Using the Fast Scala Compiler (FSC) made a difference, but we found it somewhat unstable.

What was more annoying was that after failed compilations in IDEA, we often found that incremental compilation would no longer work and that a full rebuild of the project was required. This was VERY slow and really hampered development.

One proven way of doing Scala development within IDEA is to use the SBT plugin and run the compilation and tests within SBT inside an IDEA window. This has proven very successful on other Scala projects I have been involved with and again with hindsight we should have investigated this further.

Some developers on the team also found the lack of refactoring options, in IDEA, when working with Scala somewhat limiting. This is a major challenge for the IDE vendors given the nature of the language and its ability to nest function and method definitions and support implicits. Personally this doesn’t bother me quite as much as my early days were spent writing C and C++ code in vi!

Libraries

Scala has excellent integration with the JVM and can make good use of the extensive range of Java libraries and frameworks. However, what we experienced in a number of cases was that integrating with Java libraries required us to implement more code than we would have liked to bridge between the Java and Scala world. Also, much of this code was less than idiomatic Scala.

What we found as the project progressed was that it became increasingly useful to switch out the Java libraries and replace them with Scala written alternatives. For example, very early we stopped using the MongoDB Java driver and introduced the Scala driver, called Casbah. Then we added the Salat library for mapping Scala case classes to and from MongoDB. This greatly simplified our persistence layer and allowed us to use much less and more idiomatic Scala code.

Aside: If you are working with MongoDB then I strongly recommend Scala. The excellent work by Brendan McAdams of 10gen and the team at Novus has created a set of Scala libraries for MongoDB that I think are unrivaled in any other language. The power of these libraries, while maintaining simplicity and ease of use, is amazing.

We never got a chance to swap out some of the other major libraries that the original project structure was built on. However with more hindsight we should perhaps have made the effort to make these swaps as they would have resulted in much less and much cleaner code. Some specific swaps that we could have done:

• Jersey for Unfiltered or Bowler
• Jackson for sjson or lift-json
• Spring for just plain Scala code

Scala projects work well with Java libraries. However when a project is being fully implemented in Scala then it’s much better to swap these Java libraries for their Scala implemented alternatives. This results in much less and more idiomatic Scala code being required.

Team Members

Having highly experienced team members makes a big difference on any project and this one was no exception. The other big advantage on this project was that all of the developers were skilled in multiple languages, even though most had not known Scala before. One thing I have found over many years is that developers who know multiple languages can learn new ones much more easily than those who only know one. I’d certainly recommend recruiting polygot developers for new Scala projects.

It also helped having at least one experienced Scala developer on the team. The knowledge of the Scala eco-system was helpful as was the ability to more quickly solve any challenging Scala type errors. However, the main benefit was knowledge transfer. All of the team members learned the Scala language, libraries and techniques much more quickly by pairing with someone experienced in the language.

Additionally, as the experienced Scala developer I found it very valuable working with good developers who were still learning the language. I found that explaining concepts really clarified my understanding of the language and how to pass this on to others. This will be of great benefit for future projects.

Any new project that is planning on using Scala should always bring on at least one developer who already has experience in the language. It creates a better product and helps the rest of the team become proficient in the language much more quickly.

Object-Oriented/Functional Mix

We found Scala’s ability to mix both object-oriented and functional approaches very valuable. All team members came from an object-oriented background and some also had experience with functional approaches. Initially development started mainly using object-oriented techniques, but leaning towards immutable data wherever possible.

Functional approaches were then introduced gradually where this was really obvious, such as implementing transformations across diverse data sets using map and filter functions. As the team got more familiar with the functional programming techniques more functionality was refactored into functional style code. We found this made the code smaller, easier to test and more reliable.

Perhaps the only real disadvantage of moving to more functional leaning Scala code is that some code became less readable for external developer with less exposure to the functional style. It became necessary to explain some of the approaches taken to external developers: something that would probably have not been necessary with more procedural style code. This is worth bearing in mind for projects were there will be a handover to a team not skilled in Scala and functional programming techniques.

Conclusions

The project was very successful. We delivered on time and more features than the customer originally expected. I don’t think we would have achieved this if the implementation was in Java, so from that point the smaller, more functional code base produced by developing in Scala was a real winner. Using Scala with MongoDB was a major benefit. Tools was the real let-down on the project. Some problems could be alleviated by moving from Java libraries and solutions to their Scala equivalents. However, there is still some way to go before Scala tools (expecially IDEs) match the speed an power of their Java equivalents.

The Power of Pairs

In my current contract we have a policy of full-time pair programming. It has been working incredibly well. I thought I'd share some of my observations as to why this might be the case.

Previous experience of pairing

I've used and worked with pairs in previous roles with various degrees of success. However, none of those previous roles adopted the full-time pairing model. I've always seen some benefit in pairing, especially on complex or critical pieces of code: two minds working together on these sorts of problems is invaluable. My previous roles have, however, always gone back to singletons for the more mundane and less critical coding tasks.

The one thing I have never seen previously is any of the purported productivity boosts from pair working. This could be because I've only used pairs on for complex tasks, which would take a long time anyway. But surely I should have seen some benefit in the past? Why didn't I?

Our pairing model

The team that I am working with at the moment is made up of five developers. Each of us comes with experience of many successfully completed projects. We all have different language and domain backgrounds, but there seems to be a common thirst from every member of the team for learning and improving. I think this is perhaps one of the key reasons why our pairing model works so well.

Each day we swap the pairs. Sometimes if we are deep in a task we might keep the same pair for two days, but never longer. The dynamic of the team is therefore always changing. It also encourages up to keep tasks small so that we can generally complete them in a single pairing session.

Hold on, don't we have an odd number of people? Yes, and this I think works very well. Each day we pair up and one person gets to work on their own for the day. We try to ensure that each person gets to be the singleton regularly. This single person picks up simple technical debt tasks, carries out investigative spike or just refactors and tidies up the code base. They are requested not to pick up any complex development. Working in pairs is incredibly demanding of concentration and being the singleton for a day also helps keep people fresh.

The benefits

So, why does this pairing model work so well, and why have we created a team that appears to be incredibly productive? Here's my thoughts…

The right team members - All of the developers on the team are good developers and more importantly they are open and keen to learn and improve. We have a number of different language and platform backgrounds on the team, but rather than being a problem it is just more useful information to share between us. The team members have created an environment of sharing and constant learning.

Willingness to question and compromise - Every member of the team has the ability and confidence to question the approach being taken by the other developer in their pair. While I think experience helps in this, you really want even junior team members to have this confidence to question. Linked to this is the fact that each team member is open to new ideas and is either prepared change their thinking or, even if they disagree, come to some form of compromise to allow the pair to continue moving forward. Developers who argue, fail to question or who are not prepared to learn and adapt will kill a pair's productivity.

Shared knowledge - By rotating the pairs on a daily basis, knowledge of the application, language and development approaches is spread throughout the team. This just wouldn't happen if pairs were fixed or people working alone.

The right tools and environment - The team have created a development and build environment that they are comfortable with and that is productive. The team are empowered to make any changes that allow them to work better. These could be changes to the build process or introduction of a new library the cuts the amount of or simplifies code.

A focus on quality - All the members of the team have the same view of the importance of quality and testing. There us a mutual agreement of what constitutes good quality and all team members strive for this goal. Teams where there are differing notions of quality between individual members just don't work as well - even in a non-pairing environment.

The ability to technically invest - By having one team member constantly assigned to cleaning up technical debt, refactoring and improving the code base it allows the pairs to spend all of their time investing in new features. The code base is kept clean and this makes it much easier to introduce new approaches or libraries that further improve the code.

Conclusion

So, would I adopt full-time pair programming again in the future? Definitely! However, to get the real benefit you need the right team of people and you need to create the right environment and culture for them to work in and give them the right set of tools. I'd also try to always have an odd number of developers so that pairs could be rotated and one person be free to clear down any technical debt so that the pairs can invest.