Showing posts with label planning. Show all posts
Showing posts with label planning. Show all posts

Friday, 14 September 2012

The Next Innovation

Here's a little comic strip I put together ...

When you have created so many innovative products, it's always going to be difficult to come up with enough differentiation between one version of a product and the next to keep getting people to spend their cash. Creating an entirely new product line is even more of a challenge. I'm wondering where our favourite fruit named company is going to go next.

Thursday, 12 July 2012

Making Agile Go Fast

Just recently I was speaking with a potential customer who had concerns about the velocity of their Scrum teams. They had recently (9 months ago) switched to an agile approach and had expected to get much more delivered than they had.

Now they were looking in to scaling up the number of Scrum teams on the project to try and increase velocity and deliver more quickly. In my experience of many agile projects, increasing the number of people on the project, especially one that is struggling, never has the desired effect of making things go faster. The advice to this customer was to therefore look at making their existing teams and process more effective rather than increasing the number of people on the project.

This got me thinking about what makes agile projects go slow. Across a number of projects that I have experienced, I’ve identified four common traits: two related to people and process and two related to technical concerns.

The Wrong People

Agile is all about using the right people. In fact this is true for any project or process, but for agile it is even more important. Use the right people and the agile process will work. Use the wrong ones and you will struggle.

The people who you don’t want on your agile project are those who are not disciplined enough to follow sound engineering practices and those who try to build complex frameworks/toolkits and who conduct personal science projects on your codebase. If you want to go fast then it’s vital that you have a clean, simple and well-refactored codebase that is backed up by a set of quality tests. Team members who don’t support this vision slow down the whole team.

To get back on track, remove those devs who aren’t following good engineering discipline and those who keep making things more complex. Bring in experienced agile developers to pair with the remaining team, giving them the responsibility of ensuring quality engineering practices are followed.

Putting The Process First

Another common mistake of companies starting out with agile projects is that they assume the process will make them go faster. Sorry to disappoint, but this is just not true. What makes agile teams go fast is creating open channels of communication and self organising and empowered teams.

If an organisation can’t create an excellent environment for communicating then no agile process will help. Agile helps teams go faster by eliminating the unnecessary documentation and waterfall steps. However, this information transfer must still happen and it happens through closer communication and collaboration.

Sadly, many organisations create agile teams and give them a process to follow but then insist on telling them how to organise, what tools and techniques to use, how to work and exactly what to do. This isn’t agile, it’s just another way of organising a traditional team. If you want agile to go fast then the team need total control over how they work and what they do.

To get back on track from here, relax on the process. Work on increasing communications. Give the agile teams the chance to decide how they will organise and work. Give them control and they will go faster. Agile is not a process, it’s about letting people get the job done.

Using The Wrong Technologies

So many agile projects struggle because they are using the wrong technologies. When this is the case it’s usually because the technology approach has been foisted on the team by an ‘ivory-tower’ architect or some senior manager who was wooed by the sales consultant from a large vendor. In other cases it may be that the team are forced into using a ‘corporate standard’ technology set.

In many cases the team will repeatedly identify the technology as a major factor in retrospectives. However, more often than not they will be told that this is not something that can be changed and that they have to stick with it. Why throw away hundreds of thousands or even millions on staff costs when backtracking on some poor technology choices would make so much difference.

Again, giving the teams the authority to make technology decisions is essential. Allowing technology choices to be reversed if they prove to be hindering delivery is also an important part of being agile.

Making The Architecture Too Complex

Many software systems are designed, or become, far too complex. In an agile environment this is a guaranteed way of making things go slower. Software must be kept as simple as possible so that it is easy to continually change and enhance.

If the architecture of an agile system has become so complex that it hinders development then it must be revisited and greatly simplified. Layers must be removed, interfaces reduced and code and technologies radically refactored to make everything simpler. Once that is done then the project must work all out on stopping it from getting more complex again in the future.


Only by using the right people, correctly empowered and communicating can the team have the ability to go fast and be truly agile. Only by making the correct technology choices, reversing bad technology choices and keeping everything simple can the ability to turned into real agile delivery.

Thursday, 1 December 2011

Increasing Agile Capacity

A scenario: you have a successful agile project. It’s made up of a number of small teams, each working on a separate component of the application suite. This has been working extremely well. You are capacity planning for the coming six months and you identify that one of these teams is likely to become a bottleneck. There are just too many stories that fit into their component area. How do you deal with this?

The above is something that looks likely on one of the projects that I am working on. I’ve been putting in some thinking on how this might pan out. This post documents my thoughts.

Traditional Reaction

There are typically two standard reactions to above problem: grow the team that is likely to be the bottleneck, or add an additional team to work on the component. In my experience neither of these works. Adding more people is rarely ever a good way to make a project go faster!

Growing the team

The best agile teams I have worked on have had one thing in common. They have all had between 4 and 6 devs, a QA, and a Scrum Master. As soon as the team gets any bigger than this I’ve found that the following occurs:

  • The performance gain of adding each additional dev is less each time.
  • It becomes much more difficult to gain consensus.
  • Team meetings become more long-winded and less productive.
  • Communication becomes more difficult.
  • It becomes difficult to maintain team values.
  • Code cruft (technical debt) increases.

Thus, adding additional devs and QAs to a team, either permanently or by borrowing from another team, just doesn’t work. Typically I’ve even found that the team goes slower as it has to spend more time cleaning up after itself.

Adding temporary devs (perhaps for a day or two) is, I’ve found, often worse than adding a permanent dev as there is less sense of ownership and shared team values. Working with good developers makes this less so, but I still believe it occurs more frequently than we like to admit.

Adding more teams

Having multiple teams working on a single component’s code base just doesn’t work well. I worked on one project where we had four teams working on the same code. We were tripping over each other all the time. Particular problems I have seen working in this way:

  • Communications between the teams becomes a big overhead.
  • More difficult to gain consensus on major technical decisions.
  • Scheduling work to avoid multiple pairs changing the same code is very difficult.
  • Very difficult to maintain shared values across multiple teams.
  • One team’s nice code is another team’s cruft.
  • Code cruft increases because one team is less willing to refactor another team’s code.
  • Different approaches to achieving the same implementation become more prevalent.
  • One team undoes another team’s work because they disagree with how it was done.
  • Important aspects like performance tend to slip between the cracks.
  • Productivity of the combined teams is far less than the sum of the productivity of each team working independently.

Multiple teams working on the same code base is a recipe for disaster. It results in more overhead, lower code quality and more cruft. Also, you get far less productivity gain that you would had you allowed each team to work independently on their own component.

Other Approaches

So, given that there is more work than the measured velocity of the team shows they can achieve in the time available. And, given that we don’t want to grow the team or add more teams. How do we solve the problem?

The aim is to keep small teams who can each work on an area of code that they have sole ownership of. This will give them the incentive to ensure that this component is clean, simple, maintainable and has a low level of cruft. It will also maximise the productivity of each individual team by minimising communication overhead.

I think a combination of four specific solutions are the best approach. These being: reducing waste, simplifying requirements, moving complexity and splitting components.

Reduce Waste

A good agile team should already be pretty good at reducing their waste. Retrospectives that generate actions which are then implemented go a long way towards this. A good Scrum Master who removes impediments is vital. The team will be provided with the best tools for the job.

Still, there is likely to be further waste that can be eliminated to allow the team to go faster. The first set comes from the category of unnecessary process waste. This includes things like:

  • Having to attend full meetings where there is very little need for members from the team to be present.
  • Long-winded design sessions that could be cut down with a bit more decisiveness.
  • Unnecessarily revisiting architecture and design decisions over and over again.
  • Re-enacting the same debate repeatedly because consensus wasn’t achieved.

The other set comes from giving the bottlenecked team stories to play that aren’t essential to what they are trying to deliver. This can be things like:

  • Helping ops with tracking down a problem
  • Adding additional logging of messages to an external system
  • Trying to reproduce an infrequent defect
  • Updating a build pipeline

In an ideal world, all of the above should be done by the team. However, if that team has a high volume of work to get through then these tasks can easily be divested out to teams that are perhaps not running at full capacity, allowing the team that is to maximise their throughput of important functional stories.

Simplify Requirements

Often performance of teams can be greatly hampered by having requirements that are either too complex or not clearly thought out. Ensure the backlog is groomed to contain only the most essential stories. Make sure these have been paired down to the absolute minimum set of requirements needed to fulfill the business needs.

The worst scenario is having a heavily loaded team working on features that are overly complex and time consuming. I’ve seen many cases where teams have worked on a fairly complex story only to later find that the clients uses just the core 80% of the feature, which could have been built in 20% of the time. The extra 80% of the time spent delivering that last 20% of non-essential requirement is just waste.

Analysts and the business need to scrutinise the backlog, squeezing every unnecessary piece of work from the stories. If the business doesn’t need some small aspect right now then create a separate story and play it later when the team has less critical path work.

Move complexity

In most large software systems it is possible to locate complexity in different components. For example table paging could be implemented in the server code or in javascript on the browser. If complexity in one place is causing capacity problems then it may be possible to move elements of this complexity to another team that has more spare capacity.

This may not always be the most technically pure solution, but it may be the best trade-off in order to deliver a product on time and budget. The complexity can always be relocated in the future.

As an example, consider a story where one component has 8 points of work do to delivery a feature and it’s two clients each have 2 points of work to consume the feature. Total 12 points. However, it might be possible to split the work differently, so that the core component implements something much simpler for 3 points, while the two clients take on some complexity and implement 5 points each. Total 13 points. So, the two approaches are roughly equivalent in size. The second might result in some duplication on behalf of two teams, but this allows the team with a large backlog to complete more stories. A pragmatic trade-off.

Split Components

If a team has multiple components, then the logical option is to relocate one or more of those components to another team. What if they only have one component?

Ideally, buy reducing waste, simplifying requirements, and moving complexity it should be possible to shrink the size of the backlog for a component. Hopefully the smaller backlog should be sufficiently sized that the team can complete it. What if they still can’t?

To me this seems to be a problem of decomposition. A component that is so large and complex that a team of six devs can’t maintain and enhance it is a component that is too big. In this case it should be split into two or more smaller components, with the current team keeping one part and a different team taking the other part(s).

Making the split is always technically challenging, but provided it is made across a clear interface boundary then my experience is that it is usually possible divide a component up into smaller constituent parts.

This is probably the only way that I would recommend going about adding more people to a project - introduce a new team (or repurpose an existing team) and give them their own code base and component(s) to work on.

Wednesday, 17 November 2010

Why Releases Should be Automated and Environments Equivalent

Last weekend I was involved in a fairly major release for one of my customers. Ultimately we were unable to complete the release due to a number of factors (some within our control and others outside). However, the release was still deemed successful as we were able to cleanly abort and rollback to the previous working version with an absolutely minimal service outage. The release did however highlight a number of places where the release process could be improved.
So, where did this release go wrong. There were three main problems that were encountered:
  • A switch failure in one of the data centres that temporarily rendered some servers inaccessible.
  • Mistakes in the manual actions in the release plan.
  • A data inconsistency between two different data centres.

Unexpected Events During a Release

Murphy's Law: Anything that can go wrong, will go wrong.
Unfortunately, no matter how well you plan, there will always be things that you can't anticipate. In this particular release we had a hardware switch fail right in the middle of the release. An erroneous configuration in the data centre caused this failure to render a number of our servers unreachable. We were therefore mid-release with no way to go forward and an arduous rollback process to restore us back to a working state.
Fortunately due to diligent disaster recovery work undertaken previously we were able to fail the entire site over to the alternative data centre. A pragmatic decision was eventually taken that we couldn't wait on the chance that we MIGHT be able to complete the release so we rolled back. This was well planned and went smoothly enough that within 30 minutes of a new switch coming online we were back up and running on the original software versions. Nice work.
However, even flushed with the success of our miraculous escape from the clutches of Murphy, we have to consider what might have been. Our rollback was a largely manual process and although tested in the test environment, our production is just different enough as to add a level of uncertainty to the rollback process. There's also the greater chance of errors in manual steps when under pressure of a live situation.
So how could we be more certain of our rollback plan?

Untested Manual Release Plans

Our release had a very detailed plan that had been well reviewed before hand. A variation of the plan had been executed in test but, as I already said, the production environment is just different enough as to require a different (and slightly more complex) plan. Additionally, this release had some additional complexities requiring a number of extra manual steps.
What we discovered during the release was that some of the steps specific to production were in fact not quite correct. Couple this with the chance of making errors when executing manual steps and we have a number of places with a high potential for failure.
While the plan did run pretty smoothly, how could we reduce the chances of errors and mistakes?

Problems That Can't be Found Before Production

The final problem encountered on the release was one of database consistency. Database in two different data centres were thought to be consistent. It turns out they were not! With hindsight, we should have checked this and have been continually monitoring these over time. We live and learn.
What should really have happened is that we detected the possibility of inconsistent databases before we got to production. Unfortunately all the lower environments only have a single data centre model for the database in question so there was no way that anything could ever get out of sync in those cases. Hence, no way to detect the problem before reaching production.
How can we find these sorts of problems earlier?

Automate Releases, Equivalent Environments

Fortunately, all of the above problems can be fixed with two solutions. I wanted to say SIMPLE solutions, but unfortunately they are not particularly simple and require a fair amount of work to get right. These solutions are:
  1. Automate the release and rollback pipelines.
  2. Ensure Integration and Test environments are identical equivalent to Production.
Automated Releases
In a modern computing environment there is really no need to include manual steps in a release. Even when releases have lots of complexity it should be possible to script all the required steps. An automated script can be tested, fixed and re-tested many times in order to ensure it is correct before running it against a production environment. This is much better than a manual process that is difficult to test and open to human error.
Don't get me wrong, creating automated releases is hard, but the benefit of smoother, faster and more frequent releases to production gives significant enough business benefit to make the effort worthwhile.
Equivalent Environments
A manufacturer building a product wouldn't create a prototype and then go straight into mass production with a design that varied significantly from that prototype. Companies building software shouldn't do so either. The purpose of Continuous Integration is to find problems early when they are quicker and cheaper to fix. In order to work well, the lower environments need to match production in terms of applications, server structure, configuration and so on.
Additionally, if we are to build automated deployment plans then we need production like environments to develop and test them against. An unproven automated plan is of no more value than a fully manual one.

Conclusions

By automating releases and having equivalent environments we give ourself the best possible chance of undertaking regular, successful and stress-free releases. When Murphy does come along, knowing we have an automated way to get back to a working system saves a whole lot of worry and stress. Yes, it's hard to get to this point, but the long term benefit to a business from doing so is worth every penny and every hour spent.