Increasing Agile Capacity

A scenario: you have a successful agile project. It’s made up of a number of small teams, each working on a separate component of the application suite. This has been working extremely well. You are capacity planning for the coming six months and you identify that one of these teams is likely to become a bottleneck. There are just too many stories that fit into their component area. How do you deal with this?

The above is something that looks likely on one of the projects that I am working on. I’ve been putting in some thinking on how this might pan out. This post documents my thoughts.

Traditional Reaction

There are typically two standard reactions to above problem: grow the team that is likely to be the bottleneck, or add an additional team to work on the component. In my experience neither of these works. Adding more people is rarely ever a good way to make a project go faster!

Growing the team

The best agile teams I have worked on have had one thing in common. They have all had between 4 and 6 devs, a QA, and a Scrum Master. As soon as the team gets any bigger than this I’ve found that the following occurs:

• The performance gain of adding each additional dev is less each time.
• It becomes much more difficult to gain consensus.
• Team meetings become more long-winded and less productive.
• Communication becomes more difficult.
• It becomes difficult to maintain team values.
• Code cruft (technical debt) increases.

Thus, adding additional devs and QAs to a team, either permanently or by borrowing from another team, just doesn’t work. Typically I’ve even found that the team goes slower as it has to spend more time cleaning up after itself.

Adding temporary devs (perhaps for a day or two) is, I’ve found, often worse than adding a permanent dev as there is less sense of ownership and shared team values. Working with good developers makes this less so, but I still believe it occurs more frequently than we like to admit.

Adding more teams

Having multiple teams working on a single component’s code base just doesn’t work well. I worked on one project where we had four teams working on the same code. We were tripping over each other all the time. Particular problems I have seen working in this way:

• Communications between the teams becomes a big overhead.
• More difficult to gain consensus on major technical decisions.
• Scheduling work to avoid multiple pairs changing the same code is very difficult.
• Very difficult to maintain shared values across multiple teams.
• One team’s nice code is another team’s cruft.
• Code cruft increases because one team is less willing to refactor another team’s code.
• Different approaches to achieving the same implementation become more prevalent.
• One team undoes another team’s work because they disagree with how it was done.
• Important aspects like performance tend to slip between the cracks.
• Productivity of the combined teams is far less than the sum of the productivity of each team working independently.

Multiple teams working on the same code base is a recipe for disaster. It results in more overhead, lower code quality and more cruft. Also, you get far less productivity gain that you would had you allowed each team to work independently on their own component.

Other Approaches

So, given that there is more work than the measured velocity of the team shows they can achieve in the time available. And, given that we don’t want to grow the team or add more teams. How do we solve the problem?

The aim is to keep small teams who can each work on an area of code that they have sole ownership of. This will give them the incentive to ensure that this component is clean, simple, maintainable and has a low level of cruft. It will also maximise the productivity of each individual team by minimising communication overhead.

I think a combination of four specific solutions are the best approach. These being: reducing waste, simplifying requirements, moving complexity and splitting components.

Reduce Waste

A good agile team should already be pretty good at reducing their waste. Retrospectives that generate actions which are then implemented go a long way towards this. A good Scrum Master who removes impediments is vital. The team will be provided with the best tools for the job.

Still, there is likely to be further waste that can be eliminated to allow the team to go faster. The first set comes from the category of unnecessary process waste. This includes things like:

• Having to attend full meetings where there is very little need for members from the team to be present.
• Long-winded design sessions that could be cut down with a bit more decisiveness.
• Unnecessarily revisiting architecture and design decisions over and over again.
• Re-enacting the same debate repeatedly because consensus wasn’t achieved.

The other set comes from giving the bottlenecked team stories to play that aren’t essential to what they are trying to deliver. This can be things like:

• Helping ops with tracking down a problem
• Adding additional logging of messages to an external system
• Trying to reproduce an infrequent defect
• Updating a build pipeline

In an ideal world, all of the above should be done by the team. However, if that team has a high volume of work to get through then these tasks can easily be divested out to teams that are perhaps not running at full capacity, allowing the team that is to maximise their throughput of important functional stories.

Simplify Requirements

Often performance of teams can be greatly hampered by having requirements that are either too complex or not clearly thought out. Ensure the backlog is groomed to contain only the most essential stories. Make sure these have been paired down to the absolute minimum set of requirements needed to fulfill the business needs.

The worst scenario is having a heavily loaded team working on features that are overly complex and time consuming. I’ve seen many cases where teams have worked on a fairly complex story only to later find that the clients uses just the core 80% of the feature, which could have been built in 20% of the time. The extra 80% of the time spent delivering that last 20% of non-essential requirement is just waste.

Analysts and the business need to scrutinise the backlog, squeezing every unnecessary piece of work from the stories. If the business doesn’t need some small aspect right now then create a separate story and play it later when the team has less critical path work.

Move complexity

In most large software systems it is possible to locate complexity in different components. For example table paging could be implemented in the server code or in javascript on the browser. If complexity in one place is causing capacity problems then it may be possible to move elements of this complexity to another team that has more spare capacity.

This may not always be the most technically pure solution, but it may be the best trade-off in order to deliver a product on time and budget. The complexity can always be relocated in the future.

As an example, consider a story where one component has 8 points of work do to delivery a feature and it’s two clients each have 2 points of work to consume the feature. Total 12 points. However, it might be possible to split the work differently, so that the core component implements something much simpler for 3 points, while the two clients take on some complexity and implement 5 points each. Total 13 points. So, the two approaches are roughly equivalent in size. The second might result in some duplication on behalf of two teams, but this allows the team with a large backlog to complete more stories. A pragmatic trade-off.

Split Components

If a team has multiple components, then the logical option is to relocate one or more of those components to another team. What if they only have one component?

Ideally, buy reducing waste, simplifying requirements, and moving complexity it should be possible to shrink the size of the backlog for a component. Hopefully the smaller backlog should be sufficiently sized that the team can complete it. What if they still can’t?

To me this seems to be a problem of decomposition. A component that is so large and complex that a team of six devs can’t maintain and enhance it is a component that is too big. In this case it should be split into two or more smaller components, with the current team keeping one part and a different team taking the other part(s).

Making the split is always technically challenging, but provided it is made across a clear interface boundary then my experience is that it is usually possible divide a component up into smaller constituent parts.

This is probably the only way that I would recommend going about adding more people to a project - introduce a new team (or repurpose an existing team) and give them their own code base and component(s) to work on.

Is Scala Like EJB2?

In this post and then in this followup, Stephen Colebourne (@jodastephen) states his criticisms of the Scala language and how it feels to him similar to the same feeling of when he was working with EJB2. A colleague of mine also tweeted the following:

"This is why I will never use Scala: def ++ [B >: A, That] (that: TraversableOnce[B])(implicit bf: CanBuildFrom[List[A], B, That]) : That"

Here's my thoughts and experience with the above.

I started out as a C and C++ developer, but joined the Java set right at the very beginning. I've been fortunate enough to have been working in Java since 1998. Or unfortunate, if you include the early days of the Servlet API & JSP, EJB1 and then EJB2! About 2007 I really started to feel the need for a different language, one that would allow me to express my ideas more concisely and without writing so much boilerplate code. I looked at Groovy, but found it too easy to keep dropping back and writing Java. I then learned Ruby. This taught me how expressive a good language can be and how powerful closures can be. I dabbled with a few more languages before finally finding Scala at the beginning of 2009. These days I mostly code a mix of Scala, Java and Groovy. Next challenge is Clojure.

Changing Mindset

Now, I'm an engineer at heart and my typical approach is to not only understand how to use something but also (and usually at the same time) understand how and why it works. This approach always served me well when learning languages like Java and Ruby. These languages have a fairly small surface area and a relatively simple type system. It's easy to read up on them, understand how to use them and understand exactly how they work. It's easy to look at the standard and third-party libraries and see how these work as well. For me, Scala was different and involved me making a fundamental shift in mindset about how I learned the language.

When I started out I approached Scala in the same way I did other languages: I read a book and tried to build some test applications while at the same time delving into the library and language to see how they worked. This was daunting! I can fully appreciate Stephen's position that it feels a bit like EJB2. Let's put that in context. EJB2 was a horrid mess of unnecessary complexity and complication. It aimed to simplify building enterprise apps involving persistent entities and business services, but it just added way more complexity and mind effort than it saved.

Now, back to Scala. It's a big language with a very extensive and powerful type system. If you attack trying to understand it as a whole I can see how it could be mistaken as another EJB2: there seems to be loads of complexity, a lot of which you can't see the reason for and a lot of stuff just requires so much mind effort to understand. Much of it initially seems unecessary.

It was after my initial failure to understand how the Scala language and libraries actually were built and worked that I took a step back and reconsidered my learning style. I could already see some value to the language and I wanted to explore more, but how? I then made a fundamental change in my approach. I decided that it was okay for me to start using the language without necessarily understanding exactly how all the details worked. As I used the language more I would gradually try to understand the how and why, one small step at a time.

I started learning to write Scala code. When I wanted to achieve something I researched how to do it until I was familiar with the use of a particular language feature or method. If there was some syntax, implementation detail or concept that I didn't get at that point I just noted it down as something to come back to. Now after a year or so writing Scala I look back at what I didn't initially understand and it mostly makes sense. I still occasionally encounter things that don't make total sense initially, so I just learn enough to use them and then come back later to learn the why or how. It's a system that seems to work incredibly well.

When you attack looking at and learning Scala in this more gradual way, first understanding how to use it then gradually, and incrementally, delving more deeply into the type system, you begin to realised that it's not so bad. Yes, there is complexity there, but it's not the unnecessary complexity of EJB2, it's moderate complexity that helps you be expressive, write safe code and get things done. As my knowledge of Scala has grown I've yet to find any features that I haven't been able to see the value of. It's a very well thought out language, even if it doesn't quite seems so at first.

Switching from a mindset where I have to have a full understanding or how something works before I use it to one where I'm comfortable with how to use it even if I don't know why it works was a very disconcerting change in my thinking. I'm not going to say it was an easy change to make, because it wasn't. However the effort was worth it for me because it got me using a language that I now find to be many more times more productive than Java. I think I also understand the language more deeply because I have put it to use and then come back later to understand more deeply exactly how and why it worked. I would never have got to this depth of understanding with my approach to learning other languages.

An Example

Consider the method:

  def ++ [B >: A, That] (that: TraversableOnce[B])
                        (implicit bf: CanBuildFrom[List[A], B, That]) : That

There's a lot of implementation detail in there. However, in order to use this method all I really need to understand is that it allows me to concatenate two immutable collections, returning a third immutable collection containing the elements of the other two. That's why the Scala api docs also include a use case signature for this method:

  def ++ [B](that: GenTraversableOnce[B]): List[B]

From this I understand that on a List I can concatenate anything that is GenTraversibleOnce (Scala's equivalent of a Java Iterable) and that I will get a new list returned. Initially I need understand nothing more in order to use this method. I can go away and write some pretty decent Scala code.

Some time later I learn that the >: symbol means 'same or super type of' and understand this concept. I can then refine my understanding of the method so that I now know that the collection that I want to concatenate must have elements that are the same (or a supertype) as the collection I am concatenating onto.

As I later progress and learn type classes I can see that CanBuildFrom is a type class that has a default implementation for concatenating lists. I can then take this further and create my own CanBuildFrom implementations to vary the behaviour and allow different types to be returned.

I'm first learning how to use the code, then gradually refining my understanding as my knowledge of the language grows. Trying to start out by learning exactly what the first signature meant would have been futile and frustrating.

Conclusion

If you start using Scala and you take the approach of trying to see it all, understand it all and know how and why it all works then it's a scary starting place. The size and complexity of the language (particularly its type system) can seem daunting. You could easily be confused in feeling it similar to EJB2, where reams of needless complexity just made it plain unusable.

However, by switching mindset, learning to use the language and then growing understanding of how it works over time it becomes much more manageable. As you do this you gradually realise that the perceived initial complexity is not as bad as it first seemed and that it all has great value. That initial hurdle from wanting to understand it all to being happy to understand just enough is very hard to get over, but if you do then a very powerful, concise and productive language awaits you.

The Luhny Bin Challenge

Last week Bob Lee (@crazybob), from Square, set an interesting coding challenge. All the details can be found here: http://corner.squareup.com/2011/11/luhny-bin.html. I decided this sounded like a bit of fun a set to work on a Scala implementation. I'm pleased with my final result, but also learnt something interesting along the way. My full solution is available on my github account: https://github.com/skipoleschris/luhnybin.

The challenge was an interesting one, and more difficult that initial reading would suggest. In particular, the ability to deal with overlapping card number and the need to mask them both added significantly to the difficulty of the problem.

My Algorithm

The approach I selected was fairly simple. Work through the input string until a digit character was encountered. Walk forward from there consuming digit, space of hyphen characters until either the end of the string was reached, an different character was encountered or 16 digits had been collected. Next try to mach the LUHN, first on 16 digits, then 15 and finally 14. Record an object describing the index and how many characters to mask. Repeat for the next digit character and so on. After visiting all characters in the string, apply all the masks to the input.

There were some optimisations along the way. For example, if encountering 14 or less characters then there was no chance of an overlapping card number so you could safely skip over these remaining digits after the first one had been evaluated.

My Implementation

I decided to implement in Scala as this is my language of choice at the moment. I considered both a traditional imperative approach as well as a more functional programming solution. In the end I went for a very functional implementation, based around immutable data. My main reason for this choice was because I'm working on improving my functional programming skills and I though it looked like an interesting problem to try and solve in this way.

The implementation consist of for main pieces of code. Each implements a transformation of input into output, is fairly self contained. I fell that by writing them in this way I have created code that is possible to reason about and also reuse in other scenarios. I tried for elegant code rather then optimising for the most efficient and minimal code base. I think I achieved my goals.

Conclusion

While I am very happy with my solution and had great fun building it, it was very interesting to look at other solutions that had been attempted. In particular it's quite clear that the ~900ms that I was able to active on my mid-2010 MBP was a number of times slower than the fastest solutions. The inevitable conclusion being that in some cases adopting a more mutable and imperative approach may be preferable. This is a typical example of this case, when we are in effect creating a logging filter, and speed of operation could easily be argued as more important than adhering to the principles of immutability. An interesting conclusion.

Concise, Elegant and Readable Unit Tests

Just recently I was pair programming with another developer and we were writing some unit tests for the code we were about to write (yes, we were doing real TDD!). During this exercise we came to a point where I didn't feel that the changes being made to the code met my measure of what makes a good test. As in all the best teams we had a quick discussion and decided to leave the code as it was (I save my real objections for times when it is more valuable). However, this got me thinking about what actually I was uncomfortable with, and hence the content of this post.

As a rule I like my code to have three major facets. I like it to be concise: there should be no more code than necessary and that code should be clear and understandable with minimum contextual information required to read it. It should be elegant: the code should solve the problem at hand in the cleanest and most idiomatic way. Also, it must be readable: clearly laid out, using appropriate names, following the style and conventions of the language it is written in and avoiding excessive complexity.

Now, the test in question was written in Groovy, but it could just as well be in any language as it was testing a fairly simple class that maps error information into a map that will be converted to Json. The test is:

@Test
void shouldGenerateAMapOfErrorInformation() {
    String item = "TestField"
    String message = "There was an error with the test field"
    Exception cause = new IllegalStateException()

    ErrorInfo info = new ErrorInfo()
    info.addError(item, message, cause)

    Map content = info.toJsonMap()
    assert content.errors.size() == 1
    assert content.errors[0].item == item
    assert content.errors[0].message == message
    assert content.errors[0].cause == cause.toString()
}

Now, to my mind this test is at least three lines too long - the three lines declaring variables that are then used to seed the test and in the asserts. These break my measure of conciseness by being unnecessary and adding to the amount of context needed in order to follow the test case. Also, the assertions of the map contents are less than elegant (and also not very concise), which further reduces the readability of the code. I'd much rather see this test case look more like this:

@Test
void shouldGenerateAMapOfErrorInformation() {
    ErrorInfo = new ErrorInfo()
    info.add("TestField", "There was an error with the test field", 
                 new IllegalStateException())

    Map content = info.toJsonMap()
    assert content.errors.size() == 1
    assert hasMatchingElements(contents.errors[0], 
          [ item : "TestField",
             message : "There was an error with the test field",
             cause : "class java.lang.IllegalStateException" ])
}

This test implementation is more concise. There are less lines of code and less context to keep track of. To my eye I find this much more elegant and far more readable.

"But, you've added duplication of the values used in the test", I hear you cry! That is true and it was totally intentional. Let me explain why…

Firstly, as we have already mentioned, duplicating some test values actually makes the code more concise and readable. You can look at either the test setup/execution or the assertions in isolation without needing any other context. Generally, I would agree that avoiding duplication in code is a good idea, but in cases such at this, the clarity and readability far out way the duplication of a couple of values!

Secondly, I like my tests to be very explicit about what they are expecting the result to be. For example, in the original test, we asserted that the cause string returned was the same as the result of calling toString() on the cause exception object. However, what would happen if the underlying implementation of that toString() method changed. Suddenly my code would be returning different Json and my tests would know nothing about it. This might event break my clients - not a good situation.

Thirdly, I like the intention of my code to be very clear, especially so in my tests. For example, I might create some specific, but strange, test data value to exercise a certain edge case. If I just assigned it to a variable, then it would be very easy for someone to come along, think it was an error and correct it. However, if the value was explicitly used to both configure/run the code and in the assertion then I would hope another developer might think more carefully as to my intent before making a correction in multiple places.

My final, and possibly most important reason for not using variables to both run the test and assert against is that this can mask errors. Consider the following test case:

@Test
void shouldCalculateBasketTotals() {
    Basket basket = new Basket()
    LineItem item1 = new LineItem("Some item", Price(23.88))
    LineItem item2 = new LineItem("Another item", Price(46.78))
 
    basket.add(item1)
    basket.add(item2)
 
    assert basket.vatExclusivePrice == 
                       item1.vatExclusivePrice + item2.vatExclusivePrice
    assert basket.vat == item1.vat + item2.vat
    assert basket.totalPrice == item1.price + item2.price
}

In this test, what would happen if the LineItem class had an unnoticed rounding error in its VAT calculation that was then replicated into the basket code? As we are asserting the basket values based on the fixture test data we may never notice as both sides of the assertion would report the same incorrectly rounded value. By being more explicit in our test code we not only create code that is more concise, elegant and readable but we also find more errors:

@Test
void shouldCalculateBasketTotals() {
    Basket basket = new Basket()
    LineItem item1 = new LineItem("Some item", Price(23.88))
    LineItem item2 = new LineItem("Another item", Price(46.78))
 
    basket.add(item1)
    basket.add(item2)
 
    assert basket.vatExclusivePrice == 58.88
    assert basket.vat == 11.78
    assert basket.totalPrice == 70.66
}

Finally, a couple more points that people might raise and my answers to them (just for completeness!):

Q: Can't we move the constant variables up to the class level and initialise them in a setup to keep the tests methods concise?
A: This makes an individual method contain less lines of code but fails my consiseness test as there is context outside of the test method that is required to understand what it does. You actually have to go and look elsewhere to find out what values your test is executing against. This is also less readable!

Q: Isn't is better to share test fixtures across test methods in order to keep the test class as a whole more concise?
A: This might be appropriate for some facets, such as any mocks that are used in the test or if there is a particularly large data set required for the test which is not part of the direct subject of the test. However, I'd rather see slightly longer individual test methods that are explicit in their test data and assertions, even if this means some duplication or a slightly longer overall test class.

Type Classes and Bounds in Scala

I've been doing some initial experiments with the Scalaz library (https://github.com/scalaz/). This is a great extension to the Scala standard library that adds a wide range of functional programming concepts to the language. Many of its features are based around the concept of Type Classes.

The type class concept originally comes from Haskell, and provides a way to define that a particular type supports a certain behavioural concept without the type having to explicitly know about it. For example, a type class may define a generic 'Ordered' contract and implementations can be defined for different types separate from the definition of those types. Any type that has an implementation of the 'Ordered' type class can then be used in any functions that work with ordering.

The Scala language provides a way to support type classes via its implicit mechanism. In order to get my head around this fully I created some examples to experiment with how this works. Now I fully grasp the mechanisms, the Scalaz library makes much more sense. I therefore thought I'd share my experiment in case it proves useful to anyone and as an aide-mémoire to myself for the future.

In addition, my examples also make use of the different type bounds mechanisms provided by Scala, so I will demonstrate these in my examples as well.

The full source code can be found in the following gist: https://gist.github.com/1247752.

Without Type Classes

In the first example, I built a simple solution that doesn't make any use of the type class concept. This solution is much like you would implement in Java, with an interface defining the behaviour and classes that implement this interface:

trait Publishable {  
  def asWebMarkup: String
}

case class BlogPost(title: String, text: String) extends Publishable {
  def asWebMarkup =
    """|
%s
       |
%s
""".stripMargin format(title, text)
}

Here we have a trait Publishable that indicates that something can represent itself as a web markup string. Then we have a case class that implements the trait and provides the method to return the markup. Next, we declare a class that can do the publishing:

class WebPublisher[T <: Publishable] {
  def publish(p: T) = println(p.asWebMarkup)
}

Note that we have defined this as a templates class and that we have used the <: bounds notation to require that our type T is only valid if it extends the Publishable trait. All we need to do now is use it:

val post = new BlogPost("Test", "This is a test post")
val web = new WebPublisher[BlogPost]()
web publish(post)

Ok, so that works fine. However, what happens when we either can't or don't want BlogPost to implement the Publishable trait? There can be many reasons for this: perhaps we don't have the BlogPost domain object source; perhaps the object is already quite complex and we don't want to pollute it with publishing knowledge; perhaps it's shared by multiple teams or projects and only ours needs publishing knowledge. So, what do we do?

Without type classes there are some options: we might extends the domain class to implement the trait; we might create a wrapper class or we might create a helper utility. However, all of these result in a level of indirection and complication in our code. Let me explain...

If we internalise the knowledge of the wrapper/helper/sub-class in our publishing code we end up with some horrific type matching:

class WebPublisher {
  def publish(p: AnyRef) = p match {
    case blogPost: BlogPost => BlogPostPublishHelper.publish(blogPost)
    case _ => …
  }
}

However, if we externalise the knowledge of publishing then our client code has to do the conversion:

class WebPublisher {
  def publish(p: Publishable) = ...
}

web publish(new PublishableBlogPostWrapper(blogPost))

Clearly, both solutions lead to fragile boilerplate code that pollutes our main application logic. Type classes provide a mechanism for isolating and reducing this boilerplate so that it is largely invisible to both sides of the contract.

Type Classes with Implicit Views

Fortunately, Scala provides a mechanism whereby we can declare an implicit conversion between our BlogPost and a Publishable version of our instance. So, let's start with the simple stuff:

trait Publishable {
  def asWebMarkup: String
}

case class BlogPost(title: String, text: String) 

So, we now have a blog post that doesn't implement the Publishable trait. Let's now define the conversion that can implicitly turn our blog post into something that supports publishing (we would typically add this into our publishing code rather than the domain object in order to isolate all knowledge of Publishable to just the area that needs it):

implicit def BlogPostToPublishable(blogPost: BlogPost) = new Publishable {
  def asWebMarkup =
    """|
%s
       |
%s
""".stripMargin format(blogPost title, blogPost text)
}

Our conversion just creates a new Publishable instance that wraps our blog post and implements the required methods. But, how do we make use of this? We have to change our web publisher very slightly:

class WebPublisher[T <% Publishable] {  
  def publish(p: T) = println(p.asWebMarkup)
}

All we have in fact changed is the <: bounds to a <% bounds. This new one is called a view bounds and defines that we can instantiate a WebPublisher with type T only if there is a view (in this case as implicit conversion) in scope from T to Publishable.

Our code to call this remains the same:

val post = new BlogPost("Test", "This is a test post")
val web = new WebPublisher[BlogPost]()
web publish(post)

Fantastic, our publish code gets objects that it knows are publishable, while our client code can just pass domain objects. We have all the boilerplate for conversion separated out from the main code.

However, while this all seems good, this approach does have its downsides. As you can see we are using the wrapper approach: creating a new instance of a Publishable class that wraps the original object. In a high volume system, the additional object allocations of new Publishable wrapper instances on each call to the publish method may have some less than ideal memory and garbage collection impacts.

The other problem with this approach is that is reduces our ability to compose methods in interesting ways. The reason for this is that once the publishing code has actually invoked the implicit conversion it now has the Publishable wrapper rather than the original object. If it passes this Publishable to other methods or classes than these are not aware of the original wrapped type or instance.

We can overcome this problem by modifying the Publishable trait to have a generic parameter type and support a get method to extract the wrapped value - but this then should really be called PublishableWrapper and some of the simplicity starts to break down.

I think the root of the problem here is that type classes are a very functional concept and implicit views tries to coerce these into a hybrid object/functional world. Fortunately, as of Scala 2.8 there is an additional way to implement the type class approach that leads to a more functional style of coding...

Type Classes with Implicit Contexts

An alternative to implicitly converting one class to a wrapper version of that class that adds additional behaviour is to follow more of a helper like approach. In this model we provide an implicit evidence parameter that implements the type class specific behaviour. This is a much more functional approach in that we don't alter the type we are working on. So, on with the code...

trait Publishable[T] {     
  def asWebMarkup(p: T): String
}

case class BlogPost(title: String, text: String)

This is our new Publishable trait and BlogPost class. Note that our Publishable is no longer intended to be implemented or used as a wrapper. Instead, it is now a contract definition of functional behaviour that takes a parameter of type T and transforms it into a String. A much cleaner abstraction. In fact, we can even create a single object instance that implements this behaviour for blog posts:

object BlogPostPublisher extends Publishable[BlogPost] {     
  def asWebMarkup(p: BlogPost) =
     """|
%s
        |
%s
""".stripMargin format(p title, p text)
}

We are also going to need our implicit. This time however, rather than being a conversion it becomes an evidence that we have something that implements Publishable for the type BlogPost:

implicit def Publishable[BlogPost] = BlogPostPublisher 

We also need to update our WebPublisher a bit:

class WebPublisher[T: Publishable] {  
  def publish(p: T) = println(implicitly[Publishable[T]] asWebMarkup(p))
}

There are two interesting things about this class. First, the bounds has now switched from view (<%) to context (:). The context bounds effectively modifies the class declaration to be:

class WebPublisher[T](implicit evidence$1: Publishable[T]) {  ...
}

The second change is the use of implicitly[Publishable[T]] which is a convenience for getting the implicit evidence of the correct type so that you can call methods on it.

Our code to call this remains exactly the same:

val post = new BlogPost("Test", "This is a test post")
val web = new WebPublisher[BlogPost]()
web publish(post)

One advantage that should be immediately obvious is that we are no longer creating new instances for each implicit use. Instead, we are using the implicit evident parameter (which is in this case an object) and passing our instance to it. This is more efficient in terms of allocations.

Also, we explicitly show where we make use of the implicit evidence, which is clearer. This also means that we are never creating a new wrapped type of T that we pass on to other code. We always pass on instances of type T that have implicit evidence parameters that allow T to behave as a particular type class instance.

Conclusion

We have looked at two different approaches to solving the problem of adding behaviour to an existing class without requiring it to explicitly implement a particular contract or extend a specific base class. Both of these make use of Scala implicits and type classes.

Implicit conversions with view bounds provides an approach where a type T can be converted into the type required by the type class. This works, but there are issues associated with the wrapping or transformation aspects of the conversion.

Implicit evidence parameters within context bounds overcome these problems and provide a far more functional approach to solving the same problem.

Learnings From A Scala Project

I’ve recently been working as an associate for Equal Experts. I’ve been part of a successful project to build a digital marketing back-end solution. The project started out life as a JVM project with no particular language choice. We started out with Java and some Groovy. However, the nature of the domain was one that leant itself to functional transformations of data so we started pulling in some Scala code. In the end we were fully implementing in Scala. This post describes the learning gained on this project.

The Project

The project team consisted of five experienced, full-time developers and one part-time developer/scrum master. One of the developers (me) was proficient in Scala development (18 months, part-time) while the others were all new to the language.

The initial project configuration (sprint zero) was based on a previous Java/Groovy project undertaken by Equal Experts. The initial tooling, build and library stack was taken directly from this project with some examples moved from Groovy to Scala. The initial project stack included:

• Gradle (groovy based build system) with the Scala plugin bolted in
• IntelliJ IDEA with the Scala plugin
• Jersey for RESTful web service support
• Jackson for Json/Domain Object mapping
• Spring Framework
• MongoDB with the Java driver

Tooling

The biggest hurdle faced by the project was tooling, specifically tool support for the Scala language. While Scala tooling has come on a huge amount over the last year, it is still far from perfect. The project faced two specific challenges: build tool support and IDE support.

Build Tool Support

The Gradle build tool is primarily a build tool for Java and Groovy. It supports Scala via a plugin. Similar plugins exist for other build tools like Maven and Buildr.

What we experienced on this project was that this was not the ideal scenario for building Scala projects. Build times for loading the Scala compiler were rather slow and support for incremental compilation was not always as good as we would have liked. More often than not we just had to undertake a clean and build each time. This really slowed down test-driven development.

Currently (in my opinion) the only really workable build solution for Scala is the Simple Build Tool, also known as SBT (https://github.com/harrah/xsbt/wiki). This is a Scala specific build tool and has great support for fast and incremental Scala compilation plus incremental testing.

With hindsight it would probably have been better to switch over from Gradle to SBT as soon as the decision was made to use Scala for the primary implementation language.

IDE Support

The Scala plugin for IntellJ IDEA is one of the best editors for Scala. In particular its code analysis and highlighting has come on in leaps and bounds over the last few months. However, the support for compilation and running tests within the IDE is still less than ideal. Even on core i7 laptops we were experiencing long build times due to slow compiler startup. Using the Fast Scala Compiler (FSC) made a difference, but we found it somewhat unstable.

What was more annoying was that after failed compilations in IDEA, we often found that incremental compilation would no longer work and that a full rebuild of the project was required. This was VERY slow and really hampered development.

One proven way of doing Scala development within IDEA is to use the SBT plugin and run the compilation and tests within SBT inside an IDEA window. This has proven very successful on other Scala projects I have been involved with and again with hindsight we should have investigated this further.

Some developers on the team also found the lack of refactoring options, in IDEA, when working with Scala somewhat limiting. This is a major challenge for the IDE vendors given the nature of the language and its ability to nest function and method definitions and support implicits. Personally this doesn’t bother me quite as much as my early days were spent writing C and C++ code in vi!

Libraries

Scala has excellent integration with the JVM and can make good use of the extensive range of Java libraries and frameworks. However, what we experienced in a number of cases was that integrating with Java libraries required us to implement more code than we would have liked to bridge between the Java and Scala world. Also, much of this code was less than idiomatic Scala.

What we found as the project progressed was that it became increasingly useful to switch out the Java libraries and replace them with Scala written alternatives. For example, very early we stopped using the MongoDB Java driver and introduced the Scala driver, called Casbah. Then we added the Salat library for mapping Scala case classes to and from MongoDB. This greatly simplified our persistence layer and allowed us to use much less and more idiomatic Scala code.

Aside: If you are working with MongoDB then I strongly recommend Scala. The excellent work by Brendan McAdams of 10gen and the team at Novus has created a set of Scala libraries for MongoDB that I think are unrivaled in any other language. The power of these libraries, while maintaining simplicity and ease of use, is amazing.

We never got a chance to swap out some of the other major libraries that the original project structure was built on. However with more hindsight we should perhaps have made the effort to make these swaps as they would have resulted in much less and much cleaner code. Some specific swaps that we could have done:

• Jersey for Unfiltered or Bowler
• Jackson for sjson or lift-json
• Spring for just plain Scala code

Scala projects work well with Java libraries. However when a project is being fully implemented in Scala then it’s much better to swap these Java libraries for their Scala implemented alternatives. This results in much less and more idiomatic Scala code being required.

Team Members

Having highly experienced team members makes a big difference on any project and this one was no exception. The other big advantage on this project was that all of the developers were skilled in multiple languages, even though most had not known Scala before. One thing I have found over many years is that developers who know multiple languages can learn new ones much more easily than those who only know one. I’d certainly recommend recruiting polygot developers for new Scala projects.

It also helped having at least one experienced Scala developer on the team. The knowledge of the Scala eco-system was helpful as was the ability to more quickly solve any challenging Scala type errors. However, the main benefit was knowledge transfer. All of the team members learned the Scala language, libraries and techniques much more quickly by pairing with someone experienced in the language.

Additionally, as the experienced Scala developer I found it very valuable working with good developers who were still learning the language. I found that explaining concepts really clarified my understanding of the language and how to pass this on to others. This will be of great benefit for future projects.

Any new project that is planning on using Scala should always bring on at least one developer who already has experience in the language. It creates a better product and helps the rest of the team become proficient in the language much more quickly.

Object-Oriented/Functional Mix

We found Scala’s ability to mix both object-oriented and functional approaches very valuable. All team members came from an object-oriented background and some also had experience with functional approaches. Initially development started mainly using object-oriented techniques, but leaning towards immutable data wherever possible.

Functional approaches were then introduced gradually where this was really obvious, such as implementing transformations across diverse data sets using map and filter functions. As the team got more familiar with the functional programming techniques more functionality was refactored into functional style code. We found this made the code smaller, easier to test and more reliable.

Perhaps the only real disadvantage of moving to more functional leaning Scala code is that some code became less readable for external developer with less exposure to the functional style. It became necessary to explain some of the approaches taken to external developers: something that would probably have not been necessary with more procedural style code. This is worth bearing in mind for projects were there will be a handover to a team not skilled in Scala and functional programming techniques.

Conclusions

The project was very successful. We delivered on time and more features than the customer originally expected. I don’t think we would have achieved this if the implementation was in Java, so from that point the smaller, more functional code base produced by developing in Scala was a real winner. Using Scala with MongoDB was a major benefit. Tools was the real let-down on the project. Some problems could be alleviated by moving from Java libraries and solutions to their Scala equivalents. However, there is still some way to go before Scala tools (expecially IDEs) match the speed an power of their Java equivalents.

Technical Investment vs Technical Debt

A subject that has been bothering me for a while is the concept of Technical Debt. It’s an interesting analogy, but I think it has become increasingly misused and misunderstood in the agile world. In this post I want to focus on the real meaning of Technical Debt and describe an alternative approach that worked on a current project.

What is Technical Debt?

In its simplest form, Technical Debt is work that you really should be doing but which you put off until later in order to meet some short-term delivery goal - it’s a compromise! For example, I might know that my JSON to Domain Object mapping approach is sub-optimal bit I decide to live with that for a while in order to complete a set of urgent features required for an important customer demo.

The key part of Technical Debt is that you MUST pay it back at some point in the future; and the sooner you do the better. To continue my example: as soon as the customer demo is complete, the fixing of the mapping approach should be prioritised as an urgent task.

How is it Misused?

One of the main problems I have encountered on many projects is that they misuse Technical Debt. The two most common misuses are:

• Being far too willing to accept Technical Debt; and
• Allowing Technical Debt to exist for too long (i.e. not paying it back quick enough)

Both of the above burden a project and, in my experience, they are allowed to happen because people don’t understand the true implications of Technical Debt correctly. There seems to be a general miscomprehension that Technical Debt can be captured as a story in the backlog that can be played at some time in the future much like any other story.

What I believe that most projects miss is the fact that having a Technical Debt story in the backlog not only requires that the debt is paid back in the future, but that its mere presence incurs an ongoing interest payment on the project. This interest payment comes in many different forms, some of which include:

• increased difficulty understanding the code base
• unnecessary complexity in the code
• more difficult and time consuming refactoring
• needing to write more code than necessary to add new features
• slower build and test times

All of the above (and other forms) result in one major impact: reduced team velocity. So, any Technical Debt, besides being a story on the backlog that has to be played in the future, also reduces velocity. Too much of it and velocity grinds to a halt.

Cumulative Impact

Projects that tend to allow Technical Debt to be readily created and to persist also tend to allow multiple pieces of Technical Debt to be present at the same time. This has a cumulative effect in that features are built on one piece of debt and then new debt grows on these features and so on. After a while the ongoing interest payments increase exponentially and the effort required to repay each piece of Technical Debt increases significantly.

At this point a project tends to come to a halt completely because the interest payments of having the Technical Debts stop the project adding any new features, while the effort required to pay back the debt has become too huge. The project either fails or needs a reboot.

Is There An Alternative?

From time to time, any project must accept some compromise in order to meet some critical goal. I see no problem with this. However, what projects need to do is to ensure that these compromises are kept as small as possible and have the smallest possible payback cost. So, how do we ensure this? The solution I would propose is what I have coined as ‘Technical Investment’.

Using the debt/investment analogy for personal finances, we can see two approaches: I can borrow a load of money to buy what I need now and then have to deal with interest payments until I can pay the debt back; or, I can put some money away into an investment so that when I need to buy something I can do so without having to go into debt. I temporarily reduce the size of my investment to make the purchase but then I quickly replenish it and grow it again ready for another future purchase. Hopefully my investment also grows over time so I can buy more things or reduce my investment portfolio by less each time.

So, how does this Technical Investment work in agile software development? What we do is we dedicate a small portion of the team during each iteration to refactoring, cleaning, rewriting, upgrading libraries and so on. This may be in any area of the code, not just that directly touched by the current stories in play. We invest in the code base a small amount each iteration. Perhaps we might deliver one small story or feature less each sprint, but as the project progresses we avoid the large reductions of velocity that are inherent in a Technical Debt driven model.

So, Does It Work?

On a recent project we adopted the above approach (although somewhat informally, which is why I wanted to document it more formally in this post). We had a team of five developers working on weekly sprints. Each day we created two pairs to work on features, while the fifth team member was left free to invest in the code base. The lone person changed each day or so. What did we find?

Firstly we found that the need to incur Technical Debt (or draw down from our investment, to follow the analogy) was significantly reduced. Quite often the areas where we would have had to make a compromise to meet a sprint goal were already addressed and improved before making the compromise became necessary. We were able to preempt many areas of improvement before they became an issue.

Secondly, we found that in cases where a compromise was made (where we drew down from our investment) that we were able to quickly fix the compromise and get back to adding to our investment much sooner. We also avoided the problem of any cumulative compromises, because payback was smaller, which made correcting individual cases much simpler when we decided to accept them.

By working in this way we were able to have a continually improving code base, while still maintaining a high velocity of new features. The code base continually improved in quality and clarity throughout the project even though features were continually being added.

But Shouldn’t This Happen Anyway?

This is an interesting question. Yes, a good agile team should be continually refactoring and improving code as part of every story. However in practice, the pressure to complete features often reduces the amount of improvement that is done. It’s also often difficult to justify changes to code in an area that no-one is currently working on just because it needs some love: which may benefit some unknown work in the future.

Having a concept called ‘Technical Investment’ also gives a concrete thing that can be talked about with Product Owners. We can discuss the investments made during reviews and present them in a positive way, something that I have found much more difficult to do with debt.

I suspect that it may also be possible to measure and visualise the investment in a similar way to the measuring of velocity. This is certainly something I’d like to explore in the future as being able to show how investing in the code reduces future reductions in velocity is a great way to justify the practice.

Conclusion

Keeping the code base clean and building up good quality through continual Technical Investment on each iteration has many advantages over the use of Technical Debt to capture and deal with compromises. When applied, the need to compromise becomes less and the effort involved in fixing each compromise is significantly reduced. The chances of building up cumulative, velocity zapping compromises also lessens significantly. If you have a longer-running or larger project then continual Technical Investment is the only sure way to make sure the project doesn’t grind to a halt due to debt interest as some time in the future.

The Power of Pairs

In my current contract we have a policy of full-time pair programming. It has been working incredibly well. I thought I'd share some of my observations as to why this might be the case.

Previous experience of pairing

I've used and worked with pairs in previous roles with various degrees of success. However, none of those previous roles adopted the full-time pairing model. I've always seen some benefit in pairing, especially on complex or critical pieces of code: two minds working together on these sorts of problems is invaluable. My previous roles have, however, always gone back to singletons for the more mundane and less critical coding tasks.

The one thing I have never seen previously is any of the purported productivity boosts from pair working. This could be because I've only used pairs on for complex tasks, which would take a long time anyway. But surely I should have seen some benefit in the past? Why didn't I?

Our pairing model

The team that I am working with at the moment is made up of five developers. Each of us comes with experience of many successfully completed projects. We all have different language and domain backgrounds, but there seems to be a common thirst from every member of the team for learning and improving. I think this is perhaps one of the key reasons why our pairing model works so well.

Each day we swap the pairs. Sometimes if we are deep in a task we might keep the same pair for two days, but never longer. The dynamic of the team is therefore always changing. It also encourages up to keep tasks small so that we can generally complete them in a single pairing session.

Hold on, don't we have an odd number of people? Yes, and this I think works very well. Each day we pair up and one person gets to work on their own for the day. We try to ensure that each person gets to be the singleton regularly. This single person picks up simple technical debt tasks, carries out investigative spike or just refactors and tidies up the code base. They are requested not to pick up any complex development. Working in pairs is incredibly demanding of concentration and being the singleton for a day also helps keep people fresh.

The benefits

So, why does this pairing model work so well, and why have we created a team that appears to be incredibly productive? Here's my thoughts…

The right team members - All of the developers on the team are good developers and more importantly they are open and keen to learn and improve. We have a number of different language and platform backgrounds on the team, but rather than being a problem it is just more useful information to share between us. The team members have created an environment of sharing and constant learning.

Willingness to question and compromise - Every member of the team has the ability and confidence to question the approach being taken by the other developer in their pair. While I think experience helps in this, you really want even junior team members to have this confidence to question. Linked to this is the fact that each team member is open to new ideas and is either prepared change their thinking or, even if they disagree, come to some form of compromise to allow the pair to continue moving forward. Developers who argue, fail to question or who are not prepared to learn and adapt will kill a pair's productivity.

Shared knowledge - By rotating the pairs on a daily basis, knowledge of the application, language and development approaches is spread throughout the team. This just wouldn't happen if pairs were fixed or people working alone.

The right tools and environment - The team have created a development and build environment that they are comfortable with and that is productive. The team are empowered to make any changes that allow them to work better. These could be changes to the build process or introduction of a new library the cuts the amount of or simplifies code.

A focus on quality - All the members of the team have the same view of the importance of quality and testing. There us a mutual agreement of what constitutes good quality and all team members strive for this goal. Teams where there are differing notions of quality between individual members just don't work as well - even in a non-pairing environment.

The ability to technically invest - By having one team member constantly assigned to cleaning up technical debt, refactoring and improving the code base it allows the pairs to spend all of their time investing in new features. The code base is kept clean and this makes it much easier to introduce new approaches or libraries that further improve the code.

Conclusion

So, would I adopt full-time pair programming again in the future? Definitely! However, to get the real benefit you need the right team of people and you need to create the right environment and culture for them to work in and give them the right set of tools. I'd also try to always have an odd number of developers so that pairs could be rotated and one person be free to clear down any technical debt so that the pairs can invest.

Scala eXchange London 2011

It's been an information packed two days of learning at the Scala eXchange 2011, hosted by Skills Matter. Twenty-one fantastic sessions covering a wide range of subjects. It was great to have so many in-depth sessions. Previous Scala conferences I have been to have tended to be full of introduction level sessions. This conference was different, all of the sessions went into a great deal of depth, which perhaps shows how mature the Scala community is becoming.

The highlight of the two days for me had to be James Strachan's entertaining keynote on the Scalate templating engine. It's certainly going to right to the top of my toolbox when it comes to needing a templating solution. The laughs were long and loud when a small table crashed to the ground in the corner and James commented "...that's the JSP developer falling over!". Scalate is certainly many steps beyond JSP.

Victor Klang's imaginatively titled session "The Promising Future of Akka" was my second highlight. Not only an inspired name of the presentation but some great in depth details on how Akka is implementing the Future/Promise model and how these complement the existing Actor model.

The overall impression I got from the two days was that it was a gathering of very smart people who "get it". The Scala community seems to be made up of the best and the brightest, who are investing in finding better ways to write software. The depth of the talks and the many conversations had over coffee show a great maturity of thought. These are people thinking beyond simple POJOs, DI Frameworks and ORM tools. I think we are on the verge of another major shift in the way software is designed, constructed and deployed.

The other thing that I drew from the two days is that Scala is no longer an experimental plaything, it's now a mature technology. The core language is now stable, the tools are mature to the point where they are almost as functionality complete as their Java counterparts and there is a wide understanding of how to use and apply Scala. The effort has now shifted onto solving more advanced problems like huge parallelism, complex enterprise integrations and complex event driven challenges. From being trailblazers, the Scala community are instead becoming ambassadors: showing how Scala leads to cleaner code, more elegant solutions and how Scala fits so well with the agile process.

Invariance, Covariance and Contravariance

While learning Scala I spent a fair amount of time getting my head around covariant and contravariant concepts. Just recently I found myself tryying to explain these to a collegue. I've since refined my explanation and decided to write it down for all those people who want a clear, maths free explanation.

Invariance

Let's consider a simple domain model. We have a base class 'Employee' and a sub-class 'Manager':

scala> class Employee(val number: String, val name: String)
defined class Employee

scala< class Manager(number: String, name: String, val department: String) extends Employee(number, name)
defined class Manager

Next, we have an award class that we implement using a parameterised recipient type so that awards can be used for more than just our Employee/Manager model:

scala> class Award[T](val recipient: T)
defined class Award

So, lets create an award that is for one of our managers:

scala> val mike = new Manager("1", "Mike", "Sales")
mike: Manager = Manager@712c20d9

scala> val managerAward = new Award[Manager](mike)
managerAward: Award[Manager] = Award@411a9435

All well and good. We have a manager award instance. However, say we have a variable that is of type Award[Employee]. It would seem natural to be able to assign our manager award to this variable as a manager is a type of employee, so their award should also be part of the general set of awards given to employees, right? Well...

scala> val employeeAward: Award[Employee] = managerAward
<console>:12: error: type mismatch;
 found   : Award[Manager]
 required: Award[Employee]
Note: Manager <: Employee, but class Award is invariant in type T.
You may wish to define T as +T instead. (SLS 4.5)
       val employeeAward: Award[Employee] = managerAward

So, we see that this is not possible. We also can't add our manager award to a list of awards given to employees, even though it seems natural that we should be able to do so:

scala> val awards = List[Award[Employee]](managerAward)
<console>:12: error: type mismatch;
 found   : Award[Manager]
 required: Award[Employee]
Note: Manager <: Employee, but class Award is invariant in type T.
You may wish to define T as +T instead. (SLS 4.5)
       val awards = List[Award[Employee]](managerAward)

This is an example of an invariant type. Even though the parameterised type is a subclass of the more general one, we can't assign it to an instance that has been parameterised with the more general type. This is the standard generics model of Scala and is also the way that the Java language handles generics.

Covariance

Fortunately, Scala's type system allows us to overcome this problem by allowing covariant types. We can redefine the awards class using a plus (+) character on the type to indicate that it is covariant:

scala> class Award[+T](val recipient: T)
defined class Award

Now, we can create a new manager award and can assign it to a variable of type employee award or add it to a collection of employee awards:

scala> val managerAward = new Award[Manager](mike)
managerAward: Award[Manager] = Award@7796649

scala> val employeeAward: Award[Employee] = managerAward
employeeAward: Award[Employee] = Award@7796649

scala> val awards = List[Award[Employee]](managerAward)
awards: List[Award[Employee]] = List(Award@7796649)

By using covariant types we are in effect saying that for any parameterised class its super classes include those parameterised with any of the superclasses of the type used as the parameter. Java provides a very basic mechanism for supporting covariance with the extends keyword in generic declarations:

Award<? extends Employee> employeeAward = managerAward;

However, the Java solution is less powerful because it is the variable using the instance that defines whether it allows covariance rather than allowing this to be encapsulated inside the class itself.

Contravariance

It is slightly more difficult to explain contravariance, and we first need to expand our domain model slightly. We now want to create different classes of awards and restrict who they can be presented to. Firstly we start by adding an additional generic parameter to the Award class to control the class of the award. We also create a couple of example awards restricted on this parameter:

class Award[+T, V](val recipient: T)

class AttendanceAward[T](recipient: T) extends Award[T, Employee](recipient)
class TeamLeadershipAward[T](recipient: T) extends Award[T, Manager](recipient)

The above effectively states that Attendance Awards are available to employees, while a Team Leadership Award can only be presented to Managers. Let's add a couple of functions that deal with presenting the awards:

def presentManagementAward(award: Award[Manager, Manager]) = { /* lots of pomp */ }
def presentEmployeeAward(award: Award[Employee, Employee]) = { /* a simple affair */ }

These state that a management award (with all the pomp) must be an award for managers that is presented to a manager, while an simple affair is available to an employee receiving and employee award. Wait, what happens when we want to present an employee class award to a manager with all the pomp?

scala> presentManagementAward(new AttendanceAward(mike))
<console>:14: error: type mismatch;
 found   : AttendanceAward[Manager]
 required: Award[Manager,Manager]
Note: Employee >: Manager (and AttendanceAward[Manager] <: Award[Manager,Employee]), but class Award is invariant in type V.
You may wish to define V as -V instead. (SLS 4.5)
       presentManagementAward(new AttendanceAward(mike))

We can't do this as the presentManagementAward function is expecting an award of class manager given to a manager but instead we are passing an award of class employee. However, in this case it seems wrong that this doesn't work. Why shouldn't the manager also be able to receive more general awards? It turns out that we can achieve this my modifying the award classifier to be contravariant. This makes it possible to substitute a more general parameter. If we can present a management class award then we can also present a more general employee award:

class Award[+T, -V](val recipient: T)

scala> presentManagementAward(new AttendanceAward(mike))

So there we have it, invariance, covariance and contravariance.

Alternative Votes, Politics and Software Projects

Aside for my non-UK readers: On May 5th 2011, the UK is holding a referendum on switching our electoral voting system from 'First Past The Post' (voters select a single candidate and the one with the most votes wins) to 'Alternative Vote (AV)' (voters rank candidates in order of preference, multiple analysis rounds take place where the candidate with the least votes is eliminated and their votes are redistributed to the next preferences until one candidate has 50% of the vote).

Yes, this is a software blog and yes, this is a political posting. Have I gone mad? Well, not quite. On May 5th 2011 I will be voting YES to change the UK electoral voting system from First Past The Post to Alternative Vote (AV). I have felt for a long time that the current political system is fundamentally broken. Switching to AV is the first tiny step in what will hopefully become a gradual move to a better way of doing politics.

So why do I think this is the case, and what has it got to do with software? Read on to find out....

A Badly Run Project

Consider a badly run software project. Typically the project starts out full of hope. The business defines its requirements and a development team is established to build the solution. An Architect is appointed, technologies are selected and a high-level architecture is defined that will make delivering all the requirements a doddle. Development commences. After many months the project fails to deliver, moral among the development team is very low and the business is disillusioned with the project, the Architect and development team.

Why has this happened? I'm sure any engineer who has worked on such a project can produce a myriad of possible reasons! However, there are some broad areas that tend to be common across most of these failed projects:

Over Ambitious Requirements and Scope: At the start of the project the business defines a set of requirements that encompass everything they want to see done. They expect the new software project to be able to change the world and the developers don't give them any reason to doubt this is possible. As the project starts to struggle, the business is reluctant to let go of or revise these requirements so the development team is forced to press on even though they now know it will lead to failure.

Overly Complex Architecture and Poor Technology Choice: The project's Architect has some great new technologies and patterns they have read about and want to apply. The technical requirements for the project are defined in a way to justify needing these new approaches and an architecture is built around them. As the project develops it is found that some of the choices were not good but it's either too late to go back and undo them or the Architect is unwilling to admit that he/she got it wrong. Extra architecture is added to work round the limitations of the original choices and complexity grows. Productivity is significantly reduced and quality suffers.

Poor Day-to-Day Project Management The management team for the project fails to adequately manage the scope of the project. They allow features to creep in from the requirements that are not truly essential to the completed project. They allow technical debt to accumulate in order to deliver agreed upon feature sets on prematurely early fixed dates. They fail to manage the relationship between the developers and the business. Deadlines are missed, problems covered up and an air of mistrust develops between the project team and the business.

So what happens next? Well, the project fails to deliver and the business gets upset. The project is stopped and the senior people are removed (CTO, Architect, Programme Lead and so on). The business still needs the software so it looks for a new team to take over the project. The new team reviews what was done before and decides that the only way forward is to rip up most of what was done and start over. The project is full of hope. The business reviews the requirements and comes up with a new over ambitious set. The new Architect selects his set of preferred new technologies and patterns. The team are excited again and agree to deliver the new requirements on the new architecture. The project managers vow to not repeat the poor management of the old project. Every thing starts again and the same happens all over again. Sound familiar?

A Bad Political System

Now, let's take the above and replace 'business' with 'electorate', 'development team' and 'project management' with 'government' and 'Architect' with 'Prime Minister / President'. See where I'm coming from?

It's election time and the electorate have become disillusioned with the current politicians' ability to deliver. They create a whole lot of expectations of what the government should be doing (requirements). The various parties and candidates promise they have a set of policies (architecture) that can deliver these expectations. A new party (development team) with its great leader (Architect) is voted into power. They start by throwing out the policies of the previous government and putting their new approaches into place. They make sacrifices to the long term (technical debt) to allow demonstration that they can deliver. After a while they find their their policies are not making the changes that they had hoped, but it's too late to turn back. They push on, adding extra policies to cover up the failed short-termist ones that came before. Finally their failure to deliver becomes entrenched, the electorate becomes disillusioned, a new party is voted into power and the whole process starts over again. See the similarity?

A Well Run Project

Let's consider the alternative of a well run software project. Yes, they do exist!

In this project the business has a set of requirements that they want to achieve. A good working dialogue is established between the development team and the business. The requirements are discussed. The dev team identifies those requirements that will be difficult or complex to implement. They add their own input about additional (non-functional) requirements that will be needed plus work that will be required to ensure the long-term health of the project and avoid the build up of technical debt. The business and dev team then prioritise the work, ensuring that there is a good balance between delivering essential requirements early while also ensuring the long-term objectives are taken into account.

The Architect doesn't throw away everything that came before just to try something new. Instead he selects tried and tested solutions and the stuff that is working. In those areas where there are problems or specific challenges then new approaches are tried, perhaps using a new technology that seems to be the correct fit for the problem. Planning is more about long-term adaptation and evolution of the architecture through refactoring rather than sudden and seismic shifts in architectural approach.

Given this more evolutionary approach to the architecture, the business is more willing to commit to investing some percentage of development effort into the architecture and ensuring technical debt is not built up. At the same time the development team sees an ongoing commitment to the programme and is thus less likely to take short-termist decisions to meet immediate goals that they know will hurt them later.

Communication between the business, dev team and project management remains high, open and honest. When things go wrong (and they always will) then this can be brought into the open, discussed and priorities changed, without recrimination, to resolve the issue. If requirements change then a good dialogue is already present that allows a different direction to be taken with minimal upheaval.

This is a perfect example of a collaborative, agile project that is not stuck in a particular dogma of process, technology or mindset.

What About Politics?

So, if a well run software project can work in this way, why not government? My belief is that the current adversarial, party-political system is fundamentally in conflict with this collaborative approach. The fact is that that generally one party is in power. They are trying to deliver an over ambitious set of promises built around a particular philosophy quickly enough to ensure that immediate results are visible to ensure that they get elected again. This is identical to the model of our failing software project.

What I ultimately want to see is a model much closer to our well run project: one where there are no party politics. All politicians should be elected on their individual merit. They should then come together to form a government of collaboration. There is honesty and openness with the electorate over what can be achieved and when. Policies that are working are allowed to continue (as opposed to being abandoned because they are too left/right-wing). Those that are failing can be addressed with new approaches. Policies that will need a longer period of time to deliver results are allowed the time to do so. There should be continuity rather than fundamental flips in philosophy every ten years or so.

And the AV Referendum?

My true hope is that by voting for a switch to the AV system that we will end up with a series of coalition governments. This will start a process of continued government through collaboration. There should also be a level of continuity as, in the three party system we have in the UK, it will mean that one party always remains in joint power across elections. I would then hope that over time the UK electorate will grow to expect a collaborative coalition government and that this will foster the parties to work together rather than against each other. From a small seed of AV, hopefully a large (and fundamentally different) tree will grow.

We all want to live in a successful, well governed country don't we? I know that I certainly want to be on the well run software project.

Thought For The Day: Programmers

A BAD programmer...

...writes loads of code that usually doesn't work. When they do get it to work they have no idea why it does so [programming by coincidence].

An AVERAGE programmer...

...writes loads of code that works. They are happy that it is working and they broadly understand why it does.

A GOOD programmer...

...implements the same working functionality as the average programmer, but they use far less (and more simple) code to do so. As their code is smaller and simpler, they fully understand why it works.

An OUTSTANDING programmer...

...writes the same code as the good programmer but then refactors until most of it gone. What remains is clean an elegant. Their code is so good that even the bad programmer can understand it.

Cleaner Code with the Scala Type System

Last week I happened to be sitting with a colleague looking at some prototype Java code. We came across a method that neither of us was particularly happy with, but which represented the typical messy code that occurs when wanting to do anything involving types and classes in Java. My initial comment was that it "would be so much cleaner to do in Scala". I therefore thought I'd better have play with the code to prove (if only to myself) that I was right. This post discusses the result of my effort and my conclusions from having done this work.

The Problem

In order to understand the code, let's first look at the problem that it is tying to solve:

The domain is one related to content that can appear on a website. We have a persistent domain model representing this content. However, a number of the content items are sourced from and can be updated in external systems (e.g. a CMS system or a remote content provider). We therefore need to synchronise the state of these external systems into our domain model from time to time. The java code that we were looking at was inside this synchronisation component.

The content is retrieved from the external systems, wrapped in a subtype of ForeignObject. The subtype is based on where the content was obtained from. The ForeignObject implementations can examine the content they contain and return information about its type and its unique business identifier. The synchronisation code then uses this information to lookup the associated object in the persisted domain (which may or may not exist). The ForeignObject and domain object are then passed to a mapper that updates the domain object from data that it extracts from the ForeignObject instance.

The full source code and tests described below can be obtained from my Github account: http://github.com/skipoleschris/type-safe-sync.

The Java Code

Let's first take a look at the Java code that we were less than happy about. Firstly, the ForeignObject implementation that returns the detail about the type of content that it contains:

public class CMSForeignObject implements ForeignObject {
     // Other code removed

    @Override
    public Class getLocalClass() {
        if ("Story".equals(type)) return Story.class;
        else if ("Image".equals(type)) return Image.class;
        else if ("BinaryObject".equals(type)) return BinaryObject.class;
        else if ("RelatedLinks".equals(type)) return RelatedLinks.class;
        else throw new IllegalStateException("Unknown type: " + type);
    }
}

As we can see it looks at some string based type field that it extracts from the wrapped content and returns the associated domain class that represents that content. Perhaps not the nicest code and returning class instances always feels a bit wrong. However, I don't have too many problems with this approach. Where the messy code comes in is in the synchronisation logic that uses this class:

public class SynchronisationService {
     // Setup code remove for clarity

    @SuppressWarnings( { "unchecked" })
    public Content sync(final ForeignObject foreignObject) {
        return findById(foreignObject.getLocalClass(), foreignObject);
    }

    @SuppressWarnings("unchecked")
    private  T findById(final Class clazz, final ForeignObject foreignObject) {
        if(Story.class.equals(clazz)) {
            CMSForeignObject cmsForeignObject = (CMSForeignObject)foreignObject;
            return (T)repository.findStoryById(cmsForeignObject.getId());
        }
        else if(Image.class.equals(clazz)) {
            CMSForeignObject cmsForeignObject = (CMSForeignObject)foreignObject;
            return (T)repository.findImageById(cmsForeignObject.getId());
        }
        else if(BinaryObject.class.equals(clazz)) {
            CMSForeignObject cmsForeignObject = (CMSForeignObject)foreignObject;
            return (T)repository.findBinaryObjectById(cmsForeignObject.getId());
        }
        else {
            throw new IllegalStateException(
                "We have managed to try and synchronise an unsupported class. This is a bug.");
        }
    }
}

How's that for a nice bit of logic? Imagine when we have many different types! I know this is only prototype code and there's certainly plenty we can do to improve it. However, it's quite typical to see this sort of stuff in production code whenever you have to go down into type based logic in Java.

The First Scala Version

To come up with a better solution, the first thing I did was to just convert the Java code directly into Scala. Nothing special here, other than using Scala syntax and modifying to return Option[Story] rather than null. So, first the ForeignObject implementation:

class CMSForeignObject(typeName: String, val id: String) extends ForeignObject {
  def localClass[C <: Content]() = typeName match {
    case "Story" => classOf[Story]
    case "Image" => classOf[Image]
    case "BinaryObject" => classOf[BinaryObject]
    case "RelatedLinks" => classOf[RelatedLinks]
    case _ => throw new IllegalStateException("Unknown type: " + typeName)
  }
}

And the synchronisation code:

class SynchronisationService(repository: ContentRepository) {

  def sync(foreignObject: ForeignObject): Option[Content] = findById(foreignObject.localClass, foreignObject)

  private def findById[T <: Content](clazz: Class[_], foreignObject: ForeignObject) = {
    if ( clazz == classOf[Story] )
      repository.findStoryById(foreignObject.asInstanceOf[CMSForeignObject].id).asInstanceOf[Option[T]]
    else if ( clazz == classOf[Image] )
      repository.findImageById(foreignObject.asInstanceOf[CMSForeignObject].id).asInstanceOf[Option[T]]
    else if ( clazz == classOf[BinaryObject] )
      repository.findBinaryObjectById(foreignObject.asInstanceOf[CMSForeignObject].id).asInstanceOf[Option[T]]
    else throw new IllegalStateException("We have managed to try and synchronise an unsupported class. This is a bug.");
  }
}

Works fine, but not really much better than the Java code.

Adding Scala Type Goodness

Once I had an initial Scala version I then spent a couple of hours playing around with the code and some Scala type goodness. The resulting synchronisation code is:

class SynchronisationService(repository: ContentRepository) extends ContentFinder {

  def sync(foreignObject: ForeignObject): Option[Content] = findById(foreignObject)

  private def findById[C <: Content, FO <: ForeignObject](foreignObject: FO) =
    Option(finder(findStory) orElse finder(findImage) orElse 
           finder(findBinaryObject) apply foreignObject)

  private def findStory(foreignObject: CMSForeignObject) = repository findStoryById (foreignObject id)
  private def findImage(foreignObject: CMSForeignObject) = repository findImageById (foreignObject id)
  private def findBinaryObject(foreignObject: CMSForeignObject) = repository findBinaryObjectById (foreignObject id)

No nasty if/else block. No need to deal with Class instances. No nasty type casting. Much more readable. Hopefully this is much more friendly for developers and will be far easier to maintain in the future as we come to add more types of content to be synchronised. So, how did I manage to achieve this much more readable code?

The first piece of the puzzle was to eliminate the returning of Class instances from the ForeignObject. Instead we now return a new ContentType instance:

class CMSForeignObject(typeName: String, val id: String) extends ForeignObject {

  def contentType[C <: Content, FO <: ForeignObject] = {
    val contentType = typeName match {
      case "Story" => new ContentType[Story, CMSForeignObject]
      case "Image" => new ContentType[Image, CMSForeignObject]
      case "BinaryObject" => new ContentType[BinaryObject, CMSForeignObject]
      case "RelatedLinks" => new ContentType[RelatedLinks, CMSForeignObject]
      case _ => throw new IllegalStateException("Unknown type: " + typeName)
    }
    contentType.asInstanceOf[ContentType[C, FO]]
  }
}

The ContentType instance supports generics on both the type of content and the type of foreign object. It captures the actual erased types used via implicit manifests on its constructor:

class ContentType[C <: Content, FO <: ForeignObject]
        (implicit val mC: Manifest[C], val mFO: Manifest[FO]) {

  def compatibleWith(cClass: Class[_], foClass: Class[_]) = (cClass == mC.erasure) && (foClass == mFO.erasure)
}

It also provides a compatibleWith method that is used internally within the finder code that we will look at below.

This final piece of the puzzle is the implementation of theContentFinder trait that is mixed in to the synchronisation code. This is where most of the type magic occurs in order to simplify the actual synchronisation code:

trait ContentFinder {
  implicit protected def finderToPartialFunction[FO <: ForeignObject, C]
        (handler: LookupHandler[FO, C])(implicit mFO: Manifest[FO], mC: Manifest[C]) =
    new PartialFunction[AnyRef, C] {
      def apply(foreignObject: AnyRef) =
        if ( isDefinedAt(foreignObject) ) handler(foreignObject.asInstanceOf[FO])
        else throw new IllegalStateException("We have managed to try and synchronise an unsupported class. This is a bug.")

      def isDefinedAt(foreignObject: AnyRef) =
        foreignObject.asInstanceOf[FO].contentType.compatibleWith(mC.erasure, mFO.erasure)
    }

  protected def finder[FO <: ForeignObject, C](lookup: FO => Option[C]) = new LookupHandler(lookup)

  protected class LookupHandler[FO <: ForeignObject, C](lookup: FO => Option[C]) {
    def apply(foreignObject: FO) = lookup (foreignObject).getOrElse(null).asInstanceOf[C]
  }
}

The first insight that I had when working with this code is that the if/else condition across classes is actually a partial function across content types. We have a ForeignObject that can return a ContentType instance that contains the type information about the type of content and the type of foreign object. We therefore just have to see if there is a partial function defined at that particular combination and if there is then we can apply it.

The resulting implementation therefore defines a finder method that takes a function containing the actual code that should be applied to the ForeignObject/Content pair. It returns this wrapped in a LookupHandler instance. The synchronisation code can then just define find functions for each type that it can deal with and pass them to the finder method. Finally we define an implicit conversion for the LookupHandler instances that convert them to partial functions so that we can compose them with the orElse method. The partial function captures the erased types from the handler using implicit manifests and compares these with those captured by the content type to see if the function can be applied.

Conclusion

We can clearly see that the resulting synchronisation code is significantly cleaner, more easily understood and more maintainable than the original Java code. However, this comes at the expense of slightly more lines of Scala code and some fancy logic around finders and implicit partial functions. Is the benefit really worth the extra?

I can see both sides of the argument. I can certainly understand that for the Scala novice that the ContentFinder is somewhat impenetrable. However, its the synchronisation code that is likely to be updated a number of times as more types of content are bought into the domain. The code to deal with all of the type goodness is unlikely to change that much and can be easily maintained by more experienced Scala developer. Therefore the readability and maintainability of the business code wins out in my mind over some extra, well abstracted complexity.

This tends to be the typical approach to the Scala language and one of the reasons that it can be so incredibly powerful. Use all the power of the language and its incredible type system in the framework and support code so that the actual application logic can be as minimal and simple as possible.

Selecting The Correct Test Scenarios and Data

Yesterday I happened to be reviewing some integration tests with my excellent tester colleague. These tests were defined using a set of highly simplified synthetic data selected for the purpose of making the tests pass. We both agreed that tests should actually be written using a mixture of realistic and synthetic data selected for the purpose of trying to make the tests fail! As developers we are often too keen to write tests that prove that our code is good rather than writing tests to find the problems in our code.

This exercise bought to mind a situation I had a few years ago with some code provided by an offshore development team. It was a very simple piece of code and came with some tests to show that it worked. However, it exhibited a single fundamental flaw that the synthetic data designed to make the tests pass would never have found. I've recalled the problem below (somewhat simplified), and I've update the code from Java/JUnit to Scala and Cucumber (because quite frankly I prefer these languages!).

The Requirement

The specific requirement for the system being built was that it would collect a user's date of birth during the sign-in process. If the user was under 17 then they would not be allowed to use the system. If they were 17 or over then they would be taken to the main menu. The particular piece of code in question was the age validation algorithm.

The Supplied Solution

The initial solution to this problem consisted of the following piece of code:

class AgeValidation {
  private val AuthorisedAge = 17

  def isAuthorisedAge(user: User) = {
    val cal = Calendar.getInstance
    val currentYear = cal.get(Calendar.YEAR);

    cal.setTime(user.dateOfBirth)
    val birthYear = cal.get(Calendar.YEAR);

    (currentYear - birthYear) >= AuthorisedAge
  }
}

Tests were supplied that covered two specific scenarios:

Feature: The application should ensure only users over 17 are allowed access
    As a Application Security
    I want only users aged over 17 to access the system

    Scenario: User aged over 17 can access the system
        Given a user aged over 17
        When the age check is applied
        Then they are authorised to continue

    Scenario: User aged under 17 cannot access the system
        Given a user aged under 17
        When the age check is applied
        Then they are not authorised to continue

And the implementation of these tests was as follows:

class AgeValidationSteps extends ScalaDsl with EN with ShouldMatchers {
  private var user: User = _
  private var valid: Boolean = _
  private val dateFormatter = new SimpleDateFormat("dd/MM/yyyy");

  Given("""^a user aged over 17$""") {
    user = new User("test", dateFormatter.parse("01/01/1970"))
  }

  Given("""^a user aged under 17$""") {
    user = new User("test", dateFormatter.parse("01/01/2010"))
  }

  When("""^the age check is applied$""") {
    val validator = new AgeValidation
    valid = validator.isAuthorisedAge(user)
  }

  Then("""^they are authorised to continue$""") {
    valid should be (true)
  }

  Then("""^they are not authorised to continue$""") {
    valid should be (false)
  }
}

Extra credit if you can spot the problem with this implementation and why the synthetic test data would have never found it before you read on!

The Problem With This Code

Did you spot it? Here's a hint: what if my 17th birthday is not until tomorrow? Yes, that's right, the code has a fundamental defect in that anyone who is 17 in the current year but who has not yet had their birthday is granted access to the system even though they are still 16. And yes, this was the actual algorithm submitted to us by the offshore developers!

Quite clearly, the synthetic test data used in the supplied tests was selected to ensure the tests passed, but not to find the problems in the code. Let's correct the feature spec to base it on values that actually might find something:

Feature: The application should ensure only users over 17 are allowed access
    As a Application Security
    I want only users aged over 17 to access the system

    Scenario: User who was 17 last year can access the system
        Given a user whose 17th birthday was last year
        When the age check is applied
        Then they are authorised to continue

    Scenario: User who is 17 next year cannot access the system
        Given a user whose 17th birthday is next year
        When the age check is applied
        Then they are not authorised to continue

    Scenario: User who was 17 yesterday can access the system
        Given a user whose 17th birthday was yesterday
        When the age check is applied
        Then they are authorised to continue

    Scenario: User who is 17 today can access the system
        Given a user whose 17th birthday is today
        When the age check is applied
        Then they are authorised to continue

    Scenario: User who is 17 tomorrow cannot access the system
        Given a user whose 17th birthday is tomorrow
        When the age check is applied
        Then they are not authorised to continue

And the associated test steps:

class AgeValidationSteps extends ScalaDsl with EN with ShouldMatchers {

  private var user: User = _
  private var valid: Boolean = _
  private val dateFormatter = new SimpleDateFormat("dd/MM/yyyy");

  Given("""^a user whose 17th birthday was last year$""") {
    val cal = Calendar.getInstance
    cal.add(Calendar.YEAR, -18)
    user = new User("test", cal.getTime)
  }

  Given("""^a user whose 17th birthday is next year$""") {
    val cal = Calendar.getInstance
    cal.add(Calendar.YEAR, -16)
    user = new User("test", cal.getTime)
  }

  Given("""^a user whose 17th birthday was yesterday$""") {
    val cal = Calendar.getInstance
    cal.add(Calendar.YEAR, -17)
    cal.add(Calendar.DATE, -1)
    user = new User("test", cal.getTime)
  }

  Given("""^a user whose 17th birthday is today$""") {
    val cal = Calendar.getInstance
    cal.add(Calendar.YEAR, -17)
    user = new User("test", cal.getTime)
  }

  Given("""^a user whose 17th birthday is tomorrow$""") {
    val cal = Calendar.getInstance
    cal.add(Calendar.YEAR, -17)
    cal.add(Calendar.DATE, 1)
    user = new User("test", cal.getTime)
  }

  When("""^the age check is applied$""") {
    val validator = new AgeValidation
    valid = validator.isAuthorisedAge(user)
  }

  Then("""^they are authorised to continue$""") {
    valid should be (true)
  }

  Then("""^they are not authorised to continue$""") {
    valid should be (false)
  }
}

Run this set of tests against the existing code and the final scenario 'User who is 17 tomorrow cannot access the system' fails. Our tests have actually done something useful: they have proved that our code doesn't work. Test data selected with the specific goal of trying to find problems with the code has done its job. I'll leave it as an exercise for the reader to fix the implementation so that it all works correctly!

Further Improvements

Even with our improved set of test data there are still some significant improvements that can be done to these tests by selecting some additional scenarios and test data values that might find problems:

• Testing the algorithm works with data boundary cases such as first/last day of the year, leap years and so on
• What happens when the user enters a date of birth in the future, does the code still work?
• What about if the person is over 100? Are we dealing with 4 digit or 2 digit years?
• Is this an international system? If so, then what about timezones? A person in a timezone east of us may already by 17, but may not be 17 in our timezone.

Apply The Same Rules To Our Tests

We can also apply the same thinking to our tests. For example, if we run our improved tests against the original code but do so on the first day of the year then they will all pass. The reason being that subtracting one day from the current date would push the birth date into the previous year. We therefore need to think not only about how our synthetic test data might find problems in our code by also how they may cause problems in our tests. In this case, solving this problem is just a case of improving our code to allow our tests to define the current date rather than using whatever the current system date is:

class AgeValidation(baseDate: () => Date) {
  ...
}
object AgeValidation {
  def apply = new AgeValidation(() => new Date)
  def apply(baseDate: String) = new AgeValidation(() => new SimpleDateFormat("dd/MM/yyyy").parse(baseDate))
}

// And in the tests
val validator = AgeValidation("15/07/2010")

Our test code now has the ability to control exactly what current date is used as a basis for the comparison, so it can experiment with a wider range of possible scenarios.

Does This Change Our TDD Approach

All of the above got me thinking about how this impacts on the Test-Driven Development approach. In this model we first write a test to validate the expected behaviour of our code. We then write minimal code so that the test fails. Next we write the code to make the test pass. Finally we refactor to make the code better without breaking the tests. Most developers will approach this by picking some synthetic values that are designed to prove that the 'happy path' through the code works when they have written it. Perhaps we should be focusing more on TDD where the tests we start out writing are for the scenarios that are more likely to identify problems - that way the code we write to make these pass has to deal with these cases. Certainly something I will be experimenting with in my next TDD session.