Monday, 26 November 2012

Encoding State Requirements in Types

In my previous post I looked at a simple way to eliminate conditional expressions by encoding them into the Scala type system. In this follow-up post I want to look at taking this further by using the Scala type system to encode state rules. The aim, as always, is to create code that fails to compile rather than code that fails at runtime. By doing this we also reduce the number of unit tests required.

The Problem

Consider a simple shopping workflow. I collect information about a Basket, the Customer and their Payment Method. Once I have all this in place I can create a processOrder function that completes the workflow process. As a naive starting point, lets encode this as a simple domain model holding optional values:

  case class CheckoutWorkflow(basket: Option[Basket], 
                              customer: Option[Customer], 
                              paymentMethod: Option[PaymentMethod])

  object CheckoutWorkflow {
    def empty = new CheckoutWorkflow(None, None, None)
  }                             

Then we need some functions that populate the workflow:

  def processBasket(workflow: CheckoutWorkflow): CheckoutWorkflow = {
    if ( workflow.basket.isDefined )
      throw new IllegalStateException("Basket workflow step already processed")

    // Do some processing...
    workflow.copy(basket = Some(basket))
  }

  def processCustomer(workflow: CheckoutWorkflow): CheckoutWorkflow = {
    if ( workflow.customer.isDefined )
      throw new IllegalStateException("Customer workflow step already processed")

    // Do some processing...
    workflow.copy(customer = Some(customer))
  }

  def processPaymentMethod(workflow: CheckoutWorkflow): CheckoutWorkflow = {
    if ( workflow.paymentMethod.isDefined )
      throw new IllegalStateException("Payment Method workflow step already processed")

    // Do some processing...
    workflow.copy(paymentMethod = Some(paymentMethod))
  }

Note that each of the above functions has a guard condition to stop them being called multiple times for the same workflow. Each of these guard conditions would require a separate unit test to ensure that it works and to avoid regressions should it be accidentally removed in the future.

Finally, we need the method to process the order. Given our domain model above, this class needs to contain some conditional check to ensure that all the workflow requirements are satisfied before processing of the order can commence. Something like:

  def processOrder(workflow: CheckoutWorkflow) = workflow match {
    case CheckoutWorkflow(Some(basket), Some(customer), Some(paymentMethod)) => {
      // Do the order processing
    }
    case _ => throw new IllegalStateException("Workflow requirements not satisfied")
  }

None of the above is obviously ideal as there are a number of places that have the potential to error at runtime. The conditionals pollute our code with non-business logic. Also, we have to write good unit tests to ensure all the conditionals are working correctly and have not been accidentally removed. Even then, any client may call our code having not met the requirements encoded in the conditionals and they will receive a runtime error. Let's hope they unit test as thoroughly as we do! Surely we can do better than this?

A Less Than Ideal Solution

Well, we could use the approach outlined in my previous post and use domain model extensions:

  case class WorkflowWithBasket(basket: Basket)
  case class WorkflowWithBasketAndCustomer(basket: Basket, customer: Customer)
  case class WorkflowWithAllRequirements(basket: Basket, 
                                         customer: Customer, 
                                         paymentMethod: PaymentMethod)

  def processOrder(workflow: WorkflowWithAllRequirements) = {
     // Do the order processing
  }                                         

While this does allow removal of all the conditionals and associated tests, it unfortunatley also reduces the flexibility of our model quite significantly in that the order that the workflow must be processed is now encoded into the domain model. Not ideal. We'd like to keep the flexibility from the first solution but in a type safe way. Is there a way that we can encode the requirements into the type system?

Levaraging The Type System

First, lets consider what we want to achieve. Our aim to encode unsatisfied and satisfied workflow requirements and only allow methods to be called when the correct combinations are set. So, let's first encode the concept of requirements:

  trait BasketRequirement
  case object UnsatisfiedBasketRequirement extends BasketRequirement
  
  trait CustomerRequirement
  case object UnsatisfiedCustomerRequirement extends CustomerRequirement
  
  trait PaymentMethodRequirement
  case object UnsatisfiedPaymentMethodRequirements extends PaymentMethodRequirement

Here we have defined requirement traits for each of the different workflow stages. We also defined case objects to represent the unsatisfied state of each requirement. Next we need to indicate the satisfied states, which are our actual domain object classes:

  case class Basket(items: List[LineItem]) extends BasketRequirement
  case class Customer(id: String) extends CustomerRequirement
  case class PaymentMethod(paymentType: PaymentType) extends PaymentMethodRequirement

Next job is to make sure that our workflow object can represent these requirements and be strongly typed on either the satisfied or unsatisfied state. We do this by adding type bounds to each of the workflow types. This also allows us to eliminate the need for the Option types. We also define an 'unsatisfied' instance as the starting point for our workflow:

  case class CheckoutWorkflow[B <: BasketRequirement,
                              C <: CustomerRequirement,
                              PM <: PaymentMethodRequirement]
                              (basket: B, customer: C, paymentMethod: PM)

  object CheckoutWorkflow {
    val unsatisfied = CheckoutWorkflow(UnsatisfiedBasketRequirement,
                                       UnsatisfiedCustomerRequirement,
                                       UnsatisfiedPaymentMethodRequirements)
  }

Now we need the functions that actually process each individual workflow stage. Note how each one defines type bounded parameters for the things it doesn't care about. However, for the stage that it actually manipulates it requires that it is called with the Unsatisfied type and returns the Satisfied type. Thus, you can no longer call any of these methods with a workflow that already has that stage satisfied: the compiler won't allow it:

  def processBasket[C <: CustomerRequirement, PM <: PaymentMethodRequirement]
    (workflow: CheckoutWorkflow[UnsatisfiedBasketRequirement.type, C, PM]): 
      CheckoutWorkflow[Basket, C, PM] = {

    // Do some processing...
    workflow.copy(basket = basket)
  }

  def processCustomer[B <: BasketRequirement, PM <: PaymentMethodRequirement]
    (workflow: CheckoutWorkflow[B, UnsatisfiedCustomerRequirement.type, PM]): 
      CheckoutWorkflow[B, Customer, PM] = {

    // Do some processing...
    workflow.copy(customer = customer)
  }

  def processPaymentMethod[B <: BasketRequirement, C <: CustomerRequirement]
    (workflow: CheckoutWorkflow[B, C, UnsatisfiedPaymentMethodRequirements.type]): 
      CheckoutWorkflow[B, C, PaymentMethod] = {

    // Do some processing...
    workflow.copy(paymentMethod = paymentMethod)
  }

Finally, our processOrder function becomes super simple. You just can't call it any more unless it has no Unsatisfied types:

  def processOrder(workflow: CheckoutWorkflow[Basket, Customer, PaymentMethod]) = {
    // Process the order
  }

One observation that was made to me by someone (a Java dev) who looked at my initial draft of the code was that the final solution looks more complex due to all the type bounds and that there's actually more classes due to the need to define the requirement traits and the Unsatisfied state objects. However, don't forget that this solution eliminates at least four conditional blocks and associated unit tests, simplifies others and also possibly reduces the number of tests that clients need to write on their code as well. Also, there's no possibility of runtime failure. If the code compiles then we have a much higher confidence that it will work. Well worth a tiny bit more type complexity in my book.

Wednesday, 21 November 2012

Eliminating Conditionals Using Types

I've been at Scala eXchange for the last two days. One of the topics that came up a couple of times was how the type system can be used to encode knowledge and rules, making it possible for the compiler to enforce these rather than needing unit tests to verify them. There were questions from some about how this actually works in practice, so I thought I'd produce some blog posts to demonstrate. This first one covers the most simple case.

Eliminating Conditionals Using Types

Consider the following (highly simplified) domain model:

  case class CreditCheckResult(statusCode: String, creditLimit: Int)
  case class Customer(id: Long, name: String, creditCheckStatus: Option[CreditCheckResult])

This model indicates that a Customer may or may not have completed a credit check. However, we only want to register the customer if they have successfully passed through a credit check. The registration function therefore looks something like:

  def register(customer: Customer): Either[String, String] = customer.creditCheckStatus match {
    case None => Left("Attempt to register non-credit checked customer")
    case Some(ccs) => {
      // Do the registration here
      Right(registrationCode)
    }
  }

This is a very common pattern in any places where an Option (or in Java a null) is used to indicate some conditional state (rather than a truly optional piece of data). The result is that we now need a unit test for both conditional cases:

  class RegistrationSpec extends Specification {

    "The registration process" should {
 
      "register a credit checked customer" in {
        // Test code here
      }
   
      "fail to register a non-credit checked customer" in {
        register(Customer(1L, "Test")) must_== Left("Attempt to register non-credit checked customer")
      }
    }
  }

However, we can use the type system to completely remove the need for this conditional check and thus remove an entire unit test case from our code base. How do we do this? By representing the different states that a customer can be in with different types:

  case class CreditCheckResult(statusCode: String, creditLimit: Int)

  trait Customer {
    def id: Long
    def name: String
}

  case class UncheckedCustomer(id: Long, name: String) extends Customer
  case class CreditCheckedCustomer(id: Long, name: String, creditCheckStatus: CreditCheckResult) extends Customer

Our registration method can now be greatly simplified:

  def register(customer: CreditCheckedCustomer) = {
    // Do the registration here
  }

And our test now needs to cover only the valid registration case:

  class RegistrationSpec extends Specification {

    "The registration process" should {
 
      "register a credit checked customer" in {
        // Test code here
      }
    }
  }

Aaaahaaa, I hear you cry, but doesn't the complexity instead move to the creation of the correct type of Customer? In some cases this might be true, but you could easily validate this by constructing customers using this code as part of the above test. Hopefully you will be using a functional programming style and you will already have a function somewhere that takes an UncheckedCustomer and transforms them into a CreditCheckedCustomer as part of the credit check process: which will already by type safe and tested!

I'll add some more examples of using types to reduce code and unit testing over the coming week or two.

Friday, 14 September 2012

The Next Innovation

Here's a little comic strip I put together ...

When you have created so many innovative products, it's always going to be difficult to come up with enough differentiation between one version of a product and the next to keep getting people to spend their cash. Creating an entirely new product line is even more of a challenge. I'm wondering where our favourite fruit named company is going to go next.

Tuesday, 17 July 2012

Unit Testing with MongoDB

Back in June I presented at the London Scala User Group (video here) on our experiences on a recent Scala project that used MongoDB.

One of the questions I was asked was how we go about mocking MongoDB for testing purposes (especially unit testing)? My answer was that we don’t! This raised a couple of eyebrows in the audience, but we’ve found this works just fine for us.

In this blog post I will explore our thoughts and reasoning in a bit more detail.

How do we test with MongoDB?

I’ll be honest: for the majority of our tests (even the unit ones) we just run against a local MongoDB instance. When we are testing repository classes we write tests against a real repository wired up against a real MongoDB instance. We assert against the contents in the MongoDB collections. In some cases, we even test service classes or even controllers with a live repository backend that writes to and retrieves from a real MongoDB instance.

“But you can’t do that!”, I hear you cry. Traditional thinking says that writing tests directly against the database will be too slow. We’ve actually found MongoDB (especially when running locally) to be so blindingly fast that we don’t need to consider doing anything else. For most updates and queries we tend to get sub-millisecond responses.

It’s also a hell of a lot easier and less fragile to back tests with real repository instances than it is to create and configure a load of mocks or strange database testing frameworks.

I’m pragmatic. If using a real database backend was making our tests run too slow or causing them to be too complex then I’d happily consider alternatives. Our experience with MongoDB is that it is both fast enough and that it’s query syntax is succinct enough that we don’t need to bother with those alternatives.

Do we use mocks at all for the database then?

Actually, I rarely use mocks in any of my Scala project now days. I’ve switched much more towards a style of isolating my calls to dependencies and then using mixin traits to override specific behavior for testing. (I’ll write a blog post about this approach soon).

From time to time we do need to write tests that make a repository dependency exhibit a specific behavior which is difficult to emulate against an actual database. In these cases we just use a custom test trait mixed into the repository in order to override methods to exhibit the behavior that we require. Keeps it simple and makes the tests much more readable and much less fragile than using mocks or other frameworks.

Thursday, 12 July 2012

Making Agile Go Fast

Just recently I was speaking with a potential customer who had concerns about the velocity of their Scrum teams. They had recently (9 months ago) switched to an agile approach and had expected to get much more delivered than they had.

Now they were looking in to scaling up the number of Scrum teams on the project to try and increase velocity and deliver more quickly. In my experience of many agile projects, increasing the number of people on the project, especially one that is struggling, never has the desired effect of making things go faster. The advice to this customer was to therefore look at making their existing teams and process more effective rather than increasing the number of people on the project.

This got me thinking about what makes agile projects go slow. Across a number of projects that I have experienced, I’ve identified four common traits: two related to people and process and two related to technical concerns.

The Wrong People

Agile is all about using the right people. In fact this is true for any project or process, but for agile it is even more important. Use the right people and the agile process will work. Use the wrong ones and you will struggle.

The people who you don’t want on your agile project are those who are not disciplined enough to follow sound engineering practices and those who try to build complex frameworks/toolkits and who conduct personal science projects on your codebase. If you want to go fast then it’s vital that you have a clean, simple and well-refactored codebase that is backed up by a set of quality tests. Team members who don’t support this vision slow down the whole team.

To get back on track, remove those devs who aren’t following good engineering discipline and those who keep making things more complex. Bring in experienced agile developers to pair with the remaining team, giving them the responsibility of ensuring quality engineering practices are followed.

Putting The Process First

Another common mistake of companies starting out with agile projects is that they assume the process will make them go faster. Sorry to disappoint, but this is just not true. What makes agile teams go fast is creating open channels of communication and self organising and empowered teams.

If an organisation can’t create an excellent environment for communicating then no agile process will help. Agile helps teams go faster by eliminating the unnecessary documentation and waterfall steps. However, this information transfer must still happen and it happens through closer communication and collaboration.

Sadly, many organisations create agile teams and give them a process to follow but then insist on telling them how to organise, what tools and techniques to use, how to work and exactly what to do. This isn’t agile, it’s just another way of organising a traditional team. If you want agile to go fast then the team need total control over how they work and what they do.

To get back on track from here, relax on the process. Work on increasing communications. Give the agile teams the chance to decide how they will organise and work. Give them control and they will go faster. Agile is not a process, it’s about letting people get the job done.

Using The Wrong Technologies

So many agile projects struggle because they are using the wrong technologies. When this is the case it’s usually because the technology approach has been foisted on the team by an ‘ivory-tower’ architect or some senior manager who was wooed by the sales consultant from a large vendor. In other cases it may be that the team are forced into using a ‘corporate standard’ technology set.

In many cases the team will repeatedly identify the technology as a major factor in retrospectives. However, more often than not they will be told that this is not something that can be changed and that they have to stick with it. Why throw away hundreds of thousands or even millions on staff costs when backtracking on some poor technology choices would make so much difference.

Again, giving the teams the authority to make technology decisions is essential. Allowing technology choices to be reversed if they prove to be hindering delivery is also an important part of being agile.

Making The Architecture Too Complex

Many software systems are designed, or become, far too complex. In an agile environment this is a guaranteed way of making things go slower. Software must be kept as simple as possible so that it is easy to continually change and enhance.

If the architecture of an agile system has become so complex that it hinders development then it must be revisited and greatly simplified. Layers must be removed, interfaces reduced and code and technologies radically refactored to make everything simpler. Once that is done then the project must work all out on stopping it from getting more complex again in the future.


Only by using the right people, correctly empowered and communicating can the team have the ability to go fast and be truly agile. Only by making the correct technology choices, reversing bad technology choices and keeping everything simple can the ability to turned into real agile delivery.

Friday, 22 June 2012

Cucumber JVM Support for SBT

I am happy to announce release 0.5.0 of my xsbt plugin for cucumber. This is a major new update that finally switches from using the Ruby version of Cucumber (running under JRuby) to the newly released Cucumber-JVM.

Full details of how to use the plugin can be found on the github page: https://github.com/skipoleschris/xsbt-cucumber-plugin

A list of significant changes for this release include:

  • Switched to cucumber-jvm
  • Less configuration options due to no longer requiring JRuby
  • Improved control of output options

Sunday, 10 June 2012

Pairing is Wearing

In this post back in August 2011 I was talking about the many benefits of Pair Programming. Having now been full-time pair programming for almost an entire year, I wanted to mention one of the downsides that I have discovered.

A Holiday

I've just finished my holiday. It wasn't the best one ever, mainly due to camping in the UK in June: which meant heavy rain and strong winds. Still, we had a good time and it was very relaxing.

As the week progressed and I had a chance to relax, unwind and get some lie-ins I started to discover that my mind was starting to become more alert. I was feeling less tired. Great ideas were starting to flow again. It was at this point I realised just how weary I had been at the start of the holiday. I had known I was feeling pretty knackered, just not quite how much.

This then got me thinking as the the possible cause of this tiredness and how it could gradually build up over time without me being fully aware of it. I looked back at what I've been doing for the past year...

  • I've been travelling a lot to get to/from work - but less than my previous job.
  • I've been learning lots of new stuff - but I'm a compulsive learner, so nothing new here.
  • We've put at least 3 major releases live - but I've put plenty of software live before.
  • We've been pairing full time - this is new!

Is Pairing More Tiring?

The conclusions that I have had to draw is that working in a paired environment is more mentally tiring that working alone or individually within a team. There's more of a requirement to stay focused when working as a pair - it's particularly noticable when one member of the pair is feeling unwell or tired. Also while pairing you are multitasking more - writing code, thinking about solutions but also communicating your thoughts and maintaining a running dialogue. All of this extra work requires extra mental energy and thus increases fatigue.

So, how can we pair over a period of time without burning out? Here's some thoughts that I hope to experiment with over the coming months:

  • Take more frequent and longer breaks than you would do while working individually. The productivity gain of working in pairs easily allows this to be supported
  • Do a lot more walking at lunchtime!
  • Try to provide team members regular tasks that they can undertake outside of a pair to give their mental energy drain a rest. Stuff that doesn't require extensive multitasking.

Anyone else got any thoughts or ideas?

Monday, 30 April 2012

Life Without Objects

Over the last few years I have been learning and experimenting with a wider range of programming languages. In particular I have started using Scala as my primary development language, adopting a more and more functional style. I have also become very interested in Haskell (a pure functional language) and Clojure (a modern lisp dialect).

I have therefore been moving away from the object-oriented development principles that have made up the bulk of my 17 year career to date. More and more I am beginning to feel that objects have been a diversion away from building concise, well structured and reusable software.

As I pondered on this topic, I realised that this isn’t a sudden switch in my thinking. The benefits of objects have been gradually declining over a long period of time. The way I use objects today is very different to how I used them when they were new and shiny. In this post I explore this change in my thinking about object-oriented development.

The Promise of Objects

Back in the 1990s, objects were new and exciting. The promise of being able to create reusable classes built around common design patterns was seductive. The ability to then combine these classes into reusable and configurable business components seemed like the Mecca of software development. New languages like C++ and then, slightly later, Java held the promise of a new way of building great software.

Business Components Aren’t Reusable
It didn’t take us long to discover that the ability to create reusable business components was just a giant fallacy. Each business is significantly different from another, even in the same industry. Each similar project has very different business rules.

The only way to build reusable business components at this level is to make them hyper-configurable by adding such things as rules engines and embedded scripting languages. Hardly a component model and more like a bloatware model. This promise gone, people either buy into the big bloatware systems (suckers) or build their custom business objects on a project by project basis.

Patterns Don’t Build Well Structured Software
The next thing we learnt was that excessive reliance on design patterns doesn’t lead to the good software structure. Instead it leads to software that is overly complex, hard to understand and difficult to maintain. Some patterns even turned out to be anti-patterns (the singleton pattern makes software almost impossible to unit test, for example).

We soon learnt to use patterns judiciously. More often than not it’s just cleaner to code the software as you understand the model rather than try to abstract it into a more generalised pattern.

Frameworks for Class and Component Reuse Give Few Benefits
Another early promise of objects was rich, tightly coupled frameworks of classes which when used together would make building applications from reusable component a breeze by hiding all the technical complexity and implementation plumbing. Think EJB and similar. Experience soon showed that these just did not work. They were just too restrictive and cumbersome for what people were trying to achieve.

These heavy-weight frameworks soon died out to be replaced with more lightweight libraries and toolkit type approaches. Collections of more loosely coupled classes that you can pull together as needed are now the preferred way to build software. Take just what you need and nothing more.

Inheritance Creates Brittle Software
The ability to support interface and implementation inheritance was one of the key tenets of object oriented software development. We could spot common code and behaviour and push this down into a shared base class so that future abstractions could benefit from having this shared code available to build on.

Sadly, this just didn’t work out well. Each sub-class turns out to be subtly different from its peers, resulting in lots of overrides of base class behaviour or making the base classes even more generic. The net result was super-fragile software, where any small changes to a common base class would break most, if not all, of the sub-class implementations.

These days we don’t use inheritance much, and especially not for creating technical base classes. Its use is pretty much restricted to interface inheritance to indicate an object supports a certain behaviour or to occasional domain models where there is a true inheritance relationship. Other than that we tend to extract commonality in to separate ‘mixin’ type classes and compose them together through an aggregation approach.

Violation of Encapsulation
Another key feature of the object-oriented model was the ability to encapsulate state and then expose behaviours (via methods) that access and update this state. Unfortunately it turns out that there are a large number of cases, where we are actually interested in the vales of the state rather than the behaviour.

For example, asking an object to render itself as HTML turns out to be a pretty poor approach. Knowledge of HTML rendering gets spread across the code base and a small change in approach causes us to change many, many classes. Instead we tend to pass the object to a dedicated HTML rendering/template component, which pulls the data values from the object.

Anti-patterns have even emerged around this to allow us to have light-weight objects that just encapsulate state without behaviour (Java Beans, Data Transfer Objects and so on). If we are doing this, then why not just work directly with first-class structured data as opposed to objects?

Mutable State Causes Pain

Another perceived benefit of encapsulation was the ability to mutate the state of an object instance without impacting on the clients that use that object. However, anyone who has built a significant sized object-oriented system can tell you stories of trawling through many files of code to find the location that mutated the state of an object to an unexpected value that happened to make your software blow up in a completely different place (usually where you output or store the state of that object).

More and more we now favour immutable state and stateless services so that these problems do not occur. There’s also the additional benefit that immutable state is a much better model for building highly concurrent systems and for getting the most out of modern multi-core hardware. It’s also far easier and less error prone than trying to work with threads, locks and concurrency safe data structures.

Behavioural Interfaces Cause Code Complexity and Bloat
One of the things we do frequently in object-oriented languages is create small marker interfaces that have just a single method. Any class wanting to support this behaviour extends the interface and implements the method. We can also declare anonymous implementations for ad-hoc use.

However, we have found that neither of these approaches are particularly good. Implementing the marker interfaces often pollutes classes with implementations that are not their direct concern (thus violating the single responsibility principle). Anonymous classes are just unnecessary bolierplate that makes our code more difficult to understand and maintain.

Life Without Objects

So, is it possible to go back on 17 years of experience and contemplate a life without objects? I’m not sure that I’m 100% there just yet, but using a multi-paradigm language like Scala is allowing me to overcome many of the limitations of the object-oriented approach.

For example, Scala’s support for mixin traits makes it almost unnecessary to ever use implementation inheritance. It’s rich collections framework plus the ability to use case classes to create data structure like concepts obviates working around encapsulation issues. A recommendation to use immutable data and collections makes code easier to debug and reason about. The ability to use functions as general abstractions and type classes to extend behaviour while maintaining single responsibilities makes it much easier to build well structured, reusable software.

In fact, what I find I am doing more and more is using simple objects in the form of case classes to represent data structures, with a few behavioural methods to simplify working with this data. Then I’m just using mixin traits as a modular approach for grouping related functions together. Then I’m combining these together to form components in which I compose together various functions that transform data from one for into another.

Perhaps I’m further away from the pure object-oriented approach than I’d thought. I’m certainly building smaller, cleaner and better structured software than I ever was before.

Monday, 26 March 2012

Option

In my previous post I introduced Dave and we looked at map, flatten and flatMap. In this new episode we take a look at the Scala Option type:

Here's some code...

case class Person(name: String, partner: Option[Person])


sealed trait Product
case object XBox extends Product
case class WeekendAway(person1: Person, person2: Person) extends Product

type Cash = Int
case class Purchase(product: Product, change: Cash)


def buyXBox(cash: Cash): Purchase = Purchase(XBox, cash - 20000)

def bookWeekendAway(p1: Person, p2: Person, cash: Cash): Purchase = Purchase(WeekendAway(p1, p2), cash - 15000)


val lucy = Person("lucy", None)
val dave = Person("dave", Some(lucy))
//val dave = Person("dave", None)
val wages = 50000

val purchase = dave.partner map (bookWeekendAway(dave, _, wages)) getOrElse buyXBox(wages)
println(purchase)

Tuesday, 20 March 2012

Map, Flatten and FlatMap

I've recently been working with some developers who are new to the Scala world. They've picked up the language really easily and in only a few week have become competent Scala developers. Who says it's too complex and difficult to learn?

Anyway, one of these developers is a visual thinker such as myself. We prefer communication through diagrams rather than words. While explaining Scala I've been drawing some pictures to illustrate some of the more interesting concepts. She suggested that I should publish these and we had the idea that an animation would be fun.

Here's my first animated post showing the concepts for Map, Flatten and FlatMap. It's a pretty amiturish stab at animation, but hopefully it explains the concepts while also adding a bit of humour along the way.

If you are interested here's the Scala code for the concepts covered in the animation:

case class World[+A](revolvesAround: A) {

  def map[B](f: A => B): World[B] = World(f(revolvesAround)) 

  def flatten[B](implicit ev: A <:< World[B]): World[B] = World(revolvesAround.revolvesAround)

  def flatMap[B](f: A => World[B]) = World(f(revolvesAround).revolvesAround)

  def get: A = revolvesAround
}

def lift[A](entity: A): World[A] = World(entity)

case class Person(name: String) {
  def smite = DeadBody(name)
}
case class DeadBody(usedToBe: String)
case class Insect(species: String)

def kill(p: Person): DeadBody = p.smite
def reincarnate(b: DeadBody): World[Insect] = World(Insect("ant"))

val dave: Person = Person("Dave")
val antDave: Insect = lift(dave) map kill flatMap reincarnate get 

println(antDave)

Wednesday, 14 March 2012

Done and Dusted

In a recent retrospective our team dug out our Definition of Done and gave it a good review. We ended up removing a number of unnecessary items. This got me thinking about what is Done?

In previous places that I have worked the Definition of Done was a long check-list of things that had to have been completed and verified. Is this the best that a philosophy that claims to value "individuals and interactions over processes and tools" can come up with? This got me thinking...

Isn’t Done actually just the state when the team agree that the story is complete? Why should it need to be any more than that?

Done: The state when all team members agree that the story is complete

We could expand this slightly to define the roles that the different team members play in this definition:

  • Developers agree that the code is complete, tested and refactored to an acceptable standard
  • QAs agree that the acceptance criteria for the story has been met, that the functionality works and that they have not found any defects introduced by the new feature
  • BAs and POs agree that the feature meets the business objectives for the story
  • (we could also add points for DBAs, Ops etc, as needed)

So, if we accept that the team can decide for themselves when something is Done, why do we end up with long checklists, often including a whole range of metrics? It could be because 'management' aren’t able to release their need to monitor everything and produce nice reports. However, it’s more likely a matter of trust.

Teams are just not trusted to self-organise and make good decisions. If the team say that a story is Done then for many managers there is an immediate suspicion that the team is not telling the truth and is trying to cut corners. They then require that there is concrete proof that this is not the case. This is despite the fact that a group of professionals all happen to agree that it is Done. It just doesn’t make sense.

If the team has never released a story as Done when it clearly wasn’t then why doubt they will suddenly stop doing this in the future? Show trust in the team that is delivering completed work.

If the team are marking stories as Done when they clearly aren’t then there is a much more serious problem with the team and they probably either need replacing, mentoring or a firm guiding hand. Adding a requirement to satisfy a long checklist and provide associated metrics will never improve a bad team.

As with many things agile, it’s not about the process: it’s about changing attitudes; it’s about trusting teams to deliver the best solutions possible; it’s about not bogging the teams down with unnecessary controls.

Monday, 13 February 2012

Sometimes you just have to be lazy...

...or, solving problems with lazy evaluation.

I’ve recently been working on a simple template system, which also has the potential to grow into a full-blown document generation system. The basic principle is that a document is composed of a number of fragments. Each fragment is called in turn to generate its piece of the document and these pieces are concatenated together to product the final result.

My first implementation was a super simple one. We just fold over the list of fragments, accumulating the output from each into a string builder. Each fragment may alternatively generate an error, which we collect along the way as well. At the end of the fold we return either the string document or a list of errors.

  def render(fragments: List[Fragment]): Either[List[String], String] = {
    fragments.foldLeft(RenderResult()) { (acc, fragment) =>
      fragment.generate fold (
        { err => acc.copy(errors = err :: acc.errors) },
        { text => acc.doc.append(text); acc }     
      )
    }.toEither
  }

  case class RenderResult(doc: StringBuilder = new StringBuilder(), errors: List[String] = List()) {
    def toEither: Either[List[String], String] = ...
  }

Now, this works great for documents of only a few kilobytes in size. However, as the documents grow to multi-megabyte sizes this approach becomes infeasible. The amount of memory required to hold the document being generated becomes huge. Multiple concurrent generations become impossible. We need a better solution.

The typical suggestion at this point is to stream output from each fragment straight into an output writer of some form rather than collect and build a string.

  def render(fragments: List[Fragment], writer: Writer): Option[List[String]] = {
    fragments.foldLeft(List[String]()) { (acc, fragment) =>
      fragment.generate fold (
        { err => err :: acc.errors },
        { text => writer.write(text); acc }     
      )
    }.getErrors
  }

Unfortunately this doesn’t work here because of the possibility that fragments may generate errors. We don’t want to stream out partially complete documents. We could output to a temporary file and then copy this to the ultimate output on success, but this seems like a less than ideal solution.

A common approach in an imperative style would be to first run a check phase to see if there are any errors and then run a second render phase to produce the output.

  def check(fragments: List[Fragment]): List[String] = 
    fragmens.foldLeft(List[String]()) { (acc, fragment) =>
      fragment.checkForErrors map (_ :: acc) getOrElse acc
    }

  def render(fragments: List[Fragment], writer: Writer): Unit = 
    fragments foreach { fragment => writer.write(fragment.render) }

However, in this case the building of fragments might be quite costly and we don’t want to have to process them once to see if there is an error and then again to render.

However, as we are using Scala and programming in a functional style, we have an alternative approach. Instead of returning a string or writing direct to a writer we can return a lazily evaluated function. This function can encapsulate all the side-effect generating logic of doing the actual write - which is also nice from the point of view of being able to reason about our rendering code.

  def render(fragments: List[Fragment]): Either[List[String], (Writer) => Unit] = {
    fragments.foldLeft(RenderResult()) { (acc, element) =>
      fragment.generate fold (
        { err => acc.copy(errors = err :: acc.errors) },
        { func => acc.copy(fragmentFuncs = func :: fragmentFuncs }     
      )
    }.toEither
  }

  case class RenderResult(fragmentFuncs: List[() => String] = List(), errors: List[String] = List()) {
    def toEither: Either[List[String], List[(Writer) => Unit] = {
      ...
      (writer: Writer) => fragmentFuncs foreach { f => writer.write(f()) }
      ...
    }
}

The way this works is that the render method folds over each fragment, asking the fragment to do enough work to be certain that no errors will occur (e.g. validate data, take a temporary copy of any volatile values and so on). However, it doesn’t need to do the actual string generation. Instead, each fragment returns a function that will be called at some time in the future (or an error). The render code then accumulates all these small functions into one larger function and returns this.

The returned function, when called, takes a writer and iterates over all the small string generating functions, writing the output of each straight into the writer. Job done and in a mere fraction of the memory required by the original solution.

In cases where you need to generate large amounts of data or where you want to delay side effecting execution to a well-defined location in your code then lazy evaluation is a great approach to follow.

NOTE: All above code example are significantly simplified from the real code in order to demonstrate the approach.

Saturday, 4 February 2012

Some TDD Advice

I’ve been doing some interviewing of new developers. Our company approach is to undertake a small pairing exercise with the candidate. We pick a relatively simple code kata and ask them to complete this. They drive the keyboard and do most of the work, while we, the interviewer, observe, ask questions, help point out obvious problems and provide assistance if necessary.

The only main requirement for these pairing sessions is that all work is done using a Test-Driven Development (TDD) approach. We do this for two reasons:

  1. We TDD all our code and therefore want to be comfortable that the candidate can work in this way.
  2. Building a system gradually using tests gives us a good insight into how the person thinks, how they go about solving problems and how they evolve their code.

Unfortunately, we are finding that many of the candidates are struggling with these sessions. They tend to start out strongly and then a few tests into the problem they become bogged down or stuck on furthering the solution.

I found this slightly puzzling, so I decided to undertake one of the katas following the exact same approach as we expect the candidates to follow. I picked a kata from one of the other devs that I had not seen before, so as to not prejudice my result. I started small with a simple test and gradually evolved the solution one test at a time. I was able to complete the kata easily within the allotted interview time and the code I produced was (IMHO) very clean and well structured.

While doing this I noticed one major difference between my approach and that of the people I have been interviewing. I follow a cycle as follows: write the test -> implement the code to make the test pass -> refactor the code to make it better -> move on to the next test. The devs that I have been interviewing who struggle miss out one of these key steps. Namely, they do: write the test -> implement the code to make the test pass -> move on to the next test. They miss the refactoring step at the end of each cycle.

What this tends to result in is that they easily build a first pass at some code to make a test pass. Then they build the code to pass the next test on top of this. This works for three of four iterations - but by this time their code is becoming quite complex and messy. Then as the requirements get more complex they struggle to change their code, understand how it actually works and how to change it to pass the next test.

At this point they either become so bogged down that they fail to make any progress, or they start a massive refactoring exercise that breaks loads of code and most of their previously working tests. Either way, they pretty much fail the interview at this point.

By adding a small refactor step into every cycle we ensure that the code we implement to pass the next test is always built on a solid foundation of well structured, clean code from the previous one. Doing this would make so much difference to those interview candidates.

Let’s finish by looking at an example of the difference this approach makes:

In this example we are working on a kata to build a speaking clock. For example: 0:00 speaks 'midnight', 15:30 speaks 'half past three', 11:45 speaks 'quarter to twelve' and so on. After writing code to pass the first couple of tests we have something like:

class SpeakingClock {

  def speak(time: DateTime): String ={
    val hour = time.hourOfDay().get()
    val minute = time.minuteOfDay().get()

    if ( hour == 0 && minute == 0 ) "midnight"
    else if ( hour == 12 && minute == 0) "noon"
    else "unknown"
  } 
}

The next tests would be to cover a case for speaking the o'clock and half past times. Our candidate would most likelky go on to just add to the above code, something like:

class SpeakingClock {

  def speak(time: DateTime): String ={
    val hour = time.hourOfDay().get()
    val minute = time.minuteOfDay().get()

    if ( hour == 0 && minute == 0 ) "midnight"
    else if ( hour == 12 && minute == 0) "noon"
    else if ( minute == 0 ) hourToText(hour) + " o'clock"
    else if ( minute == 30 ) "half past" + hourToText(hour)
    else "unknown"
  } 
}

Then we start adding tests covering quarter past and quarter to. At this point the algorithm starts to get more complex and the if/else block starts growing out of control. Refactoring this large if/else block becomes a major exercise under interview conditions and it is at this point that the solution invariably goes wrong and the candidate starts to struggle.

Now, by doing some refactoring after each step, we can change the first piece of code to something like:

class SpeakingClock {
  
  private val FixedTimes = Map[(Int, Int), String]((0, 0) -> "midnight",
                               (12, 0) -> "noon")

  def speak(time: DateTime): String = {
    val hour = time.hourOfDay().get()
    val minute = time.minuteOfHour().get()

    FixedTimes get ((hour, minute)) getOrElse {
      "unknown"
    }
  } 
}

Now we don't need to worry about the first case any more. It's nice and clean and out of the way. Also, if there are any new fixed times (say 17:00 is 'dinnertime') then we can easily add these to the solution with minimal additional code. Next, we add in the support for speaking the o'clock times:

class SpeakingClock {
  
  private val FixedTimes = Map((0, 0) -> "midnight",
                               (12, 0) -> "noon")

  private val HourNames = "twelve" :: "one" :: "two" :: "three" :: "four" :: "five" :: "six" ::
                          "seven" :: "eight" :: "nine" :: "ten" :: "eleven" :: Nil

  private val HoursToNames = (0 to 11) zip HourNames toMap

  def speak(time: DateTime): String = {
    val hour = time.hourOfDay().get()
    val minute = time.minuteOfHour().get()

    FixedTimes get ((hour, minute)) getOrElse {
      val adjustedHour = if ( hour > 11 ) hour - 12 else hour
      val hourName = HoursToNames(adjustedHour)
      "%s o'clock".format(hourName)
    }
  } 
}

Then we again go through the refactoring step and end up with something like:

class SpeakingClock {
  
  private val FixedTimes = Map((0, 0) -> "midnight",
                               (12, 0) -> "noon")

  private val HourNames = "twelve" :: "one" :: "two" :: "three" :: "four" :: "five" :: "six" ::
                          "seven" :: "eight" :: "nine" :: "ten" :: "eleven" :: Nil

  private val HoursToNames = (0 to 11) zip HourNames toMap

  def speak(time: DateTime): String = {
    val hour = time.hourOfDay().get()
    val minute = time.minuteOfHour().get()

    FixedTimes get ((hour, minute)) getOrElse {
      "%s o'clock".format(nameForHour(hour))
    }
  } 

  private def nameForHour(hour: Int) = {
    val adjustedHour = if ( hour > 11 ) hour - 12 else hour
    HoursToNames(adjustedHour)
  }
}

By pulling out the nameForHour method we leave code that is much better structured and cleaner. It's then so much easier to see what changes to make in order to pass the next test. By growing code in this way we end up with a working solution that is also always in a clean and tidy state.

So, the moral of this story is that if you are doing TDD, in an interview or not, don't forget to refactor after each test passes to ensure your code is well written before moving on to the next test.

Wednesday, 11 January 2012

Thoughts on the UK Government's New Approach To ICT

The UK Government today announced a new approach to teaching computing in schools. The current sylabus is based on teaching how to use applications like Word, Excel and PowerPoint. This is, quite rightly, seen as highly boring and unecessary for today's computer literate pupils. The new strategy is to switch to a more 'computer science' based sylabus. This would include more on how computers actually work and would also teach programming. The Government's aim is to create a new generation of highly skilled graduates with excellent computing skills.

While this new approach sounds very interesting and laudable, I have some serious doubts about whether it will actually work. My concern can been seen if you break down the types of pupils that will be studying this new computing sylabus into three broad categories:

The Uninterested

There is likely to be a (fairly large?) category of pupils who, like with any subject at school, aren't really interested. These will be the pupils for whom a computer is about Facebook, playing games and downloading music and videos. It's pretty likely the these pupils won't really be interested in programming and will thus find it very challenging and probably quite boring. However, the sylabus won't be allowed to leave them behind, so it will have to run at a fairly slow pace in order to keep them on board.

The Swayable

This is the category of pupils who might have some interest in computing but whom have never really attempted any programming before. It's quite likely that a good portion of these might find the subject really enjoyable and will thus become the future genration of computer skilled graduates that the Government is hoping for. Unfortunately I think this category is MUCH smaller that those advocating the new sylabus believe that it is. In any given class I would think that it will be just one or two pupils - hardly a revolution.

The Hooked

Finally, there will be a group of pupils who will really love computing and programming. They will be the people likely to become the hard-core programmers of the future. Unfortunately, it's probably likely that these pupils will already have computing as a hobby at home. They will likely already be building websites and writing software. Even if they aren't at the start, then they soon will be once they get hooked. However, a sylabus created to cater for the slower pace of the uninterested will likely soon become very boring for those able and interested pupils. My fear would therefore be that they become unmotivated, unchallenged and bored by doing work that is way below their ability. This could then have a negative effect and turn those potential developers away from computing forever.

In fact, I know this will likely be the case as it happened to me. I was at school 28 years ago when computing first started being introduced. I got hooked early and my (very supportive parents) ensured that I was one of the first people in school to have a home computer. I taught myself to program at age 10 and never looked back. Unfortunately I found I was always way ahead of the curve when it came to computer studies lesson, talking to teachers about computers and so on. I got bored through not being stimulated. I had a few years of just playing games and word processing essays. I even toyed with a career in accountancy! Luckily I rediscovered my interest, changed tact, and went on to study Computer Science at University. It could have turned out so different.


My real fear of the proposed strategy is that rather than producing a new generation of highly skilled computer graduates it will instead produce a large number of moderately skilled people while turing the highly skilled pupils with the best potential away from the subject. Hey, let's build a large army of mediocracy!

So, what do I think needs to be done. I agree with the general goal of the new strategy, but I think it needs to be covered by a dual-sylabus approach. A sylabus for the uninterested and swayable that teaches general computer skills, applications, effective use of the Internet, basic programming and so on. To be taught by ordinary teachers with standard levels of IT skills. There also needs to be an Advanced Computing sylabus for the hooked and highly able that teaches programming and computing to a much greater depth. It would need to be taught by highly skilled teachers with a heavy input from universities and industry. Its aim would be to stretch the most able who could become the highly skilled people that our industry really needs.