Unit Testing with MongoDB

Back in June I presented at the London Scala User Group (video here) on our experiences on a recent Scala project that used MongoDB.

One of the questions I was asked was how we go about mocking MongoDB for testing purposes (especially unit testing)? My answer was that we don’t! This raised a couple of eyebrows in the audience, but we’ve found this works just fine for us.

In this blog post I will explore our thoughts and reasoning in a bit more detail.

How do we test with MongoDB?

I’ll be honest: for the majority of our tests (even the unit ones) we just run against a local MongoDB instance. When we are testing repository classes we write tests against a real repository wired up against a real MongoDB instance. We assert against the contents in the MongoDB collections. In some cases, we even test service classes or even controllers with a live repository backend that writes to and retrieves from a real MongoDB instance.

“But you can’t do that!”, I hear you cry. Traditional thinking says that writing tests directly against the database will be too slow. We’ve actually found MongoDB (especially when running locally) to be so blindingly fast that we don’t need to consider doing anything else. For most updates and queries we tend to get sub-millisecond responses.

It’s also a hell of a lot easier and less fragile to back tests with real repository instances than it is to create and configure a load of mocks or strange database testing frameworks.

I’m pragmatic. If using a real database backend was making our tests run too slow or causing them to be too complex then I’d happily consider alternatives. Our experience with MongoDB is that it is both fast enough and that it’s query syntax is succinct enough that we don’t need to bother with those alternatives.

Do we use mocks at all for the database then?

Actually, I rarely use mocks in any of my Scala project now days. I’ve switched much more towards a style of isolating my calls to dependencies and then using mixin traits to override specific behavior for testing. (I’ll write a blog post about this approach soon).

From time to time we do need to write tests that make a repository dependency exhibit a specific behavior which is difficult to emulate against an actual database. In these cases we just use a custom test trait mixed into the repository in order to override methods to exhibit the behavior that we require. Keeps it simple and makes the tests much more readable and much less fragile than using mocks or other frameworks.

Selecting The Correct Test Scenarios and Data

Yesterday I happened to be reviewing some integration tests with my excellent tester colleague. These tests were defined using a set of highly simplified synthetic data selected for the purpose of making the tests pass. We both agreed that tests should actually be written using a mixture of realistic and synthetic data selected for the purpose of trying to make the tests fail! As developers we are often too keen to write tests that prove that our code is good rather than writing tests to find the problems in our code.

This exercise bought to mind a situation I had a few years ago with some code provided by an offshore development team. It was a very simple piece of code and came with some tests to show that it worked. However, it exhibited a single fundamental flaw that the synthetic data designed to make the tests pass would never have found. I've recalled the problem below (somewhat simplified), and I've update the code from Java/JUnit to Scala and Cucumber (because quite frankly I prefer these languages!).

The Requirement

The specific requirement for the system being built was that it would collect a user's date of birth during the sign-in process. If the user was under 17 then they would not be allowed to use the system. If they were 17 or over then they would be taken to the main menu. The particular piece of code in question was the age validation algorithm.

The Supplied Solution

The initial solution to this problem consisted of the following piece of code:

class AgeValidation {
  private val AuthorisedAge = 17

  def isAuthorisedAge(user: User) = {
    val cal = Calendar.getInstance
    val currentYear = cal.get(Calendar.YEAR);

    cal.setTime(user.dateOfBirth)
    val birthYear = cal.get(Calendar.YEAR);

    (currentYear - birthYear) >= AuthorisedAge
  }
}

Tests were supplied that covered two specific scenarios:

Feature: The application should ensure only users over 17 are allowed access
    As a Application Security
    I want only users aged over 17 to access the system

    Scenario: User aged over 17 can access the system
        Given a user aged over 17
        When the age check is applied
        Then they are authorised to continue

    Scenario: User aged under 17 cannot access the system
        Given a user aged under 17
        When the age check is applied
        Then they are not authorised to continue

And the implementation of these tests was as follows:

class AgeValidationSteps extends ScalaDsl with EN with ShouldMatchers {
  private var user: User = _
  private var valid: Boolean = _
  private val dateFormatter = new SimpleDateFormat("dd/MM/yyyy");

  Given("""^a user aged over 17$""") {
    user = new User("test", dateFormatter.parse("01/01/1970"))
  }

  Given("""^a user aged under 17$""") {
    user = new User("test", dateFormatter.parse("01/01/2010"))
  }

  When("""^the age check is applied$""") {
    val validator = new AgeValidation
    valid = validator.isAuthorisedAge(user)
  }

  Then("""^they are authorised to continue$""") {
    valid should be (true)
  }

  Then("""^they are not authorised to continue$""") {
    valid should be (false)
  }
}

Extra credit if you can spot the problem with this implementation and why the synthetic test data would have never found it before you read on!

The Problem With This Code

Did you spot it? Here's a hint: what if my 17th birthday is not until tomorrow? Yes, that's right, the code has a fundamental defect in that anyone who is 17 in the current year but who has not yet had their birthday is granted access to the system even though they are still 16. And yes, this was the actual algorithm submitted to us by the offshore developers!

Quite clearly, the synthetic test data used in the supplied tests was selected to ensure the tests passed, but not to find the problems in the code. Let's correct the feature spec to base it on values that actually might find something:

Feature: The application should ensure only users over 17 are allowed access
    As a Application Security
    I want only users aged over 17 to access the system

    Scenario: User who was 17 last year can access the system
        Given a user whose 17th birthday was last year
        When the age check is applied
        Then they are authorised to continue

    Scenario: User who is 17 next year cannot access the system
        Given a user whose 17th birthday is next year
        When the age check is applied
        Then they are not authorised to continue

    Scenario: User who was 17 yesterday can access the system
        Given a user whose 17th birthday was yesterday
        When the age check is applied
        Then they are authorised to continue

    Scenario: User who is 17 today can access the system
        Given a user whose 17th birthday is today
        When the age check is applied
        Then they are authorised to continue

    Scenario: User who is 17 tomorrow cannot access the system
        Given a user whose 17th birthday is tomorrow
        When the age check is applied
        Then they are not authorised to continue

And the associated test steps:

class AgeValidationSteps extends ScalaDsl with EN with ShouldMatchers {

  private var user: User = _
  private var valid: Boolean = _
  private val dateFormatter = new SimpleDateFormat("dd/MM/yyyy");

  Given("""^a user whose 17th birthday was last year$""") {
    val cal = Calendar.getInstance
    cal.add(Calendar.YEAR, -18)
    user = new User("test", cal.getTime)
  }

  Given("""^a user whose 17th birthday is next year$""") {
    val cal = Calendar.getInstance
    cal.add(Calendar.YEAR, -16)
    user = new User("test", cal.getTime)
  }

  Given("""^a user whose 17th birthday was yesterday$""") {
    val cal = Calendar.getInstance
    cal.add(Calendar.YEAR, -17)
    cal.add(Calendar.DATE, -1)
    user = new User("test", cal.getTime)
  }

  Given("""^a user whose 17th birthday is today$""") {
    val cal = Calendar.getInstance
    cal.add(Calendar.YEAR, -17)
    user = new User("test", cal.getTime)
  }

  Given("""^a user whose 17th birthday is tomorrow$""") {
    val cal = Calendar.getInstance
    cal.add(Calendar.YEAR, -17)
    cal.add(Calendar.DATE, 1)
    user = new User("test", cal.getTime)
  }

  When("""^the age check is applied$""") {
    val validator = new AgeValidation
    valid = validator.isAuthorisedAge(user)
  }

  Then("""^they are authorised to continue$""") {
    valid should be (true)
  }

  Then("""^they are not authorised to continue$""") {
    valid should be (false)
  }
}

Run this set of tests against the existing code and the final scenario 'User who is 17 tomorrow cannot access the system' fails. Our tests have actually done something useful: they have proved that our code doesn't work. Test data selected with the specific goal of trying to find problems with the code has done its job. I'll leave it as an exercise for the reader to fix the implementation so that it all works correctly!

Further Improvements

Even with our improved set of test data there are still some significant improvements that can be done to these tests by selecting some additional scenarios and test data values that might find problems:

• Testing the algorithm works with data boundary cases such as first/last day of the year, leap years and so on
• What happens when the user enters a date of birth in the future, does the code still work?
• What about if the person is over 100? Are we dealing with 4 digit or 2 digit years?
• Is this an international system? If so, then what about timezones? A person in a timezone east of us may already by 17, but may not be 17 in our timezone.

Apply The Same Rules To Our Tests

We can also apply the same thinking to our tests. For example, if we run our improved tests against the original code but do so on the first day of the year then they will all pass. The reason being that subtracting one day from the current date would push the birth date into the previous year. We therefore need to think not only about how our synthetic test data might find problems in our code by also how they may cause problems in our tests. In this case, solving this problem is just a case of improving our code to allow our tests to define the current date rather than using whatever the current system date is:

class AgeValidation(baseDate: () => Date) {
  ...
}
object AgeValidation {
  def apply = new AgeValidation(() => new Date)
  def apply(baseDate: String) = new AgeValidation(() => new SimpleDateFormat("dd/MM/yyyy").parse(baseDate))
}

// And in the tests
val validator = AgeValidation("15/07/2010")

Our test code now has the ability to control exactly what current date is used as a basis for the comparison, so it can experiment with a wider range of possible scenarios.

Does This Change Our TDD Approach

All of the above got me thinking about how this impacts on the Test-Driven Development approach. In this model we first write a test to validate the expected behaviour of our code. We then write minimal code so that the test fails. Next we write the code to make the test pass. Finally we refactor to make the code better without breaking the tests. Most developers will approach this by picking some synthetic values that are designed to prove that the 'happy path' through the code works when they have written it. Perhaps we should be focusing more on TDD where the tests we start out writing are for the scenarios that are more likely to identify problems - that way the code we write to make these pass has to deal with these cases. Certainly something I will be experimenting with in my next TDD session.

My New Appreciation of Testing

Recently I have had the benefit of working with a really good tester on an agile project. This person has consistently challenged both my terminology and approach to testing. After working with him I have come to a completely new conceptual model of automated testing (checking) vs manual (and exploratory) testing, how they are very different and how they complement each other. This is the story of what I have learnt.

A Developer Model of Testing

I've been writing automated tests as part of my development process for as long as I can remember. These used to be tests written after the code to check that it worked, then I started doing TDD and using the tests to drive the design while validating that it also worked as expected. Recently I've also added BDD into the mix as a way of driving and testing requirements. However, all this time my mental model of testing has been somewhat fixed and, as I now realise, focused solely from a development perspective. Let me try to describe this view.

As a developer, my goal is simple, write some code such that:

• given a know starting state and;
• applying a known sequence of operations;
• with a defined set of input values;
• then I get an expected and valid output state

My typical approach would be to write tests that:

1. Set up some fixture code representing a state that I want to start from
2. Define a test that applies an operation with valid input values
3. Define some assertions that should be true about the state after the operation has completed
4. Implement and refactor the code until the assertions pass

I'd then go on to create some additional variations of the above test with alternative applications of the operation, adjusting the implementation code so that each test's assertions pass. Typically the alternative tests would cover a mix of:

• Apply an operation with different valid input values
• Apply an operation with invalid input values
• Apply an operation with boundary input values
• Apply an operation with non-supplied input values
• Apply an operation with potentially harmful input values
• Apply an operation forcing different failure cases in underlying code, subsystems or boundaries
• Apply operations involving concurrency or synchronisation

As a conscientious developer I'd be trying to write automated tests that cover the widest range of scenarios so that I could be confident my code would work as expected in all situations. When describing my progress in a stand-up I'd be using phrases such as:

• "The code is done and is working because all the tests pass"
• "The tests show we've fixed all the defects"
• "We've made the change and the tests show that we haven't broken anything"

From a developer point of view the functionality would then be handed over to a QA team member to check and perhaps for them to write some higher level test against to ensure the code meets the functional requirement (i.e. given a good set of data and a good set of inputs then the correct result is observed). Code complete, story done, into production, next please!

The Awkward Tester

The above was pretty much my approach to testing and is pretty typical of that taken by most developers I think. However, when working with this excellent tester on our team he started to question the statements that we made about our code. For example:

• The code is done and working because all the tests pass: "Your tests show that your code makes the tests pass, but they don't prove that it works"
• The tests show we've fixed all the defects: "The tests actually just show that you've fixed the specific identified defect case(s), all the other unknown ones are still there"
• We've made the change and the tests show that we haven't broken anything: "No, the tests show that you haven't broken the paths covered by the tests, there's probably other things that are broken though"

My initial reaction was to just laugh this off, put the tester down as an awkward, pedantic sod and carry on as before. However, based on his well deserved reputation and previous work record, I trusted and respected the skills of this particular person. This sowed some seeds of doubt in my mind: was it just the language us developers were using that he had problems with, or was he really talking about some fundamental wider view of testing that we, as developers, were all missing?

My New Developer Model of Testing

After some serious contemplation, introspection (and a number of very long lunchtime walks) I have come to realise that what I have been considering as 'comprehensive' automated testing is really just a small part of of a much wider testing world. By focusing so much on these automated test suites I had become blind to the wider implications of my code quality. Our excellent tester friend calls what we do automated 'checks', not tests, and I think he might be right! What we are building are pieces of non-deployed code that check that our real code does what we are expecting it to do in a certain set of scenarios. Testing is so much more. Let me explain my current thinking...

In automated checks we first set up fixtures that represents known starting states for our application, component or class. For all but the most trivial examples these states will only be a small subset of all of the possible states that my application could be in. My checks then pick an operation or set of operations (i.e. feature) to evaluate against this state. Again, for anything but the most trivial single function cases the operations we pick and the order they are called will represent just a small subset of all possible operations and combinations of those operations.

Next we look at input values and their combinations and the same subset rules apply. This is also true for failure scenarios that we might check. Finally, we assert that our code leaves the application in a consistent state, with those states again being just a small subset of the possible states my application could be in. We then clear down our test fixtures and move onto the next test. This can be clearly shown in the following diagram:

Now, our tester has a different view. He is trying to identify possible places where my application might break or exhibit a defect. He is therefore using his skills, knowledge and experience to pick combinations of specific states, operations, input values and failure scenarios that are both likely to occur and most likely to find the defects in the application. These may be inside the subsets used by the automated checks, but can also be outside, from the wider set of possibilities.

Additionally, the tester has another weapon in their arsenal in that they can use the output state of one set of tests as the input state into the next - something that it's not so easy to do in automated tests that might be run in a different order or individually. We can see the tester's approach to testing in the following diagram:

It is clear to me now that the two approaches are both necessary and complementary. Checking has far less value if testing is not also applied. Testing would also be far more expansive and time consuming if checking was not in place to find and eliminate a certain percentage of defects at development time.

The Role of the Tester

So, what is the role of the tester? Most developers would tend to assume that the tester's job is to prove that their code works (and I will even admit that this was probably the way I used to think). Go on admit it developers, how many times have we cursed the tester under our breath when they have found a critical flaw in out code a few days/hours before it's due to be released? Come on, we know we've all done it!

Turns out we were wrong! The role of a tester is to prove that our code doesn't work and to prove it before the code gets into production and is broken by a real user or customer. Why should a tester be trying to prove that my code works? Isn't that why I've invested so much time and effort into building my automated suite of checks - to give me confidence that my code works under the situations that I have identified. What I really want to to find out is where my code doesn't work so that I can identify the omissions from my suite of checks and fix the broken code paths. This is what a good tester should be doing for me.

First and foremost a good tester should be using his skills to find scenarios whereby my code fails and leaves the application in a broken and inconsistent state. This should be the Holy Grail for a tester and they should be rewarded for finding these problems. Who wants a production outage caused by a defect in their code? Not me, that's for sure. Next they should be finding cases where the application fails non-cleanly but is not broken and can continue to function. If they can do either of these things then they are worth their weight in salary/day rate! A tester that can only find the places where the code fails cleanly or does not fail at all is really not of much value: my automated checks already proved this! I know I will certainly be valuing my testers much more than I used to. And you should too.

So how as developers do we aid out able testers in finding the defects that are in our code? We have to help them understand how the bits of the system fit together; we have to explain the approaches we have used to determine our automated checking scenarios; we have to help them understand where the failure points might be and; we have make sure they know the expected usage patterns that we have coded to. Given all this information or excellent tester should be able to find the gaps, the failure scenarios and the unexpected usages that expose the defects that our automated checks do not. When they find them we should be glad, and keen to find out how they managed to get our lovingly created code to break. We then fix the problem, add checks for those paths that we missed and learn the lessons so that we don't repeat the same defect creating pattern in any future code we write.

The Stand-up Revisited

So, given this new understanding of testing, testers and automated checks, what should the progress descriptions in the stand-up really be:

• "The code implementation is complete and my checks for the main scenarios all pass. Mr Tester, can you find the defects please?"
• "I've fixed the specific path(s) which I identified as being a cause of the reported defect and added automated check. We need to test to find the other paths that might be causing the same problem."
• "The existing checks all run against the new code. We need to test to see which other paths or aspects of features have been broken by these changes."

See the difference? No assumptions that just because an automated suite of checks has completed successfully that the code either works, is free of defects or that its addition has not broken the application somewhere. The automated checks can reduce the probability of bugs or issues escaping the development process and can give me more confidence in my code. However, some defects and problems are still likely to be present and my application will still break under certain conditions that my checks don't cover. I now say: "...over to you Mr. Tester I'll explain what I've done and you can help me find where it is broken!".

Sprinting with BDD

After some initial experiments into doing BDD using Cucumber our team has decided to adopt a full BDD and TDD approach to our current sprint. It's slightly experimental at the moment as we work out the best ways of working. Once we are done we will retrospect and see if we want to adopt the approach going forward. Initial feelings from our first couple of days are that it is going well. Here's what we have done so far.....

Sprint Planning

After deciding to give BDD and TDD a try for the sprint, the first thing was the sprint planning. Fortunately for this sprint we have a story that is pretty well understood, but sufficiently challenging to be of value in evaluating the approach. The story basically consists of taking some existing data, retrieving additional data from a new remote system and delivering the combination of the two to a remote destination (basic decoration of existing content).

Our sprint planning consisted of two parts:

1. a high level technical discussion of the story and agreement of the general architectural structure and approach
2. the writing of Cucumber 'Feature' files containing all of the acceptance scenarios for the story

These two parts went well and we created a good set of acceptance scenarios, but we then found ourself struggling to get our heads around how to go about implementing them. Our reasoning was that rather than writing out individual technical tasks, we would just create one card for each scenario. These scenario cards would then form our sprint backlog.

However, the first scenario was a 'happy path' one and we were thinking that in order to implement it we would have to build the entire solution; the remaining scenarios would then just be tiny additions. This didn't feel right in that we would then have one scenario stuck as 'in progress' for days and then all of them moving within a few hours right at the end of the sprint. Our next thought was to therefore break down this scenario into a number of technical tasks. As it turns out, this wasn't the correct approach and it soon became clear that by doing this we were missing the benefits of BDD and TDD: we were defining the detailed solution in the planning session, rather than letting it emerge from the implementation of the tests.

We decided to take a break and think over our approach. It was then that it came to us that we were still thinking in a non-TDD mindset. To pass the first scenario, we didn't need to implement the whole solution, just enough to make the BDD test pass. As we then moved on to the next scenario we could implement more of the functionality, add depth and so on. With this in mind we decided to start the sprint and inspect and adapt as we went.

Setup and Configuration

The first step in the sprint was to add Cucumber support to our project. Given that we are using BDD for acceptance tests, we decided to add this support into our functional test suite. Unit level TDD will continue to be done using our existing testing tools (JUnit, Hamcrest and mockito).

Our functional tests are written in Java and are built and executed using Maven. We therefore added a dependency to cuke4duke, which runs Cucumber on top of the JRuby platform. This setup included adding a dependency on the maven-cuke4duke-plugin, which runs the Cucumber 'Features' within a maven test phase.

Aside: Why did we pick Cucumber (which is a Ruby framework) when we are doing a Java project? There were a number of reasons, but the main ones were:

• We liked the HTML report format
• A number of the team are already using Cucumber on external projects (in Ruby and Scala) so there was an existing familiarity
• It seems to keep out of the way and be less invasive than many of the Java based alternatives

With this set up, we added the Features file that we built during spring planning into the appropriate features directory and created step definitions with pending methods for each of the Given/When/Then cases.

The First Scenario

The first scenario was to set up some existing content and some additional content, decorate the existing content with the additional content and then verify that the content delivered at the destination had been correctly decorated. The steps for Cucumber were written accordingly and the assertions added the 'Then' case. Now it was time to implement.

This was where our TDD thinking really kicked in. Previously we would have started building the whole solution. Instead we just implemented the elements of the destination required to allow us to verify the decorated content. Given that we were only testing with one piece of content we just hardcoded to return a fixed set of additional data. Test passed.

Now, we know that this is not the full solution and that we have plenty more work to do, but we have a green on our test report. We have 8 scenarios to complete and the story is not done until they all pass, so what's wrong with this approach? Our next scenario tests with a different set of content and additional data, so this is broken by the hardcoding approach, so we can then work on the next step of delivering different sets of additional data to the destination. In this case we will still probably hardcode the data that is delivered based on the content, but progress will be made. Later scenarios will then add the calls to the new remote system to get actual data. By the end of the scenarios we will have built the entire solution driven solely by tests.

Areas of Uncertainty

Even though things are going well, we still have a number of areas of uncertainty that we need to resolve by then end of the sprint. These include:

• Whether we can successfully complete the story in this way
• Whether this approach impacts our productivity
• Whether the team members are able to adapt to working in this TDD style
• Whether the business will identify with the inherent value of this approach
• Whether we can get the BAs and the POs to start thinking of the stories as BDD scenarios

We hope to resolve these as the sprint progresses. I do think that initially productivity will be reduced due to the time it takes to learn a new approach and to become proficient in it. However, while productivity may be reduced for a short time, I believe that quality will go up due to the test-driven nature of the work and that we will be more certain of building the correct solution. I it my hope that the team members will see the value of this approach and will opt to continue with it for a few more sprints to allow them all time to become proficient in the approach. We will then be able to evaluate its success or failure more objectively.

To help the business understand the value of this approach (assuming of course that it is successful) I will plan to include a quick overview of the BDD approach in the Sprint Review next week. I will give a short presentation on the approach we took, show the scenarios that we created during planning, talk through the approach we took to implement them and show the Cucumber report that is produced. It will be interesting to see the reactions.

I'll post a follow-up at the end of the sprint to let you know how we got on.

The Best Way to Apply BDD

I am currently in the process of rewriting a simple e-commerce application for one of my customers. The new version is implemented using Scala and the excellent Lift framework and is a rewrite of a six year old struts/JSP version. The core application is largely identical but I am adding a wide range of new and improved administration functions.

Typically I build a simple, well understood application such as this using ordinary TDD principles to create unit tests and drive out the code. I'd also usually add some fairly basic UI tests using something like Selenium or even just HtmlUnit to validate the front-end. However, I like to use these sorts of projects to explore better ways of doing things. I've therefore decided to use a Behaviour Driven Development approach using Aslak Hellesøy's excellent Cucumber framework. (I'll post more details on installing and using Cucumber and cuke4duke with sbt once I get it all figured out!)

In this post I want to explore some of my thinking about exactly how and where to use BDD within the project and the places where I want to integrate the Cucumber features and scenarios.

What To Use BDD For

I am a strong advocate of the TDD approach, and I intend to follow this within this development for the low level details (i.e. unit and component tests). I'm therefore looking at using BDD as a way of capturing the higher-level business requirements and acceptance criteria. My aim is to use BDD and Cucumber to ensure that the application I am building actually does what the business expects it to. Lower level tests will still be written using ScalaTest to ensure the code that I am writing does what I expect it to do and also to drive out the architectural and implementation details.

Where To Integrate BDD

Given this positioning for BDD, where I have been struggling is the correct level in my application to target the Cucumber step definitions (step definitions are the code that actually executes the scenario and conditions described by the behaviour). The following sections describe the different options I have considered and advantages and disadvantages of each:

BDD at the User Interface - In this approach the step definitions are implemented to execute against the running web application. This is usually achieved using a technology such as Selenium or HtmlUnit. Each scenario loads web pages, clicks links, fills and submits forms and makes assertions about the HTML documents and associated resources that are returned.

Advanages:

• The behaviour is asserted across the complete application.
• The BDD approach can be used to drive the development of the UI as well as the application logic.

Disadvanages:

• You can only execute the feature steps against a built and deployed application - which may make development slower.
• A web-based UI is usually the least stable part of a web application and thus feature steps may be broken as the UI is tweaked, making it appear that behaviour is being lost. Step definitions may become quite fragile.
• Using BDD at this level may not fit well when user experience or visual designers are being used as this often results in rapid changing of the UI
• Generally testing at this level requires a lot of boilerplate code to be written - which makes wringing BDD step definitions more complex.
• Testing of web-based user interface may be better suited to a suite of UI tests that can focus just on verifying the UI display and interaction.
  

BDD at the Web Protocol - In this approach the step definitions are written to directly exercise the web protocol used to interface to the application. For example, the steps make HTTP calls to the application and assert that the returned results are correct and contain the expected content.

Advanages:

• Does not need to worry about the complexities of web pages, javascript and so on that are perhaps better tested in a dedicated suite of UI tests.
• Very well suited to RESTful or other web services that return structured data instead of HTML documents.

Disadvanages:

• You can still only execute the feature steps against a built and deployed application
• Building complex sequences of requests to simulate thinks such Ajax may be more complex than testing against the UI.
• Asserting against returned HTML content may be more complex than using assertions in a framework like Selenium at the UI layer.
• Building and testing at this layer requires that you also support a suite of UI tests.
• For a web application, the interface at the HTTP layer may change quite frequently, requiring frequent changes to the step definitions. This will be particularly common in an Ajax heavy application.

BDD at the Controllers - In this approach, we ignore the UI and view parts of the application. Instead we wire up the whole application from the controllers/snippets layer down. Step definitions are then written that invoke the controllers/snippets with pre-defined requests and assert that the response action and data that would be used to generate a view is correct.

Advanages:

• Features can be tested as part of the test phase of a build - no need to deploy.
• Testing at this level tends to be more stable and less fragile as the application emerges.
• It is usually much simpler to invoke and assert at a code level rather than a UI level

Disadvanages:

• A good set of testing features is required by the chosen web framework, so that requests and responses can be easily simulated and asserted against.
• Tests at this level tend to spend a significant portion of logic dedicated to the interactions with the UI rather than asserting against business rules.
• A suite of UI tests is required in order to test the UI and these will also generally exercise the same controllers/snippets in order to drive the UI.

BDD at the Services - In this final approach we test behaviour at the level directly below the controller/snippet. This is usually testing against service level code, but may also require writing some step definitions against domain level objects. As this level we are much more interested in testing the business behaviour of the application rather than the behaviour of how the application interacts with the user.

Advanages:

• Features can be tested as part of the test phase of a build - no need to deploy.
• Code at this level will usually be more stable than at the higher levels, thus making the tests less brittle.
• Step definitions at this level will usually be simpler to write and maintain than those written against the UI or controllers.
• We are testing against actual business rules rather than how a user might interact with the application.
• We require less testing specific tools and framework support.

Disadvanages:

• Testing at this level exercises less of the application flow than the tests at higher levels.
• A suite of tests is additionally required to validate the UI and the controllers to ensure that the user interaction with the system calls the services with the correct data in the correct order and correctly displays the results.
• The business is more likely expecting the behaviour of how they interact with the application to be verified.
• Developers must be VERY disciplined to ensure that no business rules, logic or behaviour is implemented in the controllers layer.

So, Where From Here?

After looking at all the possible places to use BDD and Cucumber, I have to draw the conclusion that there is no outright winner. It feels to me that the best mix is a combination of step definitions that test behaviour at the service layer (to verify business rules and behaviour) plus another set that test behaviour of the UI. By combining both of these approaches it should be possible to cover the application sufficiently to have confidence that its behaviour is correct.

This then leads me on to think about the best way to create the feature descriptions of the behaviour. Do we create a single description of the desired behaviour and then write two step definitions - one for the service layer and one for the UI? Or do we write a feature definition describing the required behaviour for the business services and then have an alternative behaviour specific for the user interaction with the system?

I'm still unsure of the correct answers to these questions. Anyone out there got any thoughts or experience? In the mean time I'm going to build my application trying both approaches and see which works out best.

Simple and Elegant

One of the most common traits that I come across in us software developers is the tendency to over-engineer our code. We try to future-proof our code and we try to make it deal with scenarios  that may never even be a requirement for the system we are building. Recognising when this is happening and being able to create a simpler and more elegant solution is something we should all be striving towards.

As an example, I'm currently using some of my personal time to build a solution for classifying large amounts of information (more about this in a future post). I'm currently undertaking a TDD exploration of the core domain model. One of the requirements that I have identified is the ability for a user to define a set of properties that can be held against each piece of information. Each property has a name that maps to a value - a fairly simple concept.

As I was writing the test cases and domain classes for this requirement I drifted down a route of allowing typed property values. Thus, rather than just String values, they could be Int, Boolean, ThingaMaBob or whatever you want them to be. However, definitions of properties and assignment of values are managed via a user interface rather than in compiled code so this solution started to grow very complex as it dropped into a world of runtime type validations and conversions and so forth. 

As I wrote more and more code the solution began to smell more and more like over engineering (when you start needing Scala Manifests in a domain model you know something isn't quite right). I could see myself getting bogged down in getting this working, so I decided to take a step back and look at the requirement again. Was I sure that I would actually need typed property values? No, maybe in future but not for any specific cases I could currently identify. Was having typed property values allowing me to get a greater understanding of my domain model? No. Was the requirement for typed property values a valid one, or just an excuse for over engineering? The ultimate conclusion was that I could actually do everything I wanted for the foreseeable future with simple String values and some basic validation rules.

With my new found simplification in hand I dumped a whole load of complex code and ended up with a very simple, very elegant solution that works perfectly for my current needs.

Sadly, I all too often see developers go down this same route but fail to realise that they are over engineering. They then spend an excessive amount of time building a solution that is fragile, difficult to test and unresponsive to changes in requirements that might be different to those they originally imagined when building their solution. These over engineered solutions just contribute to the technical debt of projects that allow them to exist.

Always aim for the simplest and most elegant solution that meets the current set of requirements - nothing more, nothing less. Don't over engineer and don't try to predict future requirements. Note the emphasis on elegant: just because something is very simple doesn't mean it is a good solution. A simple solution can still be built from messy, poorly structured code that is just as difficult to maintain and enhance as its complex counterpart.

If we all focus on this goal of simple, elegant code then software becomes simpler, easier to understand and much easier to maintain and enhance. This then benefits us all through increased productivity and improved ability to deliver.

The Elusive Inspiration

I'm currently working on a software idea that I have had running around in my head for a number of years. It's based around concepts for classification of information, exploration of relationships between this information and how this changes over time. The domain model is fairly complex as I want to keep it fairly generic and configurable so that individual users can customise it for their own types of information and classifications.

Unfortunately I have come up against a mental block when it comes to this domain model. I think that the ideas have been going round in my head for so long and I have so many different variations and alternatives that I  just can't find a way to move forward with something concrete.

This has got me thinking about the mental process and how to unblock my creative self on this project. To this end I've been considering the following ideas...

Go Away and Do Something Else
Often I find that if I can't get my head around something then going off and doing a different project for a while allows my brain time to unwind and process my ideas in the background. After a while I then come back to my original project and more often than not this allows me make good progress. However, on this current project I've been going off and doing something else for ages and it's time I made some progress.

Go For a Thinking Walk
I love walking - especially going somewhere I haven't been before and just exploring. Often I find that during a walk of this type I am able to focus and concentrate on a single topic until I come to some form of resolution. I think I'll try this one at lunchtime today.

Whiteboard Session
I am a very visual person. I find exploring concepts on a whiteboard a great solution for fleshing out designs and examining alternatives. I love colours as well and often find a visual with different element types in different colours really clarifies my understanding. This will probably be my next approach to explore the results of my thinking walk.

Exploratory Coding
Sometimes even the best thinking and most perfect whiteboard drawings just don't capture the essence of a problem. At times like these a couple of hours of hacking out an exploratory solution in code can clarify all sorts thoughts and ideas. When doing this sort of coding I like to use a very dynamic language (Ruby is my general choice) as I find these allow me to capture lots of thinking in a short amount of time.  Increasingly I'm also doing exploratory coding in a Functional language (usually Haskell) as I find this a great way to structure my thinking.

Test Driven Development
A completely alternative approach is to avoid all the up front thinking of how the domain model will work. Instead, define some test cases that describe what the domain model should do and then derive the model out of these tests. My final implementation will certainly be carried out in this way, but for the moment I'm still thinking at a higher level. I feel that I need to better understand exactly what I'm trying to achieve before writing production code.

How do you resolve mental block issues such as these? What other techniques are available?

Additions:


Build a Mind Map
Another thought that I had was to build a mind map to explore all of the thoughts and alternatives. By restricting this mind map to just the domain model that I am interested in I might be able to look at the problem in a new light. I use ConceptDraw MINDMAP on my Mac for this purpose.

Agile done badly.....sucks!

In this article Peter Viscarola describes why he thinks that the Agile software development methodology sucks. In many cases he’s right, the process he describes does suck - but he’s not actually describing a good Agile process. His experience seems very tainted by the bad interpretation and adoption of Agile that I see in a (sadly large) proportion of companies who have jumped on the agile bandwagon.

Let’s look at some of his points and show that they apply to ‘Agile done bad’ as opposed to ‘Agile done well’...

Peter states:

For the life of me, I do not understand why I would ever want to write code for a complex facility before I have had the chance to design that facility and consider how it will interact with everything else in its environment. ...

But Agile is all about “just writing code.” System architecture? Design? In Agile, that‘s not in the plan. What‘s in the plan is “code it up and see how it works” and “you can fix it in the next sprint.”

I say:

There’s nothing in Agile that says you have to turn off your brain, stop thinking and just churn out mindless code. Just because Agile means you are working in short increments does NOT mean that you can skip the design stage.

The planning session at the beginning of every sprint should primarily be a design session where the team work out the detail of how they are going to complete the sprint’s work. I’ve even seen planning sessions produce UML diagrams, entity models, storyboards and a myriad of other design detail that the team use for the sprint. They are not just about producing a mindless list of estimated tasks!

Additionally, a good agile team will undertake many short design sessions during the sprint. Each time something new is learnt that might challenge the design identified in the planning meeting (such as a TDD session identifying a better approach), the team should breakout around the whiteboard and discuss how this changes the design that the team is working towards.

What about system architecture? interactions? and so on? Well, your agile team should have a good Technical Architect and this person should be spending a proportion of their time thinking about this, ensuring the system architecture is fit for purpose and keeping the bigger picture in their head. They can then feed this into the sprint planning and on-going design to ensure the work being undertaken fits into the wider solution.

Peter states:

And, that brings me to the second thing that makes Agile suck so badly: The whole “estimation” process. Every time somebody insists that I estimate how long it‘s going to take me to implement some particular functionality, and in Agile it‘s not uncommon to have to do these estimates with great precision,...

Why is estimating software development time so hard? Duh! Truthfully, I can’t even estimate with good precision how long it’ll take me to go to store and get a case of beer.”

I say:

I believe this misses the entire point of estimating in an Agile project. We should accept that no estimate can ever be perfect. Even the most skilful estimator can only give a value based on what they currently know. Any unknown or unexpected event that comes along will dictate that the estimate becomes wrong and needs changing.

So, why bother estimating then? The purpose of estimating individual tasks is not to nail down exactly how long everything will take. It’s purpose is to ensure visibility for the current iteration. If things are being completed quicker than expected then the iteration will run out of work so effort needs to be put into planning what extra work to pull in. If things are coming in above estimate then the team knows that there were more unknowns or unplanned events than expected. This means that they might need to change approach, go back and design a bit more or drop some of the planned work from the iteration.

Estimates are about transparent monitoring of progress that lets the team adapt their plans. They’re not about accurately defining exactly how long something will take. Sadly I see too many Scrum Masters/Managers breathing down developer's necks asking why they have spent 1 hour more than estimated on their current task - this is just plain wrong!

But, even if estimates are only a means of achieving transparency it’s still good to get them as good as possible. How do we do this? We eliminate unknowns by using the planning session to design what we are going to do before estimating it. We also have a good BA/TA who has been pre-thinking the big-picture, finding the unknowns and turning them into knowns even before we start detailed planning. Then we only have to deal with the unexpected!

Peter states:

And that leads me to the final Agile precept that fries my potato: User Stories. Every time I hear somebody say “I‘ve entered that as a user story” I want to puke. Why? User stories are just that: Stories. They‘re data. They‘re not wisdom from the ages. ...

...You code to these particular stories. No, you don‘t get a chance to think through the overall experience for any user. This is Agile software development. You don‘t get to think. You‘re not allowed to design. You‘re allowed to “get some code working” so you can try things out. And that code just needs to meet the user stories, and pass the tests that were so lovingly crafted and stored with those stories. Anything else? Well, that‘s for next sprint.

I say:

Any Agile project that just codes user stories purely in isolation from one another or the bigger picture deserves to fail! If this were the Agile process then it would totally suck! Sadly for many Agile teams - and by the looks of it the only ones that Peter has worked on - this is the reality of Agile.

But there is a better (and correct) way. User Stories are a mechanism for describing a particular feature that a user would find valuable. They should not be treated as silver bullet recipe where just coding this story will deliver a great sprint and a great product. You still have to DESIGN what you are going to build to deliver a story. The team must consider how this story will be implemented, how it will fit into the architecture and the bigger solution and the architectural improvements that will be needed. User interaction designers still need to consider how this new feature fits into the UI to ensure a seamless, efficient experience and so on.

Agile is NOT about just taking a User Story and hacking out the minimum code to meet the Conditions of Satisfaction for that one story. It’s about breaking development down into a number of small, more easily managed iterations. Each iteration should deliver one or more User Stories but in doing so it must also consider the evolving System Architecture, must Design each feature thoroughly so that it ‘fits’ properly into the wider system and produce documentation of the features that have been built. You still have to think, plan, design, code, test and consider the bigger picture. You just do it in more manageable chunks.

Peter, if all you have done on agile projects is turn off your brain and hacked out the minimum code required to get a story to pass then I’m not surprised you think Agile sucks. But those projects haven’t been following an agile methodology - they (and you) have just been hacking code.

Addendum

Since writing this I've had a good email debate with Peter Viscarola. In particular he gave me more context about the type of projects he was trying to develop using agile. I think the nature of these projects (very complex and detailed kernel driver work) is not a naturally good fit for the agile approach.

Agile is no silver bullet - it's just another tool in your toolkit. You have to select the correct tool for the job and use it in the correct way. Sometimes you don't have a nail to knock in so picking the same trusty hammer is not the best solution. However, other times you do have a nail and the trusty hammer is the right tool. I still believe it's wrong to say that agile sucks when its being used incorrectly or for the wrong purpose.

Agile can work on complex projects such as Peter's, but you have to accept longer sprints (4-6 weeks) as two weeks is just too short to build complex features. Secondly, User Stories is probably not the best way to specify kernel driver features. Finally, projects such as these usually need a couple of technical spikes with senior architects to define the high-level architecture and concepts. Full team sprints can then be used to provide the implementation of these concepts.

The Value of Hardening

In my last post I looked at projects suffering with large amounts of technical debt and ever slowing velocities. One solution that one of these projects put forward was to undertake one (or perhaps more) hardening sprints. In these sprints, all teams would stop development of new functionality and focus on two things:

• Fixing as many of the outstanding defects as possible
• Addressing and improving the areas of the code that exhibit the largest clusters of defects

I want to question the validity of this approach as I think it offers very little value and doesn't solve the longer term problems associated with technical debt and reduced productivity.

Firstly, all code will contain some defects. Even the best code created using TDD principles will still have the occasional defect. These sort of defects can be easily addressed in the normal sprint mechanism - just get the business to prioritise them correctly along with the user stories and they will get resolved. No need to a dedicated sprint to fix these.

Secondly, we look at the defect clusters. The problem comes with trying to fix defect clusters and improve the code in a single hardening sprint. My feeling is that this is just a papering over the cracks, which, while perhaps offering some quick wins, fails to address the underlying causes of those cracks.

In most applications, defect clusters occur for one of two key reasons:

• The product has been crammed with too many features without sufficient refactoring of the architecture and code. The net result is that the code is now too complex, poorly structured and difficult to maintain. Adding new features is significantly more likely to break other features that were working.
• An initial rough architecture and code base was rushed through to meet a deadline and has not been sufficiently refactored prior to trying to scale it up - resulting in poor performance and reliability.

In the first case, trying to fix defects doesn't resolve the underlying complexity issues. In fact, fixing one defect is significantly likely to introduce other defects - especially if the automated test are lacking (common in projects that have reached this state).

Additionally, trying to make smaller fixes and refactorings to this complex code initially results in less stability and higher defect discovery rates. Given the aim of a hardening sprint is to make the product more stable and bullet-proof, these refactorings and small improvements actually can have a negative effect on quality in the short term.

In the second case, trying to address performance and stability in a single sprint just won't work. Achieving highly reliable and performant code can only be met through an in-grained philosophy in the project. All code must be well written, well tested and proved performant as part of its development. Taking some existing code and trying to tweak it to make it so is largely a futile effort.

Given the above, there seems little value to a hardening sprint (or sprints). A much better approach would be to correctly plan out technical debt into a number of DETAILED stories. These should address reducing the complexity of architecture and code in a structured way, refactoring code in a more structured way, improving the thinking about scalability and improving project focus and processes to increase quality, reliability and performance of architectures and code. The business should then be forced to prioritise these above new functionality for inclusion in the ordinary sprint process.

This approach prevents the short-term, quick fix mentality of hardening sprints. These sprints consume large amounts of resource, produce only surface improvements and leave all the fundamental problems intact, while at the same time fooling the business into thinking that things are improving. Never a good combination.