The Importance of a Good Shard Key

In MongoDB (and many other database solutions) scalability is achieved by dividing your database into a number of smaller portions. Each portion is stored on a separate set of database nodes. The theory is that this approach allows your database to scale horizontally beyond the capacity of a single node and allows load to be spread more evenly across multiple clusters of nodes. In MongoDB this approach is known as sharding (many other databases call a similar concept partitioning).

When you deploy shards in MongoDB you have to select a field (or fields) that are present in each document as the shard key. The value of this field determines which set of database nodes holds a specific document. If you get this key selection wrong then the impacts on your system can be huge.

As an example of how shard key selection is vital, consider this real-world, non-IT example, which clearly highlights how a poorly selected shard key can massively impact performance of a system:

At the weekend I participated in a large Half Marathon event, with about 20,000 other runners. The number of bags to be stored while the race was on was too large for a single baggage tent, so the organisers had wisely decided to introduce a sharded approach by having two tents, each holding half the bags. Additionally, each tent was further sub-divided into separate evenly-sized sharded collections of bags, each managed by its own team of helpers.

Now, as a shard key the organisers had decided to use race number: a seemingly sensible choice given that this would be the single unique piece of information that every runner would have. The bag tents were therefore arranged so that tent 1 was for numbers 1 to 10,000 and tent 2 was for numbers 10,001 to 20,000. The sub divisions inside the tents were further broken down into 1000 number blocks (i.e. 1 to 1000, 1001 to 2000 and so on).

For pre-race bag drop off this sharding approach worked really well (the write scenario). Runners dropping off bags were nicely distributed across both time and the full range of values, so each tent and sub-division could work in parallel for the maximum write performance. This was clearly an excellent shard key selection for the write case.

The problem occurred at the end of the race when runners returned to the tents to collect their bags (the read scenario). Now, the organisers of the race had issued the numbers based on predicted finish time (1 for the fastest predicted finisher, 20,000 for the slowest): the result being that runners finished roughly in number order.

So, what happened then is that there was a initially massive read queue at tent 1 for the 1 to 2000 numbers, while the other shard nodes were almost completely idle. Then the queue moved to the 2001 to 4000 shards, and so on. Towards the end of the race, tent 1 was idle while tent 2 now had the read queues as the later numbered runners all finished.

We have a perfect case of a shard key that seems quite sensible by design but is actually fundamentally broken by the usage scenario of the system, in this case the (near) sequential arrival of numbers for retrieving the bags.

So, how can we solve this? Issuing the race numbers in a different order is one option, but a sequential numbering system based on predicted finish time is quite sensible for many other race organisation requirements. A better option is to change the shard key used by the baggage tents.

Basing the shard key around race number is still a good approach as this is the one piece of information guaranteed to be unique for each runner and the range of possible values is clearly defined and well distributed. Bag drop off (write performance) is never going to be a problem as runners will arrive fairly well distributed across both time and the race number range.

The challenge is coming up with a way of better distributing the bag retrieval (read performance). Fortunately with a sequential numbering system this is pretty easy. Rather than shard by the whole number, just shard by the last digit of the number: tent 1 – numbers ending 0 to 4; tent 2 – numbers ending 5 to 9. Then in each tent further divide into five separate shards, one for each ending digit. If finer granularity is required then within each shard it should be easy enough to sort and index by full race number.

Selecting a good shard key is hard. Considering both the write and read scenarios, and how the shard key will be utilised within these scenarios is critical. It can make a huge difference between a well-functioning system and one that fails to gain the benefits of the sharded approach.

Sometimes you just have to be lazy...

...or, solving problems with lazy evaluation.

I’ve recently been working on a simple template system, which also has the potential to grow into a full-blown document generation system. The basic principle is that a document is composed of a number of fragments. Each fragment is called in turn to generate its piece of the document and these pieces are concatenated together to product the final result.

My first implementation was a super simple one. We just fold over the list of fragments, accumulating the output from each into a string builder. Each fragment may alternatively generate an error, which we collect along the way as well. At the end of the fold we return either the string document or a list of errors.

  def render(fragments: List[Fragment]): Either[List[String], String] = {
    fragments.foldLeft(RenderResult()) { (acc, fragment) =>
      fragment.generate fold (
        { err => acc.copy(errors = err :: acc.errors) },
        { text => acc.doc.append(text); acc }     
      )
    }.toEither
  }

  case class RenderResult(doc: StringBuilder = new StringBuilder(), errors: List[String] = List()) {
    def toEither: Either[List[String], String] = ...
  }

Now, this works great for documents of only a few kilobytes in size. However, as the documents grow to multi-megabyte sizes this approach becomes infeasible. The amount of memory required to hold the document being generated becomes huge. Multiple concurrent generations become impossible. We need a better solution.

The typical suggestion at this point is to stream output from each fragment straight into an output writer of some form rather than collect and build a string.

  def render(fragments: List[Fragment], writer: Writer): Option[List[String]] = {
    fragments.foldLeft(List[String]()) { (acc, fragment) =>
      fragment.generate fold (
        { err => err :: acc.errors },
        { text => writer.write(text); acc }     
      )
    }.getErrors
  }

Unfortunately this doesn’t work here because of the possibility that fragments may generate errors. We don’t want to stream out partially complete documents. We could output to a temporary file and then copy this to the ultimate output on success, but this seems like a less than ideal solution.

A common approach in an imperative style would be to first run a check phase to see if there are any errors and then run a second render phase to produce the output.

  def check(fragments: List[Fragment]): List[String] = 
    fragmens.foldLeft(List[String]()) { (acc, fragment) =>
      fragment.checkForErrors map (_ :: acc) getOrElse acc
    }

  def render(fragments: List[Fragment], writer: Writer): Unit = 
    fragments foreach { fragment => writer.write(fragment.render) }

However, in this case the building of fragments might be quite costly and we don’t want to have to process them once to see if there is an error and then again to render.

However, as we are using Scala and programming in a functional style, we have an alternative approach. Instead of returning a string or writing direct to a writer we can return a lazily evaluated function. This function can encapsulate all the side-effect generating logic of doing the actual write - which is also nice from the point of view of being able to reason about our rendering code.

  def render(fragments: List[Fragment]): Either[List[String], (Writer) => Unit] = {
    fragments.foldLeft(RenderResult()) { (acc, element) =>
      fragment.generate fold (
        { err => acc.copy(errors = err :: acc.errors) },
        { func => acc.copy(fragmentFuncs = func :: fragmentFuncs }     
      )
    }.toEither
  }

  case class RenderResult(fragmentFuncs: List[() => String] = List(), errors: List[String] = List()) {
    def toEither: Either[List[String], List[(Writer) => Unit] = {
      ...
      (writer: Writer) => fragmentFuncs foreach { f => writer.write(f()) }
      ...
    }
}

The way this works is that the render method folds over each fragment, asking the fragment to do enough work to be certain that no errors will occur (e.g. validate data, take a temporary copy of any volatile values and so on). However, it doesn’t need to do the actual string generation. Instead, each fragment returns a function that will be called at some time in the future (or an error). The render code then accumulates all these small functions into one larger function and returns this.

The returned function, when called, takes a writer and iterates over all the small string generating functions, writing the output of each straight into the writer. Job done and in a mere fraction of the memory required by the original solution.

In cases where you need to generate large amounts of data or where you want to delay side effecting execution to a well-defined location in your code then lazy evaluation is a great approach to follow.

NOTE: All above code example are significantly simplified from the real code in order to demonstrate the approach.

Increasing Agile Capacity

A scenario: you have a successful agile project. It’s made up of a number of small teams, each working on a separate component of the application suite. This has been working extremely well. You are capacity planning for the coming six months and you identify that one of these teams is likely to become a bottleneck. There are just too many stories that fit into their component area. How do you deal with this?

The above is something that looks likely on one of the projects that I am working on. I’ve been putting in some thinking on how this might pan out. This post documents my thoughts.

Traditional Reaction

There are typically two standard reactions to above problem: grow the team that is likely to be the bottleneck, or add an additional team to work on the component. In my experience neither of these works. Adding more people is rarely ever a good way to make a project go faster!

Growing the team

The best agile teams I have worked on have had one thing in common. They have all had between 4 and 6 devs, a QA, and a Scrum Master. As soon as the team gets any bigger than this I’ve found that the following occurs:

• The performance gain of adding each additional dev is less each time.
• It becomes much more difficult to gain consensus.
• Team meetings become more long-winded and less productive.
• Communication becomes more difficult.
• It becomes difficult to maintain team values.
• Code cruft (technical debt) increases.

Thus, adding additional devs and QAs to a team, either permanently or by borrowing from another team, just doesn’t work. Typically I’ve even found that the team goes slower as it has to spend more time cleaning up after itself.

Adding temporary devs (perhaps for a day or two) is, I’ve found, often worse than adding a permanent dev as there is less sense of ownership and shared team values. Working with good developers makes this less so, but I still believe it occurs more frequently than we like to admit.

Adding more teams

Having multiple teams working on a single component’s code base just doesn’t work well. I worked on one project where we had four teams working on the same code. We were tripping over each other all the time. Particular problems I have seen working in this way:

• Communications between the teams becomes a big overhead.
• More difficult to gain consensus on major technical decisions.
• Scheduling work to avoid multiple pairs changing the same code is very difficult.
• Very difficult to maintain shared values across multiple teams.
• One team’s nice code is another team’s cruft.
• Code cruft increases because one team is less willing to refactor another team’s code.
• Different approaches to achieving the same implementation become more prevalent.
• One team undoes another team’s work because they disagree with how it was done.
• Important aspects like performance tend to slip between the cracks.
• Productivity of the combined teams is far less than the sum of the productivity of each team working independently.

Multiple teams working on the same code base is a recipe for disaster. It results in more overhead, lower code quality and more cruft. Also, you get far less productivity gain that you would had you allowed each team to work independently on their own component.

Other Approaches

So, given that there is more work than the measured velocity of the team shows they can achieve in the time available. And, given that we don’t want to grow the team or add more teams. How do we solve the problem?

The aim is to keep small teams who can each work on an area of code that they have sole ownership of. This will give them the incentive to ensure that this component is clean, simple, maintainable and has a low level of cruft. It will also maximise the productivity of each individual team by minimising communication overhead.

I think a combination of four specific solutions are the best approach. These being: reducing waste, simplifying requirements, moving complexity and splitting components.

Reduce Waste

A good agile team should already be pretty good at reducing their waste. Retrospectives that generate actions which are then implemented go a long way towards this. A good Scrum Master who removes impediments is vital. The team will be provided with the best tools for the job.

Still, there is likely to be further waste that can be eliminated to allow the team to go faster. The first set comes from the category of unnecessary process waste. This includes things like:

• Having to attend full meetings where there is very little need for members from the team to be present.
• Long-winded design sessions that could be cut down with a bit more decisiveness.
• Unnecessarily revisiting architecture and design decisions over and over again.
• Re-enacting the same debate repeatedly because consensus wasn’t achieved.

The other set comes from giving the bottlenecked team stories to play that aren’t essential to what they are trying to deliver. This can be things like:

• Helping ops with tracking down a problem
• Adding additional logging of messages to an external system
• Trying to reproduce an infrequent defect
• Updating a build pipeline

In an ideal world, all of the above should be done by the team. However, if that team has a high volume of work to get through then these tasks can easily be divested out to teams that are perhaps not running at full capacity, allowing the team that is to maximise their throughput of important functional stories.

Simplify Requirements

Often performance of teams can be greatly hampered by having requirements that are either too complex or not clearly thought out. Ensure the backlog is groomed to contain only the most essential stories. Make sure these have been paired down to the absolute minimum set of requirements needed to fulfill the business needs.

The worst scenario is having a heavily loaded team working on features that are overly complex and time consuming. I’ve seen many cases where teams have worked on a fairly complex story only to later find that the clients uses just the core 80% of the feature, which could have been built in 20% of the time. The extra 80% of the time spent delivering that last 20% of non-essential requirement is just waste.

Analysts and the business need to scrutinise the backlog, squeezing every unnecessary piece of work from the stories. If the business doesn’t need some small aspect right now then create a separate story and play it later when the team has less critical path work.

Move complexity

In most large software systems it is possible to locate complexity in different components. For example table paging could be implemented in the server code or in javascript on the browser. If complexity in one place is causing capacity problems then it may be possible to move elements of this complexity to another team that has more spare capacity.

This may not always be the most technically pure solution, but it may be the best trade-off in order to deliver a product on time and budget. The complexity can always be relocated in the future.

As an example, consider a story where one component has 8 points of work do to delivery a feature and it’s two clients each have 2 points of work to consume the feature. Total 12 points. However, it might be possible to split the work differently, so that the core component implements something much simpler for 3 points, while the two clients take on some complexity and implement 5 points each. Total 13 points. So, the two approaches are roughly equivalent in size. The second might result in some duplication on behalf of two teams, but this allows the team with a large backlog to complete more stories. A pragmatic trade-off.

Split Components

If a team has multiple components, then the logical option is to relocate one or more of those components to another team. What if they only have one component?

Ideally, buy reducing waste, simplifying requirements, and moving complexity it should be possible to shrink the size of the backlog for a component. Hopefully the smaller backlog should be sufficiently sized that the team can complete it. What if they still can’t?

To me this seems to be a problem of decomposition. A component that is so large and complex that a team of six devs can’t maintain and enhance it is a component that is too big. In this case it should be split into two or more smaller components, with the current team keeping one part and a different team taking the other part(s).

Making the split is always technically challenging, but provided it is made across a clear interface boundary then my experience is that it is usually possible divide a component up into smaller constituent parts.

This is probably the only way that I would recommend going about adding more people to a project - introduce a new team (or repurpose an existing team) and give them their own code base and component(s) to work on.

Learnings From A Scala Project

I’ve recently been working as an associate for Equal Experts. I’ve been part of a successful project to build a digital marketing back-end solution. The project started out life as a JVM project with no particular language choice. We started out with Java and some Groovy. However, the nature of the domain was one that leant itself to functional transformations of data so we started pulling in some Scala code. In the end we were fully implementing in Scala. This post describes the learning gained on this project.

The Project

The project team consisted of five experienced, full-time developers and one part-time developer/scrum master. One of the developers (me) was proficient in Scala development (18 months, part-time) while the others were all new to the language.

The initial project configuration (sprint zero) was based on a previous Java/Groovy project undertaken by Equal Experts. The initial tooling, build and library stack was taken directly from this project with some examples moved from Groovy to Scala. The initial project stack included:

• Gradle (groovy based build system) with the Scala plugin bolted in
• IntelliJ IDEA with the Scala plugin
• Jersey for RESTful web service support
• Jackson for Json/Domain Object mapping
• Spring Framework
• MongoDB with the Java driver

Tooling

The biggest hurdle faced by the project was tooling, specifically tool support for the Scala language. While Scala tooling has come on a huge amount over the last year, it is still far from perfect. The project faced two specific challenges: build tool support and IDE support.

Build Tool Support

The Gradle build tool is primarily a build tool for Java and Groovy. It supports Scala via a plugin. Similar plugins exist for other build tools like Maven and Buildr.

What we experienced on this project was that this was not the ideal scenario for building Scala projects. Build times for loading the Scala compiler were rather slow and support for incremental compilation was not always as good as we would have liked. More often than not we just had to undertake a clean and build each time. This really slowed down test-driven development.

Currently (in my opinion) the only really workable build solution for Scala is the Simple Build Tool, also known as SBT (https://github.com/harrah/xsbt/wiki). This is a Scala specific build tool and has great support for fast and incremental Scala compilation plus incremental testing.

With hindsight it would probably have been better to switch over from Gradle to SBT as soon as the decision was made to use Scala for the primary implementation language.

IDE Support

The Scala plugin for IntellJ IDEA is one of the best editors for Scala. In particular its code analysis and highlighting has come on in leaps and bounds over the last few months. However, the support for compilation and running tests within the IDE is still less than ideal. Even on core i7 laptops we were experiencing long build times due to slow compiler startup. Using the Fast Scala Compiler (FSC) made a difference, but we found it somewhat unstable.

What was more annoying was that after failed compilations in IDEA, we often found that incremental compilation would no longer work and that a full rebuild of the project was required. This was VERY slow and really hampered development.

One proven way of doing Scala development within IDEA is to use the SBT plugin and run the compilation and tests within SBT inside an IDEA window. This has proven very successful on other Scala projects I have been involved with and again with hindsight we should have investigated this further.

Some developers on the team also found the lack of refactoring options, in IDEA, when working with Scala somewhat limiting. This is a major challenge for the IDE vendors given the nature of the language and its ability to nest function and method definitions and support implicits. Personally this doesn’t bother me quite as much as my early days were spent writing C and C++ code in vi!

Libraries

Scala has excellent integration with the JVM and can make good use of the extensive range of Java libraries and frameworks. However, what we experienced in a number of cases was that integrating with Java libraries required us to implement more code than we would have liked to bridge between the Java and Scala world. Also, much of this code was less than idiomatic Scala.

What we found as the project progressed was that it became increasingly useful to switch out the Java libraries and replace them with Scala written alternatives. For example, very early we stopped using the MongoDB Java driver and introduced the Scala driver, called Casbah. Then we added the Salat library for mapping Scala case classes to and from MongoDB. This greatly simplified our persistence layer and allowed us to use much less and more idiomatic Scala code.

Aside: If you are working with MongoDB then I strongly recommend Scala. The excellent work by Brendan McAdams of 10gen and the team at Novus has created a set of Scala libraries for MongoDB that I think are unrivaled in any other language. The power of these libraries, while maintaining simplicity and ease of use, is amazing.

We never got a chance to swap out some of the other major libraries that the original project structure was built on. However with more hindsight we should perhaps have made the effort to make these swaps as they would have resulted in much less and much cleaner code. Some specific swaps that we could have done:

• Jersey for Unfiltered or Bowler
• Jackson for sjson or lift-json
• Spring for just plain Scala code

Scala projects work well with Java libraries. However when a project is being fully implemented in Scala then it’s much better to swap these Java libraries for their Scala implemented alternatives. This results in much less and more idiomatic Scala code being required.

Team Members

Having highly experienced team members makes a big difference on any project and this one was no exception. The other big advantage on this project was that all of the developers were skilled in multiple languages, even though most had not known Scala before. One thing I have found over many years is that developers who know multiple languages can learn new ones much more easily than those who only know one. I’d certainly recommend recruiting polygot developers for new Scala projects.

It also helped having at least one experienced Scala developer on the team. The knowledge of the Scala eco-system was helpful as was the ability to more quickly solve any challenging Scala type errors. However, the main benefit was knowledge transfer. All of the team members learned the Scala language, libraries and techniques much more quickly by pairing with someone experienced in the language.

Additionally, as the experienced Scala developer I found it very valuable working with good developers who were still learning the language. I found that explaining concepts really clarified my understanding of the language and how to pass this on to others. This will be of great benefit for future projects.

Any new project that is planning on using Scala should always bring on at least one developer who already has experience in the language. It creates a better product and helps the rest of the team become proficient in the language much more quickly.

Object-Oriented/Functional Mix

We found Scala’s ability to mix both object-oriented and functional approaches very valuable. All team members came from an object-oriented background and some also had experience with functional approaches. Initially development started mainly using object-oriented techniques, but leaning towards immutable data wherever possible.

Functional approaches were then introduced gradually where this was really obvious, such as implementing transformations across diverse data sets using map and filter functions. As the team got more familiar with the functional programming techniques more functionality was refactored into functional style code. We found this made the code smaller, easier to test and more reliable.

Perhaps the only real disadvantage of moving to more functional leaning Scala code is that some code became less readable for external developer with less exposure to the functional style. It became necessary to explain some of the approaches taken to external developers: something that would probably have not been necessary with more procedural style code. This is worth bearing in mind for projects were there will be a handover to a team not skilled in Scala and functional programming techniques.

Conclusions

The project was very successful. We delivered on time and more features than the customer originally expected. I don’t think we would have achieved this if the implementation was in Java, so from that point the smaller, more functional code base produced by developing in Scala was a real winner. Using Scala with MongoDB was a major benefit. Tools was the real let-down on the project. Some problems could be alleviated by moving from Java libraries and solutions to their Scala equivalents. However, there is still some way to go before Scala tools (expecially IDEs) match the speed an power of their Java equivalents.

The Architecture Doesn't Work

I've been building web and other large-scale, enterprise type applications for many years, following the standard accepted approaches to application architecture. I've been increasingly feeling that these approaches are somewhat flawed and that we need something better. Over this series of three blog posts I'm going to talk about some new technologies and architectures that I have been having some great success with. In this first of the three posts I want to look at why...

The Architecture Doesn't Work

If like me you've been building enterprise style applications in Java (or pretty much most other similar languages) then you will be intimately familiar with this layered architectural pattern:

For those, who need a quick refresher, here's a quick summary of each layer:

• Presentation Layer - Deals with rendering the current state of the application to the user/client. Could be a web page, an GUI application or even some structured representation (e.g. XML, JSON). Usually renders the contents of the Domain Objects, often by some form of template system.
• Controllers - Handles events generated by the user/client and orchestrates the processing and Presentation. Typical approach is to accept an incoming request, extract any supplied request values, call a Service, pass the resulting Domain Objects to the appropriate Presentation component and finally return a result.
• Service Layer - Executes business logic and processes against the Domain Objects and returns the resulting objects. Typically stateless and transactional. Processing usually involves using the Persistence Layer to load Domain Objects from a Persistent Store, working with the Domain Objects to mutate state, using the Persistence Layer to save the changes and then returning the resulting Domain Objects for presentation.
• Domain Objects - The main objects representing entities in the business, their relationships and the business rules associated with each. Entities are usually persisted to a Persistent Store. Some people prefer anaemic Domain Objects, but I have argued that rich models are significantly better.
• Persistence Layer - Provides an interface between the Service Layer and the Persistent Store that talks the language of Domain Objects. Allows the Service Layer to undertake CRUD (Create, Read, Update Delete) and Find operations against the Persistent Store.
• Persistent Store - The storage for the data in the application. Usually a Relational Database Management System (RDBMS). Domain Objects are typically mapped to tables. Supports transactional semantics.
• Object-Relational Mapping (ORM) - A library that assists the mapping between Domain Objects and the Persistent Store. Tries to solve the impedance miss-match between object and relational structures. Used by the Persistence Layer. Modern solutions use annotations on the Domain Objects (e.g. JPA & Hibernate)
• Container Services - General services provided by a container. Supports features including: dependency injection; transactions; configuration; concurrency; service location; aop etc.

So, why doesn't this work?

It has become apparent to me over many projects that the above architectural approach suffers from a number of technical weaknesses. Often these have come about through the aim of simplifying the programming model for developers, but as we will see later the result is often the opposite. First, let's look at what I believe to be the main general technical problems that we find in this standard architecture. Later we will look at the impacts of these problems on the application and developers.

Thread-per-request model

Typically, in the above architecture, each request is handled in its own thread. Threads are usually managed in a pool and are then assigned to a request when it arrives. The thread and request are then linked until a response is returned, at which time the thread is returned back to the pool. This approach aims to simplify the development model by reducing the need for developers to have to think about concurrency issues - effectively giving them a single-threaded application path for handling each request.

Unfortunately, assigning a separate thread to each request is a very inefficient way of utilising system resources. Threads are fairly heavy-weight entities, having to maintain their own stacks, address space and so forth. The work involved in scheduling them is also quite large and given today's i/o intensive applications, many of them will be in a blocked state at any point in time. Additionally, most systems can only support a finite number of threads before suffering degradation in performance. (Some languages have tried to overcome the thread problem by introducing lighter weight constructs, such as fibers, but these still share many of the same problems for concurrent programming.)

The technical limitation of this thread-per-request approach is to actually stifle system scalability and impact application performance. Additionally, when the need to share state occurs, then developers must step outside their comfort zone into the world of concurrent programming: something the thread-per-request model has isolated them from.

Constant reloading from the persistent storage

Given a thread-per-request model, there is a natural approach to load all objects required to service a request from the persistent store on each request. Often this is completely unnecessary as a large percentage of the required data is unlikely to have changed from one request to another, or to be different between two separate running requests. However, with no convenient way to store state safely between requests or share infrequently changing state across threads we are forced down the load everything each time direction. The alternative is to drop down into shared state and concurrent programming - and we've already explored why that's not the best option! Techniques such as lazy loading try to minimise the overhead, but we have to accept that much of the data an application uses is being reloaded unnecessarily on each request.

Using database transactions for optimistic locking & concurrency control

Given each request is working with its own copy of application data kept in its own thread, how does the architecture handle multiple threads trying to update the same data at the same time? The general approach is to use database transactions to implement optimistic locking. Two requests can each load a separate copy of some data in their own threads and update it as they wish. When a thread's current transaction is committed, the state is written back to the persistent store. The database's transaction support ensures that each thread's changes are committed in an atomic and correctly isolated manner. Optimistic concurrency can follow a last-commit-wins model, or many ORM solutions offer a row versioning approach so the second transaction to commit changes to an object will fail with a stale data exception.

However, relying on the lowest level in the application stack to prevent concurrent updates isn't always the best solution. For a start, it allows both transactions to pretty much complete all their processing before one discovers that all its work is no longer valid. Secondly it causes problems when concurrency in the application layers above is required as some state would be shared across threads, and thus transactions - not nice to deal with. Finally, it doesn't actually simplify the problem because instead of having to deal with concurrency, the developer must now deal with the possibility of transaction rollback due to stale data, and the need for replaying of transactions. You'd be surprised how many applications I've come across that either don't address the problem (just raise a 500 error) or try to do something in this area and fail miserably!

Tight coupling between domain and persistent store through ORM

Another problem inherent in this architectural model is the tight coupling between the domain objects and the persistent store. This is caused by the need for the domain objects to be closely coupled to the object-relational mapping (ORM) solution. Why would we want to couple our domain objects this tightly? Well, it turns out that quite often we have to do this for performance reasons related to needing to reload from the persistent store for each request. Given that we are loading data so frequently we find that we have to tune our domain model not to make it optimal to our business model but to allow us to work most efficiently with the persistent store. And, don't even get me started on annotation based configuration (aka JPA) which forces us to consider and include all the details of our persistence strategy inside our representation of the business domain!

Scaling through adding servers

Once we have built our application using our architecture, we then have to deploy and scale it. Given that our thread-per-request approach doesn't make the best use of resources, the typical approach to scaling is to add additional servers, each running a separate copy of the application. Unfortunately this is not a particularly good way to achieve scale because we are still dependent on a shared database - and we are making heavy use of the database by loading data on each request and also using it for concurrency control.

Additionally, this approach to scalability also causes session based data sharing problems. With a couple of servers we can perhaps use session clustering with some success. As the number of servers grows we end up either having to use sticky sessions (i.e. tie each session to a single server - which means sessions are lost if that server crashes) or use some other way of persisting session data. This usually involves either adding session state into our already stressed database or introducing a whole new technology (distributed cache) to handle this data.

Static configuration

While not as significant as many of the above items, our standard architecture, and the libraries it is built upon, tends to push us towards a model of static configuration held in XML/YAML/Properties or other similar files. This increases the difficulty in managing our application at scale and in trying to make dynamic changes to values to tune the running system. When we want to support dynamic configuration we are often forced down the route of exposing properties to a management console (i.e. JMX) or pushing configuration down into the database or distributed cache, which we already saw above is not the ideal direction.

Poor approach to fault tolerance

Finally, we have the problem of fault tolerance. Typically, the approach to exceptions at any stage of processing a request is one of: catch the exception at the top level, log it and then display a generic "Oops, something went wrong!" message to the user. Hardly a great user experience. Unfortunately our architecture makes doing anything more than this prohibitively hard. For example, we might want to put some context information in the message. However, when the exception was thrown the developer might not have added this context information to the exception or, more likely, the exception was thrown by a library (such as the ORM) which is unaware of the context in the first place. We might also try to implement a re-try strategy, but implementing retries in a thread-per-request model is actually very difficult - you have to remember to throw everything you currently have away (because memory state in the thread may not be consistent with the database), clear all caches and so on. Try it, it's pretty hard!

And what is the impact?

So, given all of the above technical weaknesses with our current architectural approach, what effects do these have on our application?

Achieving performant applications is more difficult than it should be

The first major impact is that the architecture makes it hard to achieve application performance: the thread-per-request model makes poor use of resources and, loading unchanged data from the database for each request is inefficient and overloads the database.

Given these problems we generally find that we have to consider performance far earlier and in much greater depth than we really should. I'm not saying that we should never develop without considering performance (that would be negligent), but that the need to optimise our well written code and architecture for performance should be something we do rarely and in response to specific issues identified during performance testing. Designing a domain model so that it has the most efficient mapping to a database in order to achieve basic application performance is neither desirable nor the best way of capturing an accurate representation of the business domain. Compromising simple code for more complex but performant alternatives just to meet basic performance criteria is never desirable.

Another impact of the poor performance characteristics of our architecture is that we often end up having to circumvent the simplified thread-per-requets model in order to get a performant and usable application. What do we mean by this? We have to add some form of shared state - exactly what the simplified request model was put in place to avoid. Typically, we do this in one of two ways: adding caching or doing concurrent programming.

When an application proves to be less performant than required, most developers will initially suggest adding caches for data and results that don't change regularly. There are plenty of places in an application where caching can be applied. However, implementing caching is SIGNIFICANTLY more complex than most developers like to assume. Let's consider some of the things that need to be addressed: how do we keep the cache in sync with the database? what's the expiry strategy? what's the update strategy? do we pre-warm? do we block reads during an update or just block concurrent updates? is a cache per server fine or do we need a distributed solution? how will we ensure we don't get cache deadlock? As you can see, implementing a cache is actually a very complex option which requires great skill to get right. Caches should be reserved for special cases and should not be a catch-all for badly performing code and architecture.

The alternative approach to a cache is to introduce some form of shared state with concurrent access. Rather than loading all objects on each request, we keep common objects in memory and share them across and between requests. However, this comes with many of the same problems as using caching, plus the requirement of transaction management for updating this shared state. If done right, shared state with concurrent access can be simpler, more elegant and more flexible than caching, but it's the doing it right which is the challenge.

Now, this might be an over-generalisation, but my experience has shown that most 'average' developers can't write concurrent code using shared mutable state, multiple threads and synchronisation blocks. They usually end up with deadlocks, inconsistent state and difficult to track down bugs. Why? Because it is HARD and also because the single-threaded thread-per-requet model that they are used to programming to has isolated them from any considerations of concurrency. It's true that modern concurrency libraries (e.g. java.util.concurrent) have helped the problem, but most developers are using these without understanding exactly the reasons why. Just ask an average java programmer to explain the 'volatile' keyword and you will see what I mean!

Additionally, our dominant L-mode brain activity is optimised for linear, single-threaded thought. This in turn makes thinking about concurrency and all its implications a very unnatural process. I don't have any proof, but I have some very strong suspicions that a large proportion of the developer community is just unable to comprehend the real impacts of concurrent threads reading and writing in a shared state model.

Scaling applications is difficult

The other major impact of our current architectural approach is that it doesn't scale very well as application load increases. Firstly, the thread-per-request model just doesn't make good use of resources. Application servers have a finite set of threads that they are able to handle at any one time. Going beyond this limit significantly slows down the server - even though at any time many of the threads will be blocked on i/o. Once a server has reached thread saturation, the only solutions are to speed up performance (which we have seen is very hard) to free threads up more quickly, or add more application servers.

However, adding more application servers doesn't actually solve the problem. All it does is enable more concurrent threads to run, which in turn puts more load on the underlying database. Given the approach of re-retrieving data on each request we just end up overloading the database. We then need to scale the database to handle the extra load - and scaling databases is a VERY specialist job and VERY expensive as well. There's no way we can follow a path of continued scalability following this model. Clearly having an architecture based around cheap commodity hardware running app servers doesn't work if you need to massively scale databases to get any performance - so we are back to caching and shared state again, and we already explored what a nightmare these can be.

The final difficulty of scaling the application comes in the way configuration and shared session state is managed. At low scale, configuration by static config files is fine. Sharing session state through session clustering or sticky sessions is also acceptable. However, as the number of servers grows and infrastructure becomes more complex you soon end up in configuration hell - different versions of the application with slightly different configurations, massive challenges tuning parameters on running servers and so forth. Additionally, sticky sessions become less feasible and any form of session state clustering becomes impossible. Session state therefore needs to be pushed down into the database, increasing its overload and scalability issues or we start having to introducing distributed shared state solutions, with all the problems this entails.

So, it really doesn't work. Where do we go now?

We've seen that the current standard layered architecture that we are using to build applications causes a number of performance and scalability problems. Mostly these have come about from the desire to shield 'average' developers from the complexity of concurrent programming with shared mutable state. Unfortunately this simplicity causes performance problems and impedes scalability, requiring that developers must work around the restrictions in order to build something that works. There must be a better approach that solves these problems without needing developers to become shared mutable state concurrent programming gurus!

In the next post in this series I will look at a solution to these problems using lightweight asynchronous 'actors' and immutable messages. The final post will then look at some specific technology solutions that I have employed to implement this new approach.

Anti-Patterns and Rich Domains

At one of my customers we have been doing a lot of work on rich domain models, based on the principles of Domain Driven Design by Eric Evans. During this work we identified a particular anti-pattern in the existing code. I was trying to explain this to one of the developers on my team and it proved quite a subtle distinction. After thinking about it a bit more I decided to write this blog entry with some code examples to explain it in more detail.

Background

In the particular domain in question we have a (very simplified!) notion of Content and Metadata. Each piece of Content has some Metadata attached. However, the Content is sourced from one location (for example a CMS system) and Metadata is sourced from another (such as a remote web service). The challenge is combining the objects from these two different sources into a model that can be used by the higher layers of the application.

Anti-Pattern Solutions

The current solutions to this problem are based around the concept of an aggregator that pulls content from the different sources and merges them together into the returned objects. In all cases the returned objects follow the anaemic domain pattern of just fields with setter and getter methods.

Anti-Pattern 1: Transient Relationship

In this pattern, the aggregator pulls one object (the Content) from its source, uses an id field in this object to obtain the other object (the Metadata) from its source, and then sets the Metadata on the Content as a transient field:

public class Content {
    private String id;
    private Long metadataId;
    // Other fields

    private transient Metadata metadata;

    // Constructors, setters & getters
}

public class Metadata {
    private Long id;
    // Other fields

    // Constructors, setters and getters
}

public class Aggregator {
    // Fields, constructors and setters

    public Content getContent(String id) {
        Content content = cms.getContent(id);
        Metadata metadata = metadataService.getMetadata(content.getMetadataId());
        content.setMetadata(metadata);
        return content;
    }
}

The above code has a number of major problems:

• Clients of the Content class can use both the metadataId and metadata properties, possibly interchangeably. This makes for fragile client code that is not receptive to change.
• Any serialisation will result in the transient field being lost. How do we get it back?
• When we want to update the metadata relationship do we update the metadataId, metadata or both? How do we keep them in sync?
• What happens if we can't get the metadata, but successfully get the content? Do we throw an exception and prevent the content being used (even if it can)? or do we return content with a null transient field - requiring the client code to do null checking?

So, clearly having and id and a transient field representing the same relationship is not a good approach, can we do it without the transient?

Anti-Pattern 2: Decoupled Objects

In this pattern we avoid the transient object by keeping the two objects (Content and Metadata) separate and using the aggregator to retrieve each:

public class Content {
    private String id;
    private Long metadataId;
    // Other fields

    // Constructors, setters & getters
}

public class Metadata {
    private Long id;
    // Other fields

    // Constructors, setters and getters
}

public class Aggregator {
    // Fields, constructors and setters

    public Content getContent(String id) {
        return cms.getContent(id);
    }

    public Metadata getMetadata(Long id) {
        return metadataService.getMetadata(id);
    }
}

This code solves SOME of the problems of the first solution, but introduces problems all of its own:

• It is the total responsibility of the client code to manage the relationship between Content and Metadata - making the whole codebase more fragile
• It is very easy for client code to bypass the aggregator entirely and make direct calls to the metadata service, increasing complexity and coupling

So, what's the better way to do this?

The Rich Domain Solution

Our solution to the problem is to correctly model the relationship between Content and Metadata. We can introduce a Repository for Content that correctly sets up and manages the relationship, hiding the underlying management of ids and using real objects instead:

public class Content {
    private String id;
    private Metadata metadata;
    // Other fields

    // Constructors, setters & getters
}

public class Metadata {
    private Long id;
    // Other fields

    // Constructors, setters and getters
}

public class ContentRepository {
    // Fields, constructors and setters

    public Content getContent(String id) {
        Content content = cms.getContent(id);

        // We could proxy the metadata here for lazy loading or to delay exceptions being
        // thrown until the metadata is requested. However, in this simple example we will
        // build the relationship directly
        Metadata metadata = metadataService.getMetadata(content.getPrivateMetadataId());
        content.setMetadata(metadata);
        return content;
    }

    public void save(Content content) {
        // Persist changes and update the relationship if it has changed
    }
}

By hiding the management of the relationship between Content and Metadata in the core structure of the object model we prevent the details of how this is created and managed escaping from the domain model. Changes in the relationship between Content and Metadata can be correctly managed by the domain model. We also have options such as proxying the Metadata in Content to support lazy loading or delayed exception handling - something we can't do when details of the relationship are exposed to or managed by the client code.

Just goes to show how much a bit of good rich domain modelling can improve the structure and maintainability of software, even in very subtle ways.

Addendum

Just been thinking further on the subject. The above adequately addresses the technical aspects of the anti-pattern and the solution. However, there is also a huge cognitive element as well. The aggregator anti-pattern encourages a mental model of the domain as one of disconnected objets that are being forced together. Clients of the domain are therefore much more likely to create code that teats the model in this way: code that will be more complex and fragile. However, following the rich domain approach, clients have a mental model of a single object (Content) and it's relationship to other objects (such as Metadata) without being burdened with any of the separation knowledge. This cleaner mental model of the domain is much more likely to lead to clean, well architected client code that is easier to test, maintain and enhance.

Rich vs Anaemic Domain Models

Is your domain model a rich one, or is it one of those ultra anaemic ones? Where should business logic live - encapsulated inside the domain or in a number of higher-level services/controllers? Over and over I seem to have had the same debate with different companies and developers. So, once and for all here is my definitive view on the subject...

Anaemic Domain Models

An anaemic domain model is one typified by the Java BeansTM style of programming: simple Java domain classes containing just fields and setter/getter methods for those fields; logic for manipulating the domain objects is contained in higher level classes (typically a service layer). 

For example, consider the following simple anaemic domain model consisting of a Person and a list of Addresses associated with that person:

1:  public class Person {  
2:    
3:    private Long id;  
4:    private String forename;  
5:    private String surname;  
6:    private Date dob;  
7:    private List<Address> addresses;  
8:    
9:    public Person() {  
10:    }  
11:    
12:    public Long getId() {  
13:      return id;  
14:    }  
15:    
16:    public void setId(Long id) {  
17:      this.id = id;  
18:    }  
19:    
20:    public String getForename() {  
21:      return forename;  
22:    }  
23:    
24:    public void setForename(String forename) {  
25:      this.forename = forename;  
26:    }  
27:    
28:    public String getSurname() {  
29:      return surname;  
30:    }  
31:    
32:    public void setSurname(String surname) {  
33:      this.surname = surname;  
34:    }  
35:    
36:    public Date getDob() {  
37:      return dob;  
38:    }  
39:    
40:    public void setDob(Date dob) {  
41:      this.dob = dob;  
42:    }  
43:    
44:    public List<Address> getAddresses() {  
45:      return addresses;  
46:    }  
47:    
48:    public void setAddresses(List<Address> addresses) {  
49:      this.addresses = addresses;  
50:    }  
51:    
52:    @Override  
53:    public String toString() {  
54:      ...  
55:    }  
56:    
57:    @Override  
58:    public boolean equals(Object o) {  
59:      ...  
60:    }  
61:    
62:    @Override  
63:    public int hashCode() {  
64:      ...  
65:    }  
66:  }  
67:    
68:    
69:    
70:  public class Address {  
71:    
72:    private String line1;  
73:    private String line2;  
74:    private String line3;  
75:    private String town;  
76:    private String county;  
77:    private String postcode;  
78:    private String countryCode;  
79:    
80:    public Address() {  
81:    }  
82:    
83:    public String getLine1() {  
84:      return line1;  
85:    }  
86:    
87:    public void setLine1(String line1) {  
88:      this.line1 = line1;  
89:    }  
90:    
91:    public String getLine2() {  
92:      return line2;  
93:    }  
94:    
95:    public void setLine2(String line2) {  
96:      this.line2 = line2;  
97:    }  
98:    
99:    public String getLine3() {  
100:      return line3;  
101:    }  
102:    
103:    public void setLine3(String line3) {  
104:      this.line3 = line3;  
105:    }  
106:    
107:    public String getTown() {  
108:      return town;  
109:    }  
110:    
111:    public void setTown(String town) {  
112:      this.town = town;  
113:    }  
114:    
115:    public String getCounty() {  
116:      return county;  
117:    }  
118:    
119:    public void setCounty(String county) {  
120:      this.county = county;  
121:    }  
122:    
123:    public String getPostcode() {  
124:      return postcode;  
125:    }  
126:    
127:    public void setPostcode(String postcode) {  
128:      this.postcode = postcode;  
129:    }  
130:    
131:    public String getCountryCode() {  
132:      return countryCode;  
133:    }  
134:    
135:    public void setCountryCode(String countryCode) {  
136:      this.countryCode = countryCode;  
137:    }  
138:    
139:    @Override  
140:    public String toString() {  
141:      ...  
142:    }  
143:    
144:    @Override  
145:    public boolean equals(Object o) {  
146:      ...  
147:    }  
148:    
149:    @Override  
150:    public int hashCode() {  
151:      ...  
152:    }  
153:  }  
154:    

Then, we define some services on top of the domain model that obtain, use and update the domain objects to implement the functionality required by the business:

1:  public class PersonService {  
2:    
3:    private final PersonRepository repository;  
4:    
5:    public PersonService() {  
6:      repository = new PersonRepository();  
7:    }  
8:    
9:    public Long getPersonId(String surname, String forename) {  
10:      return repository.findPerson(surname, forename);  
11:    }  
12:    
13:    public Person getPerson(Long personId) {  
14:      return repository.getPerson(personId);  
15:    }  
16:    
17:    public void addAddress(Long personId, Address newAddress) {  
18:      Person person = repository.getPerson(personId);  
19:    
20:      List<Address> addresses = person.getAddresses();  
21:      if ( addresses == null ) {  
22:        addresses = new ArrayList<Address>();  
23:        person.setAddresses(addresses);  
24:      }  
25:      addresses.add(newAddress);  
26:    }  
27:    
28:    public void makeDefaultAddress(Long personId, Address defaultAddress) {  
29:      Person person = repository.getPerson(personId);  
30:    
31:      List<Address> addresses = person.getAddresses();  
32:      if ( addresses == null || !addresses.contains(defaultAddress) ) {  
33:        throw new IllegalArgumentException();  
34:      }  
35:    
36:      // Default address is always the first address in the list  
37:      addresses.remove(defaultAddress);  
38:      addresses.add(0, defaultAddress);  
39:    }  
40:  }  
41:    
42:    
43:  public class MailShotService {  
44:    
45:    public void sendMailShot(Person person, Long mailShotId) {  
46:      List<Address> addresses = person.getAddresses();  
47:      if ( addresses == null || addresses.isEmpty() ) {  
48:        // No mailshot can be sent  
49:        return;  
50:      }  
51:    
52:      Address sendTo = addresses.get(0);  
53:    
54:      // Code here to locate the mailshot and call the printing routine!  
55:    }  
56:  }  
57:    

The above code is overly simplistic (and somewhat contrived), but it demonstrates a key problem with this approach, namely that the encapsulation of the addresses property is broken. Specifically:

1) The fact that addresses are stored as a list is exposed to the service layer (and beyond). In fact, the PersonService is even responsible for creating the list instance. Changing the way addresses are stored in Person would mandate changing all the services (and perhaps controllers, pages and so on) that work with Person objects.

2) The knowledge that the first address in the list is the default address has escaped the domain model into the service layer. In particular there are two different services that both contain this knowledge. Should we want to change this approach we have to change and test code in two places (or more likely we change it in one place, forget the other and then wonder why our app behaves inconsistently).

Now, many proponents of the anaemic domain approach will tell you that the above problems can be avoided by correctly implementing your service layers. For example, only one service class is ever used to deal with Person. Any other services, controllers or whatever that need to access Person must use this service to do so. For example, the PersonService could have a new method: getDefaultAddress which would be called by the MailShotService. However, in my experience this never works for the following reasons:

1) Unless your developers are INCREDIBLY disciplined then this approach will always be violated. It's right before a deadline and a developer needs to access the default address from some controller in the system. Will they do all the work to inject the PersonService or will they just pull the first element off the address list? Most likely the second, and as soon as it's been done once then you can guarantee that that code will at some time be reused as a template for other code and the problem just proliferates from there. In 15 years I have never seen an anaemic domain pattern where this hasn't happened.

2) You end up with the higher level services and the controllers all having to inject large numbers of other services in order to get anything done. This results in a tighter coupling of the system and significantly increases the complexity of unit and integration testing (unit: need to define many mocks; component: need to pull in almost the whole system to test just one component). In every case I've seen, the anaemic domain pattern done in this way results in a small handful of controllers or services that pull in almost every other service in the system, which makes them really difficult to test and even more difficult to modify.

In my humble opinion, the anaemic domain model should be considered one of the most destructive anti-patterns of our time. It breaks the concept of good object oriented design and encapsulation and leads to service layers (and above) that become difficult to maintain and overly complex.

Rich Domain Models

An alternative is the rich domain model, where we attempt to encapsulate as much information about the domain inside the actual domain classes. We then expose these rich objects to higher levels, which can utilise the domain objects directly with less need for services containing arbitrary domain and business logic.

Looking at our RICH Address and Person objects:

1:  public class Person {  
2:    
3:    private Long id;  
4:    private String forename;  
5:    private String surname;  
6:    private Date dob;  
7:    private final List<Address> addresses;  
8:    
9:    public Person(final String forename, final String surname, final Date dob) {  
10:      this.forename = forename;  
11:      this.surname = surname;  
12:      this.dob = new Date(dob.getTime());  
13:      this.addresses = new ArrayList<Address>();  
14:    }  
15:    
16:    public Long getId() {  
17:      return id;  
18:    }  
19:    
20:    public void setId(final Long id) {  
21:      this.id = id;  
22:    }  
23:    
24:    public String getForename() {  
25:      return forename;  
26:    }  
27:    
28:    public void setForename(String forename) {  
29:      this.forename = forename;  
30:    }  
31:    
32:    public String getSurname() {  
33:      return surname;  
34:    }  
35:    
36:    public void setSurname(String surname) {  
37:      this.surname = surname;  
38:    }  
39:    
40:    public Date getDob() {  
41:      return new Date(dob.getTime());  
42:    }  
43:    
44:    public void setDob(Date dob) {  
45:      this.dob = new Date(dob.getTime());  
46:    }  
47:    
48:    public void addAddress(final Address address) {  
49:      addresses.add(address);  
50:    }  
51:    
52:    public void removeAddress(final Address address) {  
53:      addresses.remove(address);  
54:    }  
55:    
56:    public Collection<Address> getAllAddresses() {  
57:      return Collections.unmodifiableCollection(addresses);  
58:    }  
59:    
60:    public void makeDefaultAddress(final Address address) {  
61:      if ( !addresses.contains(address) ) {  
62:        throw new IllegalArgumentException();  
63:      }  
64:    
65:      addresses.remove(address);  
66:      addresses.add(0, address);  
67:    }  
68:    
69:    public Address getDefaultAddress() {  
70:      if ( addresses.isEmpty() ) throw new IllegalStateException();  
71:      else return addresses.get(0);  
72:    }  
73:    
74:    @Override  
75:    public String toString() {  
76:      ...  
77:    }  
78:    
79:    @Override  
80:    public boolean equals(Object o) {  
81:      ...  
82:    }  
83:    
84:    @Override  
85:    public int hashCode() {  
86:      ...  
87:    }  
88:  }  
89:    
90:    
91:  public class Address {  
92:    
93:    private final String line2;  
94:    private final String line1;  
95:    private final String line3;  
96:    private final String town;  
97:    private final String county;  
98:    private final Postcode postcode;  
99:    private final Country country;  
100:    
101:    public Address(final String line1, final String line2, final String line3,  
102:            final String town, final String county, final Postcode postcode,  
103:            final Country country) {  
104:      this.line1 = line1;  
105:      this.line2 = line2;  
106:      this.line3 = line3;  
107:      this.town = town;  
108:      this.county = county;  
109:      this.postcode = postcode;  
110:      this.country = country;  
111:    }  
112:    
113:    public String getLine1() {  
114:      return line1;  
115:    }  
116:    
117:    public String getLine2() {  
118:      return line2;  
119:    }  
120:    
121:    public String getLine3() {  
122:      return line3;  
123:    }  
124:    
125:    public String getTown() {  
126:      return town;  
127:    }  
128:    
129:    public String getCounty() {  
130:      return county;  
131:    }  
132:    
133:    public Postcode getPostcode() {  
134:      return postcode;  
135:    }  
136:    
137:    public Country getCountry() {  
138:      return country;  
139:    }  
140:    
141:    @Override  
142:    public String toString() {  
143:      ...  
144:    }  
145:    
146:    @Override  
147:    public boolean equals(final Object o) {  
148:      ...  
149:    }  
150:    
151:    @Override  
152:    public int hashCode() {  
153:      ...  
154:    }  
155:  }  
156:    
157:    
158:    

From the above you can see a couple of significant changes. Firstly, I've made as much of the data as possible immutable. In particular I've made addresses immutable and thus to change an address you have to remove the old one and insert a new one.  This stops any users of the domain model from making changes to objects that should always be managed within the domain. Secondly, the addresses field is fully encapsulated. Users of the domain model know none of its implementation detail and they cannot manipulate the contents of the underlying collection as this is never exposed in a mutable form.

Additionally, I've added specific types for Postcode and Country which will encapsulate all the conversion and validation logic for converting between user entered Strings and their actual meaning - logic that would normally be in a controller or service in an anaemic domain model.

This approach greatly simplifies the services layer. The PersonService needs only to provide methods to get the Person and the MailShotService can just call a simple get method on the Person object:

1:  public class PersonService {  
2:    
3:    private final PersonRepository repository;  
4:    
5:    public PersonService() {  
6:      repository = new PersonRepository();  
7:    }  
8:    
9:    public Long getPersonId(final String surname, final String forename) {  
10:      return repository.findPerson(surname, forename);  
11:    }  
12:    
13:    public Person getPerson(final Long personId) {  
14:      return repository.getPerson(personId);  
15:    }  
16:  }  
17:    
18:    
19:  public class MailShotService {  
20:    
21:    public void sendMailShot(final Person person, final Long mailShotId) {  
22:      Address sendTo = person.getDefaultAddress();  
23:    
24:      // Code here to locate the mailshot and call the printing routine!  
25:    }  
26:  }  
27:    

However, this is not the end of the story as there are still improvements to be made. In particular, exposing the ability to change a domain object in a layer above the services still is problematical: needs session in view pattern; may result in multiple places needing modification if a new cross-cutting concern is required (e.g. audit changes). So, it still has some of the weaknesses of the anaemic model.

We can therefore refine the domain objects even further through introduction of an interface to represent the public face of our main domain entities that expose only the accessor functionality:

1:    
2:  public interface Person {  
3:    
4:    Long getId();  
5:    
6:    String getForename();  
7:    
8:    String getSurname();  
9:    
10:    Date getDob();  
11:    
12:    Collection<Address> getAllAddresses();  
13:    
14:    Address getDefaultAddress();  
15:  }  
16:    
17:    
18:  public class BasicPerson implements Person {  
19:    
20:    ...  
21:  }  
22:    

Now we can return a Person instance that clients of the domain object can use to access the domain details, but they cannot mutate them via this interface. Then, I can update the PersonService to contain all the methods for modifying Person instances:

1:  public class PersonService {  
2:    
3:    private final PersonRepository repository;  
4:    
5:    public PersonService() {  
6:      repository = new PersonRepository();  
7:    }  
8:    
9:    public Long getPersonId(final String surname, final String forename) {  
10:      return repository.findPerson(surname, forename);  
11:    }  
12:    
13:    public Person getPerson(final Long personId) {  
14:      return repository.getPerson(personId);  
15:    }  
16:    
17:    public void modifyPerson(final Long personId, final String forename,  
18:                 final String surname, final Date dob) {  
19:      BasicPerson person = repository.getBasicPerson(personId);  
20:      person.setForename(forename);  
21:      person.setSurname(surname);  
22:      person.setDob(dob);  
23:    }  
24:    
25:    public void addAddress(final Long personId, final Address newAddress) {  
26:      BasicPerson person = repository.getBasicPerson(personId);  
27:      person.addAddress(newAddress);  
28:    }  
29:    
30:    public void removeAddress(final Long personId, final Address newAddress) {  
31:      BasicPerson person = repository.getBasicPerson(personId);  
32:      person.removeAddress(newAddress);  
33:    }  
34:    
35:    public void makeDefaultAddress(final Long personId, final Address defaultAddress) {  
36:      BasicPerson person = repository.getBasicPerson(personId);  
37:      person.makeDefaultAddress(defaultAddress);  
38:    }  
39:  }  
40:    

Finally, I make the constructor of the BasicPerson protected and add a factory so that there is now only one place that Person instance can be created (I also did the same for address, but it's pretty simple so I haven't shown it here):

1:  public class PersonFactory {  
2:    
3:    private final PersonRepository repository;  
4:    
5:    public PersonFactory() {  
6:      repository = new PersonRepository();  
7:    }  
8:    
9:    public Person create(final String forename, final String surname, final Date dob) {  
10:      BasicPerson person = new BasicPerson(forename, surname, dob);  
11:      repository.add(person);  
12:      return person;  
13:    }  
14:  }  
15:    

Thus, I have clean, well encapsulated domain objects that don't leak details of their internal implementation to the outside world. Wherever possible data has been made immutable to avoid accidental change. Any clients of the domain model can access its state via the exposed interface, but only the service can modify this state - thus making a single location for adding cross-cutting concerns (such as audit). I can therefore be certain that any code using my domain model will not be dependent on implementation details and will not be impacted by changes (provided I ensure the Person interface contract doesn't change).

You just don't get these benefits from an anaemic model without taking incredibly great care and ensuring that every developer who uses your code in the future also takes the same level of care. With a rich, well encapsulated domain model you protect yourself and those who use your code in the future by preventing the bad usage patterns from ever happening (even accidentally).