tag:blogger.com,1999:blog-54007656724054635922024-03-14T17:47:59.333+00:00So, you want to write software?Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.comBlogger64125tag:blogger.com,1999:blog-5400765672405463592.post-67430113959103077392022-04-15T22:46:00.001+01:002022-04-15T22:48:16.974+01:00Effective Knowledge and Experience Handovers<p>Over the last two months I’ve been working my way through a handover process as I prepared to leave the company where I’ve been working for the last eight years. This got me thinking a lot about what is a handover and how can it best be executed…</p>
<h2>What is a handover?</h2>
<p>The traditional view of a handover is where the employee who is leaving directly trains their replacement(s) in all the aspects of their job. However, for me this raises a number of questions and concerns:</p>
<ul>
<li>Why is the knowledge and experience of the leaver not already spread among multiple people in the company? Running with silos of knowledge owned by a single employee is a high risk strategy: that employee may suddenly be taken critically ill or suffer a serious accident.</li>
<li>Why is much of the knowledge and experience of the leaver not already documented in diagrams, playbooks or procedures? Having details stored only in employees’ heads and in their rough form notes prevents the easy transfer and preservation of that knowledge and experience.</li>
<li>What if there aren’t any direct replacements to handover to? It may have been impossible to recruit a replacement in time or the department may be restructuring and the leaver’s role may no longer exist going forward.</li>
<li>What if the replacement has a different set of skills, experience and perspective on how to undertake the role? There may be better ways to do the job that the leaver hasn’t explored and just indoctrinating the replacement to do exactly what the leaver did is likely not the most effective approach to move forward with.</li>
</ul>
<p>Clearly this traditional approach of taking someone new and just showing them how to do your job isn’t effective. So, is there a better approach and what should it be? Having gone through the process, my definition of an effective handover is now:</p>
<p><i>To ensure that knowledge and experience are preserved and communicated in such a way that they are accessible to everyone within an organisation, even those that arrive after the source of that knowledge and experience has departed.</i></p>
<p>Therefore, by definition this suggests that handover is more about producing durable materials and resources (as well as conducting direct knowledge transfer between individuals).</p>
<h2>Types of knowledge and experience</h2>
<p>While working through how best to communicate and preserve my knowledge and experience it became very obvious that this information was divided into two very clear categories:</p>
<ul>
<li>Explicit knowledge and experience - that which is clear and without vagueness or ambiguity.</li>
<li>Implicit knowledge and experience - that which is understood but not described clearly or directly, and often using implication and assumption.</li>
</ul>
<p>How these two types of knowledge and experience are identified and dealt with in a handover are very different.</p>
<h2>Explicit knowledge and experience</h2>
<p>Ideally in a well functioning engineering team this explicit knowledge should already be captured and widely used, so there should be a minimal need for handover in this area.</p>
<p>Examples of how this explicit knowledge is captured include things like:</p>
<ul>
<li>Having architecture diagrams and descriptions that are kept up-to-date as the system landscape changes.</li>
<li>Having a record of key architectural decisions, why they were made and what alternatives were considered.</li>
<li>Having documented processes where these are necessary. Examples of these include:
<ul>
<li>Processes for onboarding new team members</li>
<li>Definitions of done</li>
<li>Process for releasing code to production</li>
<li>Escalation processes for any production incidents</li>
</ul>
</li>
<li>Having Engineering Guides and Playbooks that detail things like how to configure systems, housekeeping tasks, build pipelines and so forth.</li>
<li>Readme files in code repositories that detail what they do, how to build and use them, key dependencies etc.</li>
</ul>
<p>If these are not already available then the first stage of handover work should always be to ensure that any of this explicit knowledge that the leaver is holding is captured. This likely involves a fair bit of document writing, updating knowledge bases and so forth. It’s also wise going forward to ensure mechanisms are in place within the team to ensure that this situation doesn’t occur in the future!</p>
<p>There is also great value to just surfacing and sharing this explicit knowledge. Some members of the team may not be aware of certain topics, others may have forgotten. There may also be team members who perhaps learn better from verbal communication and interactive sessions than from reading documents.</p>
<p>I found great value in being able to do interactive handover sessions, drawing on whiteboards, looking through code and so forth. This provides the team with good context on each subject which they can now use as a primer for diving into the more detailed information captured in the diagrams and documents.</p>
<p>An additional benefit I found when doing interactive sessions was that often a question or discussion topic would arise that I realised wasn’t already captured somewhere. This then highlighted the need to go back and improve on the store of knowledge, to ensure that assumptions or omissions became part of this explicit information set.</p>
<p>With hindsight there would have been great value to recording these interactive sessions and including them within the explicit knowledge set as well so that people not yet part of the team could utilise them as part of their onboarding process.</p>
<h2>Implicit knowledge and experience</h2>
<p>This part of the handover process is much more challenging. Most people, after working in a job for a long time, will have things that they just ‘know’ without realising that other team members may not ‘know’ the same things.</p>
<p>Some examples of this implicit knowledge and experience include:</p>
<ul>
<li>Handling a support issue and being immediately able to point to the cause of the problem.</li>
<li>Seeing a particular error line in a log and being able to go to the exact module and piece of code that is the likely root cause of this issue.</li>
<li>Getting a vague requirement from a customer and immediately understanding exactly what it is they are asking for, because someone asked the exact same thing in the past.</li>
<li>Doing an admin task that seems so trivial that it just never occurred to write it down anywhere.</li>
<li>Subconsciously following a process from the past that may not have been effectively communicated to newer team members.</li>
</ul>
<p>The big question I focussed on as part of my handover was how to actually identify this implicit knowledge. In the end I came up with two specific techniques that I felt worked well for me:</p>
<p>The first approach was a forensic analysis of emails and our issue tracking system. I found great value in the ‘sent mail’ folder looking for conversations that I might have contributed ideas or opinions in. The history of tickets in the issue tracking system was also incredibly useful. In particular I focussed on tickets raised by support and internal defect tickets; those where I had collaborated but not directly owned were a particularly good source of implicit knowledge.</p>
<p>The second approach was much more of a meditative one. Firstly I spent time thinking over all of the tasks that I carry out on a day-to-day basis, with a focus on both periodic and adhoc items. Replaying questions from the support team was also very useful. Secondly I made a list of all the projects that I had worked on and did the same exercise, thinking over any previous admin, support work and conversations with customers.</p>
<p>A small number of items that came up from these approaches were clearly explicit knowledge that just wasn’t captured, so this was added to the appropriate places. The remainder were very much things that were known, but vague and rather undefined. Clearly people should be aware of and have a ‘starter’ should it become necessary to work on them. However none of these additional items were clear enough to make explicit or significant enough to require dedicated interactive handover work.</p>
<p>Capturing these implicit items was where Playbooks really came into their own. For each identified item I was able to create a section that considered:</p>
<ul>
<li>How does the item get noticed, reported or spotted?</li>
<li>Are there any particular times or schedules that are appicabvle to the item?</li>
<li>What are the symptoms that might point to the item? How might these manifest themselves?</li>
<li>Has this item or question occurred in the past? How was it resolved then?</li>
<li>What resources or information are needed to investigate or diagnose?</li>
<li>Where should someone start looking?</li>
<li>What are the possible answers or solutions? Have any been considered or discounted in the past?</li>
<li>Are there any new processes that could be documented?</li>
<li>What further information would be useful to capture so that implicit items could be converted into more explicit knowledge?</li>
</ul>
<p>At the end of the process I was able to provide a Playbook of ‘starters’ to help future team members who encounter similar or closely related items from having to start from scratch.</p>
<h2>Conclusions</h2>
<p>A quality handover should include three elements: make sure all explicit knowledge and experience is captured; carry out interactive sessions to provide context and an entry point to this explicit content (ideally record them and add them to the content); and identify implicit knowledge and experience as record this as Playbook ‘starters’ that are available for team members to build on in the future.</p>
<p>My handover process was also a welcome reminder to make sure that explicit knowledge and experience is captured and updated continually. Don’t leave it until someone is leaving the team before you start generating diagrams, documenting processes and creating Playbooks for important activities.</p>
Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com0Thatcham, UK51.405805000000008 -1.26646823.095571163821162 -36.422718 79.716038836178853 33.889782tag:blogger.com,1999:blog-5400765672405463592.post-33261061995706844182022-03-12T21:13:00.002+00:002022-03-13T11:51:30.047+00:00The Value of a Safe and Open Interview Process<p>I’ve recently been interviewing for a new role. During this time I went through the interview process with three different companies. Two of these I already had in progress before finding the third role (which was the one I really wanted). Each offered a very different approach to the way they structured and conducted their interviews.</p>
<p>At the end of these processes I had very different emotional feelings about each of the three companies and roles. This got me thinking about what factors resulted in these different emotional responses.</p>
<h2>Company A - Traditional Interview Process</h2>
<p>This company had a fairly traditional interview process divided into four stages:</p>
<ul>
<li>An initial short chat with a member of the Talent Team to make sure that the role was a good fit. There’s no point in pursuing an expensive and time consuming series of interviews if the role is not a good fit for the candidate or visa-versa.</li>
<li>A technical interview with two Engineering Team members talking through experience, knowledge and approach, with a few technical and design questions along the way. This also acted as a chance to discuss and ask about engineering culture.</li>
<li>A problem interview, which for this company was conducting a code review on an example pull request. This was carried out in a low-pressure way where I was given time to do the review without the interviewers being present, followed by a short discussion of the points I had identified at the end. (Kudos to the person who wrote the code to review, it was a masterpiece of how to cram as much bad coding and design practice as possible onto a single page!).</li>
<li>A final chat with the Engineering Manager (which I didn’t complete as I had already accepted one of the other roles by this time).</li>
</ul>
<p>Overall I came away from this interview process feeling fairly neutral about both the company and the role. There was nothing that made me feel uncomfortable or unsettled, but at the same time I didn’t come away from any of the interviews feeling super excited or energised by the process.</p>
<p>They created an environment where I was able to feel fairly open to answering their questions, but still feeling that there were certain questions that they wanted to hear me give model answers to and others where the answers had to be correct.</p>
<p>I expect that this is the default experience for many people as they go through the process of finding their next role, but is this really the best we can do?</p>
<h2>Company B - Toxic Interview Process</h2>
<p>The second company I interviewed with had an interview process that I can only describe as toxic. I made it to almost the final stage of these interviews, but after my experience, there was no way I would have wanted to go and work for them anyway (even though they were the most high profile and highest paying of the three roles).</p>
<p>First off, before you could even enter their interview process you had to complete an online, timed, coding exercise. This involved two tasks where you are presented with a short problem statement, an empty code template and a set of unit tests (most of which you can’t see the inputs or expected outputs for). You then have to code up a solution that solves the problem and makes all the unit tests pass. All this takes place in an environment with a big countdown clock in the corner that ticks away your remaining time.</p>
<p>In my experience, there are very few people who can just sit down and pick up coding kata problems like these and produce great results. In order to be successful you have to have practised many times on similar tasks before actually taking the test. These practices also get you familiar with how the site doing the testing works, how it describes its problems, what sort of things it tests for and so forth. The net result is that to be able to pass the 90 minute test you also have to additionally invest a number of hours of practice time as well, and this is before you even know if the company or role is a good fit or not.</p>
<p>I also have some very specific complaints about these sort of logic/coding puzzle timed tests:</p>
<ul>
<li>All they prove is that you are good at logic/coding puzzles - they bear little resemblance to most of the engineering you will actually be doing in your job.</li>
<li>A lot of them require you to just ‘see’ the solution, and, if for that particular problem, you don’t have that immediate insight, then it’s almost impossible to complete a solution, as you just don’t have enough time to work through it and create working quality code.</li>
<li>Some people who are great engineers just don’t thrive in situations where there’s a countdown timer or someone watching their every keystroke.</li>
<li>These tests may also exclude people who feel less confident or a bit anxious from even being able to access the interview process at all.</li>
<li>They can create stress and discomfort, especially if the candidate is struggling with the particular problem they have been presented with and the clock is draining away.</li>
<li>Unit tests where you cannot see the input or the expected output, only that they are failing provide minimal value or oppertunity to improve your solution and just add additional stress and pressure.</li>
</ul>
<p>So, I did manage to pass this initial screening test. Even though my first impressions weren’t great, I decided to continue with the process just to find out more about the role, team and company - you never know, it may just have been some poorly considered HR requirement.</p>
<p>The next steps were multiple technical interviews with various Engineering Team members that all followed a similar pattern:</p>
<ul>
<li>A brief discussion about the role or some aspects of the project or engineering culture.</li>
<li>A bunch of fairly detailed technical questions covering skills and previous experience.</li>
<li>Yet more logic/coding exercises carried out live with the interviewer(s) watching and commenting on every line you wrote and expecting you to both code and narrate your thought processes as you did so.</li>
</ul>
<p>There were a number of aspects to these interviews that turned them into a hostile and stressful environment:</p>
<ul>
<li>When discussing projects or processes the interviewing engineers came across as acting in an incredibly superior way, sometimes sneering if your answer wasn’t perfect or you previously haven’t been working to the specific ‘variation’ of scrum that their teams have adopted.</li>
<li>They presented a demeanour of making you feel small or unworthy if you didn’t immediately know the Big O notation for a specific sort algorithm or couldn’t immediately code out a fully working, super efficient implementation of their chosen logic/coding problem.</li>
<li>I got the feeling that often they were trying to catch me out and just waiting for the moment to pounce: especially if I gave a slightly wrong or incomplete answer to one of their questions.</li>
</ul>
<p>The result of this sort of interview approach is that I started to feel stressed and pressured. Every question became a panic decision between do I give the answer I thought was right vs the risk that the answer might be wrong vs admitting that was something that I didn’t fully know. Rather than opening up to them I was being forced to close down. Ultimately I just didn’t feel safe or supported in that interview environment, started to doubt my own abilities and began to become flustered and unable to answer or write code any more.</p>
<p>At the end of each interview, the overwhelming emotion that I experienced was one of relief that the ordeal was over. No excitement about the role or the company was generated: just relief. After the final interview with them I was left visibly shaking from the release of stress tension built up over the duration of the interview.</p>
<p><b>Edit</b>: Having thought about this more, I'm sure that this company didn't set out to create an interview process that created these emotions. I believe they probably have the goal of selecting only the very best technical candidates. Their selected solution for doing this was to focus heavily on processes that test primarily for technical skills. The unintended side-effect of this being that they have less time to focus on cultural fit and empathy for the candidates. /end</p>
<p>If that’s their process for finding the ‘right’ people, what would they be like to work for? A culture that encourages that sort of pressure on interview candidates surely isn’t an enjoyable, safe and supportive place to work? Would the type of people who were actually able to make it through the recruitment process be the kind of people I would want as my future colleagues?</p>
<p>In the end they decided not to continue with the final stages of the interview process with me. I wouldn’t have gone for another interview with them anyway. I’ve never been happier to drop out of the running for a new job!</p>
<h2>Company C - Safe and Open Interview Process</h2>
<p>The final company I interviewed with followed a process that was a world apart from my other two experiences. This process was probably the most detailed and time consuming of the three, but every aspect of it was thoroughly enjoyable, engaging and enlightening.</p>
<p>First off was a chat with the Engineering Manager who I would be working with. This was about as far away from a technical interview as you can get. We just chatted. He spent a lot of time telling me about the company, the product and the culture. Then I was asked a simple question (I can’t remember the exact words, but it was something along the lines of): “What’s your first impression, what excites you about this role?”. What a great way to assess a candidate. If they aren’t excited by what they’ve heard then why would you want them in your team? As a candidate, if the role and culture aren’t exciting then would you really enjoy working there? Would you grow? After that we just chatted generally about experiences, background and so on.</p>
<p>Particularly interesting was that the chat was conducted in a way that felt safe. It was easy to open up, admit answers I didn’t know; ask questions that might seem strange; talk about times when things I did went wrong. The overall feeling was that the interview was about getting to know each other rather than trying to judge. I finished this step excited and already knowing that I really wanted this job.</p>
<p>The next stage of their process was a take-home exercise. I know some people aren’t a fan of these, but I’ve always found them to be quite an enjoyable way to demonstrate my abilities. If they are structured right then they allow you to show problem solving, technical approach and code that is realistic to what you would typically produce (as opposed to trivial/funky/clever code that solves a specific logic problem).</p>
<p>In this particular case the exercise was closely related to their product, so it almost felt like you were already working on something related to the job. The exercise was timeboxed to three hours (which is often a criticism of take home exercises that are open ended and thus candidates feel they have to dedicate a huge chunk of their life to completing them). I spent a day mulling over the problem in my head then sat down over a couple of sessions and produced my submission. I was really enjoying the problem so I even carried on for a few more hours after submission to improve my solution and solve a couple of little areas that I hadn’t quite got right.</p>
<p>My submission was reviewed and I was invited back to the next stage of the interview, which was to talk about what I had done. This was carried out with two members of the Engineering Team that I would be working with if successful. Again this was conducted in a way that was structured to be entirely safe and open. They seemed really interested in what I had done and how I had done it - not some kind of faked interest, but genuinely interested. At no point was there ever any judgement on the approach taken, it was always a discussion on the reasons and the thinking behind a particular approach or technique.</p>
<p>Feeling comfortable and secure, I found it easy to be open and honest about where I had made good or bad decisions and how I would do things differently as I iterated over the solution. This enabled much more discussion rather than it being a question/answer driven session. I was not only able to get a sense of how they worked as a team and how strong their engineering practices were, but also who they were as people - something that’s generally really hard to get during a typical interview.</p>
<p>Next up was another session, this time on talking through how I would go about solving a particular engineering problem. Again this was a problem relevant to their domain and product so it felt like I was actually working with the team rather than being assessed. This was also another great example of creating a safe interview space where it was easy to be open and explore different avenues. No suggestion was wrong or dismissed, I was free to explore avenues that didn’t go anywhere or solutions that might be a bit unusual. It was easy to ask questions about what might or might not be possible. The discussions were all about the directions taken to solve the problem, why they might work or not and how my thinking process worked to get there. Again this was very much a two-way process of how I would fit in with the team but also how the team would work with me.</p>
<p>I left both of the above sessions feeling even more excited about the role, excited about the people I would be working with and with a sense of comfort that I would be able to bring skills and ideas that would enrich the team and that I would get the same in return. That’s a really positive feeling to leave a technical interview with!</p>
<p>The final interview stage was back with the Engineering Manager again, this time with the Product Owner involved as well. Again this worked as much more of a discussion process rather than an interview. The PO was able to enthusiastically talk about the product and the vision for the future. There were ample opportunities to ask detailed questions and discuss thoughts. Finally there were questions around how I think about products, user experiences, technology and so forth. Again this was all conducted in a safe space where I felt at ease opening up with my thoughts and where I felt comfortable giving detailed answers and my reasoning behind them.</p>
<p>Overall this interview process was amazing. At every point in the process I felt able to give open and truly honest answers. After each interview I felt energised and excited. Suffice to say, when they offered me the position I jumped at the opportunity and I can’t wait until I join the team. In fact, I already feel part of the team just from the interview process and that’s a great feeling.</p>
<h2>Conclusions</h2>
<p>Thinking over my interview experiences there’s a few good takeaways that should be part of every organisations’ interview process:</p>
<ul>
<li>If an organisation can’t excite a candidate or the candidate can’t excite the team then the two may not be a good fit.</li>
<li>Creating a safe interview space will encourage candidates to be more open and honest with their answers, offer up more details and be more likely to ask deeper questions. Both parties learn so much more about each other.</li>
<li>Conversely, creating an interview environment that is hostile and stressful will discourage candidates sharing honest answers, will limit how much you can learn about them and ultimately will prevent them wanting to join your organisation</li>
<li>Some exploration of technical skills and experience is always relevant but ensuring the candidate is a great fit to the team culture personality wise is far more important,</li>
<li>During the interview, make the candidate part of the team and work with them as such. You will learn much more about who they are, how they work and whether they are a great fit than if you treat them as an outsider to interrogate.</li>
<li>Empathy with the candidate shows that the team they will be joining cares, and for great people that’s often a far bigger selling point than financial rewards or interesting technologies.</li>
</ul>
Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com0Thatcham, UK51.405805000000008 -1.26646823.095571163821162 -36.422718 79.716038836178853 33.889782tag:blogger.com,1999:blog-5400765672405463592.post-4031544680494185252014-03-04T08:51:00.000+00:002014-03-04T08:51:25.981+00:00The Importance of a Good Shard Key<p>In MongoDB (and many other database solutions) scalability is achieved by dividing your database into a number of smaller portions. Each portion is stored on a separate set of database nodes. The theory is that this approach allows your database to scale horizontally beyond the capacity of a single node and allows load to be spread more evenly across multiple clusters of nodes. In MongoDB this approach is known as <b>sharding</b> (many other databases call a similar concept <b>partitioning</b>).</p>
<p>When you deploy shards in MongoDB you have to select a field (or fields) that are present in each document as the <b>shard key</b>. The value of this field determines which set of database nodes holds a specific document. If you get this key selection wrong then the impacts on your system can be huge.</p>
<p>As an example of how shard key selection is vital, consider this real-world, non-IT example, which clearly highlights how a poorly selected shard key can massively impact performance of a system:</p>
<p>At the weekend I participated in a large Half Marathon event, with about 20,000 other runners. The number of bags to be stored while the race was on was too large for a single baggage tent, so the organisers had wisely decided to introduce a sharded approach by having two tents, each holding half the bags. Additionally, each tent was further sub-divided into separate evenly-sized sharded collections of bags, each managed by its own team of helpers.</p>
<p>Now, as a shard key the organisers had decided to use race number: a seemingly sensible choice given that this would be the single unique piece of information that every runner would have. The bag tents were therefore arranged so that tent 1 was for numbers 1 to 10,000 and tent 2 was for numbers 10,001 to 20,000. The sub divisions inside the tents were further broken down into 1000 number blocks (i.e. 1 to 1000, 1001 to 2000 and so on).</p>
<p>For pre-race bag drop off this sharding approach worked really well (the write scenario). Runners dropping off bags were nicely distributed across both time and the full range of values, so each tent and sub-division could work in parallel for the maximum write performance. This was clearly an excellent shard key selection for the write case.</p>
<p>The problem occurred at the end of the race when runners returned to the tents to collect their bags (the read scenario). Now, the organisers of the race had issued the numbers based on predicted finish time (1 for the fastest predicted finisher, 20,000 for the slowest): the result being that runners finished roughly in number order.</p>
<p>So, what happened then is that there was a initially massive read queue at tent 1 for the 1 to 2000 numbers, while the other shard nodes were almost completely idle. Then the queue moved to the 2001 to 4000 shards, and so on. Towards the end of the race, tent 1 was idle while tent 2 now had the read queues as the later numbered runners all finished.</p>
<p>We have a perfect case of a shard key that seems quite sensible by design but is actually fundamentally broken by the usage scenario of the system, in this case the (near) sequential arrival of numbers for retrieving the bags.</p>
<p>So, how can we solve this? Issuing the race numbers in a different order is one option, but a sequential numbering system based on predicted finish time is quite sensible for many other race organisation requirements. A better option is to change the shard key used by the baggage tents.</p>
<p>Basing the shard key around race number is still a good approach as this is the one piece of information guaranteed to be unique for each runner and the range of possible values is clearly defined and well distributed. Bag drop off (write performance) is never going to be a problem as runners will arrive fairly well distributed across both time and the race number range.</p>
<p>The challenge is coming up with a way of better distributing the bag retrieval (read performance). Fortunately with a sequential numbering system this is pretty easy. Rather than shard by the whole number, just shard by the last digit of the number: tent 1 – numbers ending 0 to 4; tent 2 – numbers ending 5 to 9. Then in each tent further divide into five separate shards, one for each ending digit. If finer granularity is required then within each shard it should be easy enough to sort and index by full race number.</p>
<p>Selecting a good shard key is hard. Considering both the write and read scenarios, and how the shard key will be utilised within these scenarios is critical. It can make a huge difference between a well-functioning system and one that fails to gain the benefits of the sharded approach.</p>
Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com0tag:blogger.com,1999:blog-5400765672405463592.post-85021821747040858862013-12-04T11:09:00.000+00:002013-12-04T11:09:47.277+00:00Scala's Maturing Community<p>I've been involved in the Scala community and attending Scala conferences for about four years now. I've just come back from <a href="http://skillsmatter.com/event/scala/scala-exchange-2013">2013 Scala eXchange</a>, hosted by <a href="http://skillsmatter.com/">Skillsmatter</a> in London. It was a great conference, one of the best I’ve been to, and the organisers and speakers should be thoroughly pleased with their efforts. One thing that I did think quite interesting was a significant change in emphasis from all of the previous Scala conferences that I’ve attended.</p>
<p>When I first started going to Scala conferences four years ago, the emphasis was definitely on an introduction to Scala. The talks focused on basic language features, how to use the collections libraries and core functional programming concepts. There were also some side talks about interesting libraries built in Scala, like Akka and Lift.</p>
<p>Last year the focus moved to a higher level, with more time spent on talking about where Scala was going and on more advanced functional programming concepts. There were talks focusing on things like Lenses and Monads and lots of detail about highly functional libraries developed using Scala. Presentaions about Akka and Lift were still present and people were starting to talk about Futures. In all of these talks, however, functional programming was the primary focus.</p>
<p>At Scala eXchange this year the emphasis was almost entirely flipped around. Most talks were about reactive programming using asynchronous approaches. Loads of stuff about Akka, actors, futures, events and the like. Some talks focused on Scala features such as macros and using type classes. However, there was very little direct talk of functional programming: it was just assumed that everyone present was using immutable data and following a functional programming approach. A huge shift in perspective from just a year ago.</p>
<p>I believe that this represents a massive shift in the maturity of the Scala community. We have moved from an immature group learning about how to use this exciting new language, how to work with immutable data and how to blend functional approaches into our code. We have instead become a group who are using this fantasic language and a set of fairly mature products and reactive techniques to build highly scalable systems. A huge leap in just a couple of years.</p>
<p>I remember a similar transition in the Java community when conferences went from talking about basic Java concepts like POJOs, collections and design patterns to talking about building complex solutions using advanced libraries like Spring and JMS. The Scala community seems to have made this leap much more quickly than the Java community did.</p>
<p>The one thing that worries me slightly about this change is that using immutable data and functional programming has almost become an unwritten assumption. Each presentation just seemed to assume that you would be programming in this way by default. While this is great for those of us in the community who have made this transition, I think we need to be careful not to skip this step for those people just transitioning into the Scala world.</p>
<p>The worst case would be developers transitioning from a language like Java straight into an asynchronous reactive programming model without first going through the transformation of thinking in terms of immutable data and functional programming. Bringing the concepts of mutable state and imperative code straight into an async world is a recipe for disastrous software projects. These in turn could tarnish Scala's representation and significantly complicate the life of those of us trying to bring Scala into traditionally conservative organisations and companies.</p>
<p>It's great the the Scala community has come so far and matured so fast. But, let's not forget the core concepts that underly this transformation. We must ensure that we continue to emphasise their importance as we move forward into the brave new world of highly scalable, asynchronous, reactive systems that Scala seems to the targeting so directly and successfully.</p>
Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com5tag:blogger.com,1999:blog-5400765672405463592.post-38900865201891485842013-04-15T08:25:00.000+01:002013-04-15T08:25:46.044+01:00Coaching: How Not What<p>Yesterday evening I was playing a game of 5-a-side football (soccer to my American readers) with some friends. One of the players on my team was a particularly competent player who was also very vocal in offering advice to his team mates on what to do.</p>
<p>For example, when a player from the opposing team was approaching me with the ball he would offer sage advice such as "Don't let them get past you!". This was rather stating the obvious, as I was quite clearly not planning on letting the opponent go past me and take a clear shot on goal.</p>
<p>In this particular situation my challenge was not in knowing what I should do, but in having the skill and experience necessary to carry out the task. No amount of shouted commands were suddenly going to increase my skill to a level beyond that which I currently possess.</p>
<p>This got me thinking about how we coach people in programming and agile practices. How often do we say to someone something like "that needs to be refactored" or "you need to keep your functions small"? Or, we might say to an agile team "don't overcommit yourselves this sprint". All of these are instructions to do something, made with the assumption that the people receiving them have the skill and experience necessary to carry out the actions.</p>
<p>What if, like me on the football field, the recipients of these instructions knows what they should do, but not how to do it. Clearly we are not being successful coaches in these cases. We need to focus much more on the how rather than the what. Advice offered should be enabling, providing the recipient with a way of learning and gaining experience.</p>
<p>For example, "if you pull this bit of code out into a separate function then this loop becomes much simpler" is much more helpful than "that needs to be refactored".</p>
<p>As coaches we need to be mindful of how we communicate in order to improve the people under our guidance. It's very easy to let it slip and just become another player shouting advice to those who haven't yet gained the skill necessary to implement that advice.</p>
Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com2tag:blogger.com,1999:blog-5400765672405463592.post-66473258019539612932013-02-06T08:18:00.001+00:002013-02-06T08:18:53.501+00:00Fast Track to Haskell Course<p>I've spent the last two days on a Fast Track to Haskell course. It was a very interesting two days and I learnt a huge amount. The course was taught by Andres Löh from <a href="http://www.well-typed.com">Well-Typed</a> and hosted by the excellent people at <a href="skillsmatter.com">Skillsmatter</a>.</p>
<p>I went into the course having read the superb <a href=""http://learnyouahaskell.com">Learn You A Haskell</a> book and having built a couple of experimental Haskell programs. From that starting point there was not a huge amount of content in the course that I was not aware of. However, it was great to have my learning verified and my understanding in many areas significantly improved. There were also a few things that changed my thinking on certain topics. All in all, a well spent two days.</p>
<p>The first day started with a whistle stop tour of the Haskell language and how to build things in it. There were plenty of excellent exercises to help cement learning and understanding. Then we looked in more detail at types and how to reason about software using types. The day concluded with looking at some more advanced type concepts including higher-order functions and type classes. The second day followed on with some more about types and then went on to look at how Haskell deals with I/O (which is very different to most other languages I have encountered). The rest of the day was then spent looking at common patterns that can be found in Haskell code and how these can be generalised into concepts such as Functors and Monads. Plenty more exercises followed.</p>
<p>I will now be going away and working on some private projects in Haskell, with the hope of attending the advanced course some time later this year. In the mean time, this blog post looks at some of the key things that I came away from the course with and how they relate back to the Scala and Java that I tend to do for my day job.</p>
<h3>Focus on the Types, they are the API</h3>
<p>When using an OO language, such as Java (or even Scala), we tend to think about classes, their interfaces and what behaviour they have, and then use this as a starting point to develop from. In Haskell you tend to think first about the types that are involved and then work forward from there. This is a very interesting approach as it makes it possible to think about a problem in a very general and quite abstract way without getting too bogged down in detail too quickly. The types become an API to your system (rather than the interfaces, methods and domain model in an OO solution).</p>
<p>My experience from the course was that this tends to lead to many more types and type aliases in Haskell than I would typically define in Java or Scala. However, this soon proves to be a huge advantage as it then makes it much easier to reason about what each function does/should do and makes its behaviour very clear via the type signature. I will certainly be looking at following this practice in any Haskell programs I write and also trying to introduce more types into my Scala code.</p>
<h3>It's Pattern Matching and Recursion All The Way Down</h3>
<p>In a language like Java there is nothing like pattern matching and recursion has to be used sparingly if you want to avoid blowing up the stack and writing poorly performing code. In Scala there is much more scope for both, although writing tail recursive functions is still very important to ensure that you get the best optimisations from the compiler. In Haskell, pattern matching and recursion are the bread and butter of the language. Almost every single function tends to be a pattern match on arguments and the language supports a naturally recursive style across the board. Given the lazy semantics of Haskell and the optimisation in the runtime for recursive algorithms this is the best approach for building programs in Haskell.</p>
<p>One area for improvement that I can see in my Scala code is to make more use of pattern matching, especially partial functions.</p>
<h3>Thinking Curried Helps a Lot</h3>
<p>In the Scala world we typically only curry functions when there are very obvious places to partially apply them. More often than not we just partially apply a function using the underscore for one of its parameters. This is needed to ensure compatibility with Java and the object-oriented model. However, in Haskell every function is completely curried. There are no multi-argument functions. Everything is a single argument function that returns either a result or another function.</p>
<p>Initially I found that this made sense in the model that I'm used to: partially applying functions. However, once it came to defining new types as actually being functions as well I found that this started to fry my mind slightly. The conclusion that I came to is that it's best to just think of everything from a curried mindset and throughout the course things gradually became clearer. This thinking is essential to understand things like the State monad (which I'm still not 100% confident about to be totally honest).</p>
<h3>Parametric Polymorphism is Very Useful</h3>
<p>One of my biggest take-aways from the course was how useful and important parametric polymorphism is. As an (overly) simple example, what does a function from Int -> Int do? It could return the same number, add something to it, multiply it, factor it, modulus it, ignore the input and return a constant - the possibilities are nearly endless. However, what does a function from a -> a do? Given no other constraints it can only do one thing - return itself (this is called the identity function - id in Haskell).</p>
<p>The ability to define functions in this polymorphic way makes it much easier to reason about what a function might do and also to restrict what a function actually can do. This second point greatly reduces the possibility of introducing unexpected defects or creating functions that have multiple interweaved concerns. As another example, consider the function: Ord a => [a] -> [a]. What does this do? Well, it could just return the same input list, but because it takes the Ord type class as a constraint it's quite likely to be some form of sorting or filtering by order function. As the writer of this function I'm restricted to working with lists and just using equality and ordering functionality so the possibility of doing anything unexpected is greatly reduced.</p>
<p>Parametric Polymorphism is possible in Scala, but not quite as powerful as there aren't type classes for may common concepts (e.g. equality, ordering, numbers, functor, applicative, monoid, monad etc.). Introducing Scalaz fixes this for quite a lot of cases, but that's not to the taste of many projects and mixed Scala/Java teams may find this a step too far. However, there's certainly more scope for using parametric polymorphism in everyday Scala code and this is something I'm going to work on ove the coming weeks (expect some more blog posts).</p>
<h3>Don't Try to Over Generalise</h3>
<p>Something that I've seen in every language is that once developers discover the power of generalising concepts they tend to go overboard with this. Remember the time when every piece of code was a combination of seven billion design patterns? The power of Haskell's type classes and parametric polymorphism makes it very easy to see patterns that may not actually hold. I can see that it would be very easy to waste a lot of time and effort trying to make a data type fit into various different type classes or trying to spot type class patterns in your code and pull them out into generic concepts.</p>
<p>As with any kind of design pattern, generalisation or similar, the best approach (in any language) is to just build what you need to solve a problem and then when it becomes apparent that it either fits an existing pattern or represents a general pattern then this change can be made as part of a refactoring process.</p>
<h3>Try to Keep Pure and I/O Code Separate</h3>
<p>Haskell's I/O model at first seems very different, unusual and restrictive. Once you get your head around it, it makes so much sense. Separating pure code from code that talks about doing an I/O action from code that actually does the I/O makes for software that is much more easy to reason about, test and reuse. Very powerful and something I'm looking forward to exploring more during my journey into Haskell.</p>
<p>In the Scala world, I'm not sure about the need to go to such lengths as using an I/O Monad or similar. However, I can really see the advantage of separating pure code from code that performs I/O. Having pure functions that transform data and then wiring them together with the I/O performing functions at a higher level seems a natural way to create software that is better structured and easier to test. I'll certainly be playing with some approaches in Scala to experiment with this concept further.</p>
<h3>Functors, Applicatives, Monoids and Monads are Nothing Special</h3>
<p>A final observation from the course is that many people learning and talking about functional programming get a bit obsessed with concepts like Functors, Applicatives, Monoids and Monads (especially Monads). What I learnt is that most of these are just simple patterns that can be represented by one or two basic functions and a few simple rules. Nothing more to them really. Certainly nothing to get all worked up about!</p>
<p>Yes, there's a great deal of skill involved in knowing when to use these patterns, when to make your own code follow a pattern and so on, but that's true of any programming language or design pattern. It's called experience.</p>
<h3>Conclusion</h3>
<p>If you are a Scala developer who is enjoying the functional programming aspects of the language I would certainly recommend taking a look at Haskell. I found that it clarified lots of things that I thought I understood from Scala but still had some doubts about. Haskell is, in my opinion, a more elegant language than Scala and once you start using it you realise just how many hoops the Scala team had to jump through to maintain compatibility with the JVM, Java code and object-oriented features. Over the two days I learnt a lot about Haskell, a lot about functional programming and a lot about how to get even more value from Scala.</p>
Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com6tag:blogger.com,1999:blog-5400765672405463592.post-13003264401354933432013-01-24T08:28:00.000+00:002013-01-24T08:28:27.982+00:00Type Classes Make For Better Software Structure<p>When people usually talk about type classes in languages like Haskell and Scala they tend to be thinking of highly generic concepts like Ordering or Numeric operations. However, in some recent experiments with Haskell I have found that type classes can be very useful for more concrete things. When used appropriately they can lead to more loosely coupled software that is more flexible and easier to change.</p>
<h3>I Just Met You, This Is Crazy, Here's My Number, Call Me Function?</h3>
<p>As an example, let's consider a telephone number. Fairly simple right, it's a string of digits?</p>
<pre class="brush:scala">
def sendMessage(toNumber: String, theMessage: Message)
</pre>
<p>Or perhaps we should create a type alias to give us some better info and scope for change:</p>
<pre class="brush:scala">
type TelephoneNumber = String
def sendMessage(toNumber: TelephoneNumber, theMessage: Message)
</pre>
<p>But wait, I'm trying to build a library that sends messages to telephones and I need to know much more about a telephone number than this 'stringly' typed data is able to tell me. Things like:</p>
<ul>
<li>Is it in international format?</li>
<li>What's the country prefix code?</li>
<li>Is it a mobile number or a landline number?</li>
<li>What is the area code?</li>
<li>Is it some premium rate number?</li>
<li>and so on…</li>
</ul>
<h3>The Naive Solution</h3>
<p>It's clear that my 'stringly' typed representation is not sufficient for this task unless I want to encode a whole load of rules about the world's telephone numbering systems into my library (believe me when I say I don't!). So, a naive approach would be to introduce a new type that holds the information in a more usable format. Something like:</p>
<pre class="brush:scala">
case class TelephoneNumber(countryCode: String, areaCode: String, number: String, isMobile: Boolean, isLandline: Boolean, …)
</pre>
<p>Unfortunately, this approach suffers from a whole host of problems, including:</p>
<ul>
<li>If I build my library implementation against this class then I can't easily change implementation details without breaking the API</li>
<li>If I copy from an instance of this type into an internal representation then I have a lot of duplication and extra code to test</li>
<li>If a client uses this type in their domain model they are tightly coupled to my library (and also a specific version of it)</li>
<li>If a client copies from their own representation into an instance of this type there is more duplication and extra code to test</li>
</ul>
<h3>A More Object-Oriented Approach</h3>
<p>So, how do we deal with this? Well what we need to do is separate the behaviour of a telephone number from its actual data representation. In an object-oriented language we would do this by introducing an interface (or in Scala, a trait):</p>
<pre class="brush:scala">
trait TelephoneNumber {
def countryCode: String
def areaCode: String
def number: String
def isMobile: Boolean
def isLandline: Boolean
// and so on...
}
</pre>
<p>Internally in my library I write all my code against this interface without any care as to how the underlying telephone number data is structured. Client applications are free to implement telephone numbers in whatever way they wish, provided they also implement the interface that exposes the behaviour that my library needs. Yay, problem solved. Or is it? There's still one problem in this object-oriented approach: somewhere in the client code there must be a class that implements the TelephoneNumber interface. This is problematical in a number of ways:</p>
<ul>
<li>The class hierarchy of the client domain model is still quite tightly coupled to my library</li>
<li>It might not be possible to modify an existing domain model to implement the TelephoneNumber interface (no access to source code, coupling restrictions etc.)</li>
<li>If the client domain represents telephone numbers as Strings then this class can be extended to implement the interface</li>
</ul>
<p>Both of the last two points would result in needing to implement the same sort of mapping layer to copy between representations that we already said lead to duplication and extra code to test.</p>
<h3>How Haskell Does It</h3>
<p>So, in a purely functional language that doesn't have the concept of interfaces how do we solve this problem. Enter the <b>Type Class</b>. These clever little beasties provide a mechanism whereby the required behaviour is captured as a series of function definitions (with possible implementations if available). For example:</p>
<pre class="brush:scala">
class TelephoneNumber a where
countryCode :: a -> String
areaCode :: a -> String
number :: a -> String
isMobile :: a -> Bool
isLandline :: a -> Bool
...
</pre>
<p>So far this looks pretty much like our interface definition above except that each method takes an instance of type 'a' as a parameter and returns the result. However, the clever bit comes in that we can implement the type class for any type in Haskell without needing to modify that type in any way. For example, we could easily implement it for Strings:</p>
<pre class="brush:scala">
instance TelephoneNumber String where
countryCode s = ...
areaCode s = ...
...
</pre>
<p>Or our client code could define its own telephone number data type and an implementation of the type class along side it:</p>
<pre class="brush:scala">
data UKTelephoneNumber = ...
instance TelephoneNumber UKTelephoneNumber where
countryCode t = ...
...
</pre>
<p>The final change is then that we have our library function require that something implementing the type class is provided rather than some concrete data type:</p>
<pre class="brush:scala">
sendMessage :: (TelephoneNumber t) => t -> Message -> IO ()
sendMessage to msg = do
...
let cc = countryCode t -- call function on the type class passing the instance we have
...
</pre>
<p>In this Haskell solution we have neatly decoupled the behaviour that our library requires of telephone numbers from the data type used to represent them. Also, any client code of our library does not need to couple its data structures/types directly to the library. They can define and modify them in any way they like. All they need to do is keep the instance of the type class up-to-date.</p>
<h3>And Finally, A Better Scala Solution</h3>
<p>As I have hopefully shown, the Haskell solution to the problem is both elegant and leads to cleaner, more loosely coupled code. So can we do something similar in Scala? Well, yes we can because Scala also supports type classes. They are a bit less elegant than Haskell, but they work just fine. First, we need to change the TelephoneNumber trait into a type class:</p>
<pre class="brush:scala">
trait TelephoneNumber[T] {
def countryCode(t: T): String
def areaCode(t: T): String
def number(t: T): String
def isMobile(t: T): Boolean
def isLandline(t: T): Boolean
// and so on...
}
</pre>
<p>Note that all we have done is added a type parameter and modified all the functions to take an instance of that type. Notice that it's now almost identical in structure to the Haskell equivalent. Next we need an implementation of this type class. Let's define one for String:</p>
<pre class="brush:scala">
object TelephoneNumber {
object StringIsATelephoneNumber extends TelephoneNumber[String] {
def countryCode(t: String) = ...
...
}
implicit def TelephoneNumber[String] = StringIsATelephoneNumber
}
</pre>
<p>I've put the implementation inside a TelephoneNumber object. If you are defining your own types, convention is usually that the type classes go in the companion object (e.g. for a case class UkTelephoneNumber the type class implementations would be in the UkTelephoneNumber companion object). This insures that the type class instances are always in scope when the type is used.</p>
<p>Finally, lets update our sendMessage function to work with type classes:</p>
<pre class="brush:scala">
def sendMessage[T : TelephoneNumber](toNumber: T, theMessage: Message) = {
...
val telNo = implicitly[TelephoneNumber[T]]
val cc = telNo countryCode toNumber
...
}
</pre>
<p>Note the use of the context bound symbol ':' plus the implicitly function to define the need for and access to the correct instance of the type class.</p>
<p>So there we have it, an elegant Scala solution the overcome all of the coupling problems inherent with the object-oriented approach of implementing a required interface. When should you use this? My opinion is anywhere that two system components communicate, where that communication can be defined in terms of behavioural functions rather than actual data and where you want to minimise the coupling between them and the leaking of types from one component to another.</p>
Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com9tag:blogger.com,1999:blog-5400765672405463592.post-76151418858206957802012-11-26T09:26:00.000+00:002012-11-26T09:26:56.301+00:00Encoding State Requirements in Types<p>In my <a href="http://skipoleschris.blogspot.com/2012/11/eliminating-conditionals-using-types.html">previous post</a> I looked at a simple way to eliminate conditional expressions by encoding them into the Scala type system. In this follow-up post I want to look at taking this further by using the Scala type system to encode state rules. The aim, as always, is to create code that fails to compile rather than code that fails at runtime. By doing this we also reduce the number of unit tests required.</p>
<h3>The Problem</h3>
<p>Consider a simple shopping workflow. I collect information about a Basket, the Customer and their Payment Method. Once I have all this in place I can create a processOrder function that completes the workflow process. As a naive starting point, lets encode this as a simple domain model holding optional values:</p>
<pre class="brush:scala">
case class CheckoutWorkflow(basket: Option[Basket],
customer: Option[Customer],
paymentMethod: Option[PaymentMethod])
object CheckoutWorkflow {
def empty = new CheckoutWorkflow(None, None, None)
}
</pre>
<p>Then we need some functions that populate the workflow:</p>
<pre class="brush:scala">
def processBasket(workflow: CheckoutWorkflow): CheckoutWorkflow = {
if ( workflow.basket.isDefined )
throw new IllegalStateException("Basket workflow step already processed")
// Do some processing...
workflow.copy(basket = Some(basket))
}
def processCustomer(workflow: CheckoutWorkflow): CheckoutWorkflow = {
if ( workflow.customer.isDefined )
throw new IllegalStateException("Customer workflow step already processed")
// Do some processing...
workflow.copy(customer = Some(customer))
}
def processPaymentMethod(workflow: CheckoutWorkflow): CheckoutWorkflow = {
if ( workflow.paymentMethod.isDefined )
throw new IllegalStateException("Payment Method workflow step already processed")
// Do some processing...
workflow.copy(paymentMethod = Some(paymentMethod))
}
</pre>
<p>Note that each of the above functions has a guard condition to stop them being called multiple times for the same workflow. Each of these guard conditions would require a separate unit test to ensure that it works and to avoid regressions should it be accidentally removed in the future.</p>
<p>Finally, we need the method to process the order. Given our domain model above, this class needs to contain some conditional check to ensure that all the workflow requirements are satisfied before processing of the order can commence. Something like:</p>
<pre class="brush:scala">
def processOrder(workflow: CheckoutWorkflow) = workflow match {
case CheckoutWorkflow(Some(basket), Some(customer), Some(paymentMethod)) => {
// Do the order processing
}
case _ => throw new IllegalStateException("Workflow requirements not satisfied")
}
</pre>
<p>None of the above is obviously ideal as there are a number of places that have the potential to error at runtime. The conditionals pollute our code with non-business logic. Also, we have to write good unit tests to ensure all the conditionals are working correctly and have not been accidentally removed. Even then, any client may call our code having not met the requirements encoded in the conditionals and they will receive a runtime error. Let's hope they unit test as thoroughly as we do! Surely we can do better than this?</p>
<h3>A Less Than Ideal Solution</h3>
<p>Well, we could use the approach outlined in my <a href="http://skipoleschris.blogspot.com/2012/11/eliminating-conditionals-using-types.html">previous post</a> and use domain model extensions:</p>
<pre class="brush:scala">
case class WorkflowWithBasket(basket: Basket)
case class WorkflowWithBasketAndCustomer(basket: Basket, customer: Customer)
case class WorkflowWithAllRequirements(basket: Basket,
customer: Customer,
paymentMethod: PaymentMethod)
def processOrder(workflow: WorkflowWithAllRequirements) = {
// Do the order processing
}
</pre>
<p>While this does allow removal of all the conditionals and associated tests, it unfortunatley also reduces the flexibility of our model quite significantly in that the order that the workflow must be processed is now encoded into the domain model. Not ideal. We'd like to keep the flexibility from the first solution but in a type safe way. Is there a way that we can encode the requirements into the type system?</p>
<h3>Levaraging The Type System</h3>
<p>First, lets consider what we want to achieve. Our aim to encode unsatisfied and satisfied workflow requirements and only allow methods to be called when the correct combinations are set. So, let's first encode the concept of requirements:</p>
<pre class="brush:scala">
trait BasketRequirement
case object UnsatisfiedBasketRequirement extends BasketRequirement
trait CustomerRequirement
case object UnsatisfiedCustomerRequirement extends CustomerRequirement
trait PaymentMethodRequirement
case object UnsatisfiedPaymentMethodRequirements extends PaymentMethodRequirement
</pre>
<p>Here we have defined requirement traits for each of the different workflow stages. We also defined case objects to represent the unsatisfied state of each requirement. Next we need to indicate the satisfied states, which are our actual domain object classes:</p>
<pre class="brush:scala">
case class Basket(items: List[LineItem]) extends BasketRequirement
case class Customer(id: String) extends CustomerRequirement
case class PaymentMethod(paymentType: PaymentType) extends PaymentMethodRequirement
</pre>
<p>Next job is to make sure that our workflow object can represent these requirements and be strongly typed on either the satisfied or unsatisfied state. We do this by adding type bounds to each of the workflow types. This also allows us to eliminate the need for the Option types. We also define an 'unsatisfied' instance as the starting point for our workflow:</p>
<pre class="brush:scala">
case class CheckoutWorkflow[B <: BasketRequirement,
C <: CustomerRequirement,
PM <: PaymentMethodRequirement]
(basket: B, customer: C, paymentMethod: PM)
object CheckoutWorkflow {
val unsatisfied = CheckoutWorkflow(UnsatisfiedBasketRequirement,
UnsatisfiedCustomerRequirement,
UnsatisfiedPaymentMethodRequirements)
}
</pre>
<p>Now we need the functions that actually process each individual workflow stage. Note how each one defines type bounded parameters for the things it doesn't care about. However, for the stage that it actually manipulates it requires that it is called with the Unsatisfied type and returns the Satisfied type. Thus, you can no longer call any of these methods with a workflow that already has that stage satisfied: the compiler won't allow it:</p>
<pre class="brush:scala">
def processBasket[C <: CustomerRequirement, PM <: PaymentMethodRequirement]
(workflow: CheckoutWorkflow[UnsatisfiedBasketRequirement.type, C, PM]):
CheckoutWorkflow[Basket, C, PM] = {
// Do some processing...
workflow.copy(basket = basket)
}
def processCustomer[B <: BasketRequirement, PM <: PaymentMethodRequirement]
(workflow: CheckoutWorkflow[B, UnsatisfiedCustomerRequirement.type, PM]):
CheckoutWorkflow[B, Customer, PM] = {
// Do some processing...
workflow.copy(customer = customer)
}
def processPaymentMethod[B <: BasketRequirement, C <: CustomerRequirement]
(workflow: CheckoutWorkflow[B, C, UnsatisfiedPaymentMethodRequirements.type]):
CheckoutWorkflow[B, C, PaymentMethod] = {
// Do some processing...
workflow.copy(paymentMethod = paymentMethod)
}
</pre>
<p>Finally, our processOrder function becomes super simple. You just can't call it any more unless it has no Unsatisfied types:</p>
<pre class="brush:scala">
def processOrder(workflow: CheckoutWorkflow[Basket, Customer, PaymentMethod]) = {
// Process the order
}
</pre>
<p>One observation that was made to me by someone (a Java dev) who looked at my initial draft of the code was that the final solution looks more complex due to all the type bounds and that there's actually more classes due to the need to define the requirement traits and the Unsatisfied state objects. However, don't forget that this solution eliminates at least four conditional blocks and associated unit tests, simplifies others and also possibly reduces the number of tests that clients need to write on their code as well. Also, there's no possibility of runtime failure. If the code compiles then we have a much higher confidence that it will work. Well worth a tiny bit more type complexity in my book.</p>
Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com4tag:blogger.com,1999:blog-5400765672405463592.post-22670834592006118412012-11-21T08:47:00.000+00:002012-11-21T08:47:25.955+00:00Eliminating Conditionals Using Types<p>I've been at <a href="http://skillsmatter.com/event/scala/scala-exchange-2012">Scala eXchange</a> for the last two days. One of the topics that came up a couple of times was how the type system can be used to encode knowledge and rules, making it possible for the compiler to enforce these rather than needing unit tests to verify them. There were questions from some about how this actually works in practice, so I thought I'd produce some blog posts to demonstrate. This first one covers the most simple case.</p>
<h2>Eliminating Conditionals Using Types</h2>
<p>Consider the following (highly simplified) domain model:</p>
<pre class="brush:scala">
case class CreditCheckResult(statusCode: String, creditLimit: Int)
case class Customer(id: Long, name: String, creditCheckStatus: Option[CreditCheckResult])
</pre>
<p>This model indicates that a Customer may or may not have completed a credit check. However, we only want to register the customer if they have successfully passed through a credit check. The registration function therefore looks something like:</p>
<pre class="brush:scala">
def register(customer: Customer): Either[String, String] = customer.creditCheckStatus match {
case None => Left("Attempt to register non-credit checked customer")
case Some(ccs) => {
// Do the registration here
Right(registrationCode)
}
}
</pre>
<p>This is a very common pattern in any places where an Option (or in Java a null) is used to indicate some conditional state (rather than a truly optional piece of data). The result is that we now need a unit test for both conditional cases:</p>
<pre class="brush:scala">
class RegistrationSpec extends Specification {
"The registration process" should {
"register a credit checked customer" in {
// Test code here
}
"fail to register a non-credit checked customer" in {
register(Customer(1L, "Test")) must_== Left("Attempt to register non-credit checked customer")
}
}
}
</pre>
<p>However, we can use the type system to completely remove the need for this conditional check and thus remove an entire unit test case from our code base. How do we do this? By representing the different states that a customer can be in with different types:</p>
<pre class="brush:scala">
case class CreditCheckResult(statusCode: String, creditLimit: Int)
trait Customer {
def id: Long
def name: String
}
case class UncheckedCustomer(id: Long, name: String) extends Customer
case class CreditCheckedCustomer(id: Long, name: String, creditCheckStatus: CreditCheckResult) extends Customer
</pre>
<p>Our registration method can now be greatly simplified:</p>
<pre class="brush:scala">
def register(customer: CreditCheckedCustomer) = {
// Do the registration here
}
</pre>
<p>And our test now needs to cover only the valid registration case:</p>
<pre class="brush:scala">
class RegistrationSpec extends Specification {
"The registration process" should {
"register a credit checked customer" in {
// Test code here
}
}
}
</pre>
<p>Aaaahaaa, I hear you cry, but doesn't the complexity instead move to the creation of the correct type of Customer? In some cases this might be true, but you could easily validate this by constructing customers using this code as part of the above test. Hopefully you will be using a functional programming style and you will already have a function somewhere that takes an UncheckedCustomer and transforms them into a CreditCheckedCustomer as part of the credit check process: which will already by type safe and tested!</p>
<p>I'll add some more examples of using types to reduce code and unit testing over the coming week or two.</p>
Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com3tag:blogger.com,1999:blog-5400765672405463592.post-69984951787904336742012-09-14T08:32:00.000+01:002012-09-14T08:32:34.555+01:00The Next Innovation<p>Here's a little comic strip I put together ...</p>
<img src="http://templemore.co.uk/productplanning.jpg" width="680"/>
<p>When you have created so many innovative products, it's always going to be difficult to come up with enough differentiation between one version of a product and the next to keep getting people to spend their cash. Creating an entirely new product line is even more of a challenge. I'm wondering where our favourite fruit named company is going to go next.</p>
Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com1tag:blogger.com,1999:blog-5400765672405463592.post-53911414258052596212012-07-17T20:57:00.000+01:002012-07-17T20:57:53.927+01:00Unit Testing with MongoDB<p>Back in June I presented at the London Scala User Group (<a href=”http://skillsmatter.com/podcast/home/experiences-real-world-scala”>video here</a>) on our experiences on a recent Scala project that used MongoDB.</p>
<p>One of the questions I was asked was how we go about mocking MongoDB for testing purposes (especially unit testing)? My answer was that we don’t! This raised a couple of eyebrows in the audience, but we’ve found this works just fine for us.</p>
<p>In this blog post I will explore our thoughts and reasoning in a bit more detail.</p>
<h3>How do we test with MongoDB?</h3>
<p>I’ll be honest: for the majority of our tests (even the unit ones) we just run against a local MongoDB instance. When we are testing repository classes we write tests against a real repository wired up against a real MongoDB instance. We assert against the contents in the MongoDB collections. In some cases, we even test service classes or even controllers with a live repository backend that writes to and retrieves from a real MongoDB instance.</p>
<p><i>“But you can’t do that!”</i>, I hear you cry. Traditional thinking says that writing tests directly against the database will be too slow. We’ve actually found MongoDB (especially when running locally) to be so blindingly fast that we don’t need to consider doing anything else. For most updates and queries we tend to get sub-millisecond responses.</p>
<p>It’s also a hell of a lot easier and less fragile to back tests with real repository instances than it is to create and configure a load of mocks or strange database testing frameworks.</p>
<p>I’m pragmatic. If using a real database backend was making our tests run too slow or causing them to be too complex then I’d happily consider alternatives. Our experience with MongoDB is that it is both fast enough and that it’s query syntax is succinct enough that we don’t need to bother with those alternatives.</p>
<h3>Do we use mocks at all for the database then?</h3>
<p>Actually, I rarely use mocks in any of my Scala project now days. I’ve switched much more towards a style of isolating my calls to dependencies and then using mixin traits to override specific behavior for testing. (I’ll write a blog post about this approach soon).</p>
<p>From time to time we do need to write tests that make a repository dependency exhibit a specific behavior which is difficult to emulate against an actual database. In these cases we just use a custom test trait mixed into the repository in order to override methods to exhibit the behavior that we require. Keeps it simple and makes the tests much more readable and much less fragile than using mocks or other frameworks.</p>Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com0tag:blogger.com,1999:blog-5400765672405463592.post-3383765639850204932012-07-12T13:47:00.000+01:002012-07-12T13:47:17.571+01:00Making Agile Go Fast<p>Just recently I was speaking with a potential customer who had concerns about the velocity of their Scrum teams. They had recently (9 months ago) switched to an agile approach and had expected to get much more delivered than they had.</p>
<p>Now they were looking in to scaling up the number of Scrum teams on the project to try and increase velocity and deliver more quickly. In my experience of many agile projects, increasing the number of people on the project, especially one that is struggling, never has the desired effect of making things go faster. The advice to this customer was to therefore look at making their existing teams and process more effective rather than increasing the number of people on the project.</p>
<p>This got me thinking about what makes agile projects go slow. Across a number of projects that I have experienced, I’ve identified four common traits: two related to people and process and two related to technical concerns.</p>
<h3>The Wrong People</h3>
<p>Agile is all about using the right people. In fact this is true for any project or process, but for agile it is even more important. Use the right people and the agile process will work. Use the wrong ones and you will struggle.</p>
<p>The people who you don’t want on your agile project are those who are not disciplined enough to follow sound engineering practices and those who try to build complex frameworks/toolkits and who conduct personal science projects on your codebase. If you want to go fast then it’s vital that you have a clean, simple and well-refactored codebase that is backed up by a set of quality tests. Team members who don’t support this vision slow down the whole team.</p>
<p>To get back on track, remove those devs who aren’t following good engineering discipline and those who keep making things more complex. Bring in experienced agile developers to pair with the remaining team, giving them the responsibility of ensuring quality engineering practices are followed.</p>
<h3>Putting The Process First</h3>
<p>Another common mistake of companies starting out with agile projects is that they assume the process will make them go faster. Sorry to disappoint, but this is just not true. What makes agile teams go fast is creating open channels of communication and self organising and empowered teams.</p>
<p>If an organisation can’t create an excellent environment for communicating then no agile process will help. Agile helps teams go faster by eliminating the unnecessary documentation and waterfall steps. However, this information transfer must still happen and it happens through closer communication and collaboration.</p>
<p>Sadly, many organisations create agile teams and give them a process to follow but then insist on telling them how to organise, what tools and techniques to use, how to work and exactly what to do. This isn’t agile, it’s just another way of organising a traditional team. If you want agile to go fast then the team need total control over how they work and what they do.</p>
<p>To get back on track from here, relax on the process. Work on increasing communications. Give the agile teams the chance to decide how they will organise and work. Give them control and they will go faster. Agile is not a process, it’s about letting people get the job done.</p>
<h3>Using The Wrong Technologies</h3>
<p>So many agile projects struggle because they are using the wrong technologies. When this is the case it’s usually because the technology approach has been foisted on the team by an ‘ivory-tower’ architect or some senior manager who was wooed by the sales consultant from a large vendor. In other cases it may be that the team are forced into using a ‘corporate standard’ technology set.</p>
<p>In many cases the team will repeatedly identify the technology as a major factor in retrospectives. However, more often than not they will be told that this is not something that can be changed and that they have to stick with it. Why throw away hundreds of thousands or even millions on staff costs when backtracking on some poor technology choices would make so much difference.</p>
<p>Again, giving the teams the authority to make technology decisions is essential. Allowing technology choices to be reversed if they prove to be hindering delivery is also an important part of being agile.</p>
<h3>Making The Architecture Too Complex</h3>
<p>Many software systems are designed, or become, far too complex. In an agile environment this is a guaranteed way of making things go slower. Software must be kept as simple as possible so that it is easy to continually change and enhance.</p>
<p>If the architecture of an agile system has become so complex that it hinders development then it must be revisited and greatly simplified. Layers must be removed, interfaces reduced and code and technologies radically refactored to make everything simpler. Once that is done then the project must work all out on stopping it from getting more complex again in the future.</p>
<p><br/><b>Only by using the right people, correctly empowered and communicating can the team have the ability to go fast and be truly agile. Only by making the correct technology choices, reversing bad technology choices and keeping everything simple can the ability to turned into real agile delivery.</b></p>Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com0tag:blogger.com,1999:blog-5400765672405463592.post-10349809345813708412012-06-22T13:22:00.001+01:002012-06-22T13:22:31.093+01:00Cucumber JVM Support for SBT<p>I am happy to announce release 0.5.0 of my xsbt plugin for cucumber. This is a major new update that finally switches from using the Ruby version of Cucumber (running under JRuby) to the newly released Cucumber-JVM.</p>
<p>Full details of how to use the plugin can be found on the github page: <a href="https://github.com/skipoleschris/xsbt-cucumber-plugin">https://github.com/skipoleschris/xsbt-cucumber-plugin</a></p>
<p>A list of significant changes for this release include:</p>
<ul>
<li>Switched to cucumber-jvm</li>
<li>Less configuration options due to no longer requiring JRuby</li>
<li>Improved control of output options</li>
</ul>Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com0tag:blogger.com,1999:blog-5400765672405463592.post-90358616448564584412012-06-10T14:55:00.001+01:002012-06-10T14:56:31.839+01:00Pairing is Wearing<p>In <a href="http://skipoleschris.blogspot.co.uk/2011/08/power-of-pairs.html">this post</a> back in August 2011 I was talking about the many benefits of Pair Programming. Having now been full-time pair programming for almost an entire year, I wanted to mention one of the downsides that I have discovered.</p>
<h2>A Holiday</h2>
<p>I've just finished my holiday. It wasn't the best one ever, mainly due to camping in the UK in June: which meant heavy rain and strong winds. Still, we had a good time and it was very relaxing.</p>
<p>As the week progressed and I had a chance to relax, unwind and get some lie-ins I started to discover that my mind was starting to become more alert. I was feeling less tired. Great ideas were starting to flow again. It was at this point I realised just how weary I had been at the start of the holiday. I had known I was feeling pretty knackered, just not quite how much.</p>
<p>This then got me thinking as the the possible cause of this tiredness and how it could gradually build up over time without me being fully aware of it. I looked back at what I've been doing for the past year...</p>
<ul>
<li>I've been travelling a lot to get to/from work - but less than my previous job.</li>
<li>I've been learning lots of new stuff - but I'm a compulsive learner, so nothing new here.</li>
<li>We've put at least 3 major releases live - but I've put plenty of software live before.</li>
<li>We've been pairing full time - this is new!</li>
</ul>
<h2>Is Pairing More Tiring?</h2>
<p>The conclusions that I have had to draw is that working in a paired environment is more mentally tiring that working alone or individually within a team. There's more of a requirement to stay focused when working as a pair - it's particularly noticable when one member of the pair is feeling unwell or tired. Also while pairing you are multitasking more - writing code, thinking about solutions but also communicating your thoughts and maintaining a running dialogue. All of this extra work requires extra mental energy and thus increases fatigue.</p>
<p>So, how can we pair over a period of time without burning out? Here's some thoughts that I hope to experiment with over the coming months:</p>
<ul>
<li>Take more frequent and longer breaks than you would do while working individually. The productivity gain of working in pairs easily allows this to be supported</li>
<li>Do a lot more walking at lunchtime!</li>
<li>Try to provide team members regular tasks that they can undertake outside of a pair to give their mental energy drain a rest. Stuff that doesn't require extensive multitasking.</li>
</ul>
<p>Anyone else got any thoughts or ideas?</p>Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com1tag:blogger.com,1999:blog-5400765672405463592.post-42089978955377647082012-04-30T17:50:00.000+01:002012-04-30T17:50:01.108+01:00Life Without Objects<p>Over the last few years I have been learning and experimenting with a wider range of programming languages. In particular I have started using Scala as my primary development language, adopting a more and more functional style. I have also become very interested in Haskell (a pure functional language) and Clojure (a modern lisp dialect).</p>
<p>I have therefore been moving away from the object-oriented development principles that have made up the bulk of my 17 year career to date. More and more I am beginning to feel that objects have been a diversion away from building concise, well structured and reusable software.</p>
<p>As I pondered on this topic, I realised that this isn’t a sudden switch in my thinking. The benefits of objects have been gradually declining over a long period of time. The way I use objects today is very different to how I used them when they were new and shiny. In this post I explore this change in my thinking about object-oriented development.</p>
<h3>The Promise of Objects</h3>
<p>Back in the 1990s, objects were new and exciting. The promise of being able to create reusable classes built around common design patterns was seductive. The ability to then combine these classes into reusable and configurable business components seemed like the Mecca of software development. New languages like C++ and then, slightly later, Java held the promise of a new way of building great software.</p>
<p><b>Business Components Aren’t Reusable</b><br/>
It didn’t take us long to discover that the ability to create reusable business components was just a giant fallacy. Each business is significantly different from another, even in the same industry. Each similar project has very different business rules.</p>
<p>The only way to build reusable business components at this level is to make them hyper-configurable by adding such things as rules engines and embedded scripting languages. Hardly a component model and more like a bloatware model. This promise gone, people either buy into the big bloatware systems (suckers) or build their custom business objects on a project by project basis.</p>
<p><b>Patterns Don’t Build Well Structured Software</b><br/>
The next thing we learnt was that excessive reliance on design patterns doesn’t lead to the good software structure. Instead it leads to software that is overly complex, hard to understand and difficult to maintain. Some patterns even turned out to be anti-patterns (the singleton pattern makes software almost impossible to unit test, for example).</p>
<p>We soon learnt to use patterns judiciously. More often than not it’s just cleaner to code the software as you understand the model rather than try to abstract it into a more generalised pattern.</p>
<p><b>Frameworks for Class and Component Reuse Give Few Benefits</b><br/>
Another early promise of objects was rich, tightly coupled frameworks of classes which when used together would make building applications from reusable component a breeze by hiding all the technical complexity and implementation plumbing. Think EJB and similar. Experience soon showed that these just did not work. They were just too restrictive and cumbersome for what people were trying to achieve.</p>
<p>These heavy-weight frameworks soon died out to be replaced with more lightweight libraries and toolkit type approaches. Collections of more loosely coupled classes that you can pull together as needed are now the preferred way to build software. Take just what you need and nothing more.</p>
<p><b>Inheritance Creates Brittle Software</b><br/>
The ability to support interface and implementation inheritance was one of the key tenets of object oriented software development. We could spot common code and behaviour and push this down into a shared base class so that future abstractions could benefit from having this shared code available to build on.</p>
<p>Sadly, this just didn’t work out well. Each sub-class turns out to be subtly different from its peers, resulting in lots of overrides of base class behaviour or making the base classes even more generic. The net result was super-fragile software, where any small changes to a common base class would break most, if not all, of the sub-class implementations.</p>
<p>These days we don’t use inheritance much, and especially not for creating technical base classes. Its use is pretty much restricted to interface inheritance to indicate an object supports a certain behaviour or to occasional domain models where there is a true inheritance relationship. Other than that we tend to extract commonality in to separate ‘mixin’ type classes and compose them together through an aggregation approach.</p>
<p><b>Violation of Encapsulation</b><br/>
Another key feature of the object-oriented model was the ability to encapsulate state and then expose behaviours (via methods) that access and update this state. Unfortunately it turns out that there are a large number of cases, where we are actually interested in the vales of the state rather than the behaviour.</p>
<p>For example, asking an object to render itself as HTML turns out to be a pretty poor approach. Knowledge of HTML rendering gets spread across the code base and a small change in approach causes us to change many, many classes. Instead we tend to pass the object to a dedicated HTML rendering/template component, which pulls the data values from the object.</p>
<p>Anti-patterns have even emerged around this to allow us to have light-weight objects that just encapsulate state without behaviour (Java Beans, Data Transfer Objects and so on). If we are doing this, then why not just work directly with first-class structured data as opposed to objects?</p>
<p><b>Mutable State Causes Pain</b><br/>
<p>Another perceived benefit of encapsulation was the ability to mutate the state of an object instance without impacting on the clients that use that object. However, anyone who has built a significant sized object-oriented system can tell you stories of trawling through many files of code to find the location that mutated the state of an object to an unexpected value that happened to make your software blow up in a completely different place (usually where you output or store the state of that object).</p>
<p>More and more we now favour immutable state and stateless services so that these problems do not occur. There’s also the additional benefit that immutable state is a much better model for building highly concurrent systems and for getting the most out of modern multi-core hardware. It’s also far easier and less error prone than trying to work with threads, locks and concurrency safe data structures.</p>
<p><b>Behavioural Interfaces Cause Code Complexity and Bloat</b><br/>
One of the things we do frequently in object-oriented languages is create small marker interfaces that have just a single method. Any class wanting to support this behaviour extends the interface and implements the method. We can also declare anonymous implementations for ad-hoc use.</p>
<p>However, we have found that neither of these approaches are particularly good. Implementing the marker interfaces often pollutes classes with implementations that are not their direct concern (thus violating the single responsibility principle). Anonymous classes are just unnecessary bolierplate that makes our code more difficult to understand and maintain.</p>
<h3>Life Without Objects</h3>
<p>So, is it possible to go back on 17 years of experience and contemplate a life without objects? I’m not sure that I’m 100% there just yet, but using a multi-paradigm language like Scala is allowing me to overcome many of the limitations of the object-oriented approach.</p>
<p>For example, Scala’s support for mixin traits makes it almost unnecessary to ever use implementation inheritance. It’s rich collections framework plus the ability to use case classes to create data structure like concepts obviates working around encapsulation issues. A recommendation to use immutable data and collections makes code easier to debug and reason about. The ability to use functions as general abstractions and type classes to extend behaviour while maintaining single responsibilities makes it much easier to build well structured, reusable software.</p>
<p>In fact, what I find I am doing more and more is using simple objects in the form of case classes to represent data structures, with a few behavioural methods to simplify working with this data. Then I’m just using mixin traits as a modular approach for grouping related functions together. Then I’m combining these together to form components in which I compose together various functions that transform data from one for into another.</p>
<p>Perhaps I’m further away from the pure object-oriented approach than I’d thought. I’m certainly building smaller, cleaner and better structured software than I ever was before.</p>Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com51tag:blogger.com,1999:blog-5400765672405463592.post-26561104257095862212012-03-26T08:46:00.000+01:002012-03-26T08:46:16.103+01:00Option<p>In my <a href="http://skipoleschris.blogspot.co.uk/2012/03/map-flatten-and-flatmap.html">previous post</a> I introduced Dave and we looked at map, flatten and flatMap. In this new episode we take a look at the Scala Option type:</p>
<iframe width="420" height="315" src="http://www.youtube.com/embed/3CWJgRUXTDw" frameborder="0" allowfullscreen></iframe>
<p>Here's some code...</p>
<pre class="brush:scala">
case class Person(name: String, partner: Option[Person])
sealed trait Product
case object XBox extends Product
case class WeekendAway(person1: Person, person2: Person) extends Product
type Cash = Int
case class Purchase(product: Product, change: Cash)
def buyXBox(cash: Cash): Purchase = Purchase(XBox, cash - 20000)
def bookWeekendAway(p1: Person, p2: Person, cash: Cash): Purchase = Purchase(WeekendAway(p1, p2), cash - 15000)
val lucy = Person("lucy", None)
val dave = Person("dave", Some(lucy))
//val dave = Person("dave", None)
val wages = 50000
val purchase = dave.partner map (bookWeekendAway(dave, _, wages)) getOrElse buyXBox(wages)
println(purchase)
</pre>Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com0tag:blogger.com,1999:blog-5400765672405463592.post-70051984391471395772012-03-20T08:03:00.000+00:002012-03-20T08:03:24.080+00:00Map, Flatten and FlatMap<p>I've recently been working with some developers who are new to the Scala world. They've picked up the language really easily and in only a few week have become competent Scala developers. Who says it's too complex and difficult to learn?</p>
<p>Anyway, one of these developers is a visual thinker such as myself. We prefer communication through diagrams rather than words. While explaining Scala I've been drawing some pictures to illustrate some of the more interesting concepts. She suggested that I should publish these and we had the idea that an animation would be fun.</p>
<p>Here's my first animated post showing the concepts for Map, Flatten and FlatMap. It's a pretty amiturish stab at animation, but hopefully it explains the concepts while also adding a bit of humour along the way.</p>
<iframe width="420" height="315" src="http://www.youtube.com/embed/_7tTpQbRPos" frameborder="0" allowfullscreen></iframe>
<p>If you are interested here's the Scala code for the concepts covered in the animation:</p>
<pre class="brush:scala">
case class World[+A](revolvesAround: A) {
def map[B](f: A => B): World[B] = World(f(revolvesAround))
def flatten[B](implicit ev: A <:< World[B]): World[B] = World(revolvesAround.revolvesAround)
def flatMap[B](f: A => World[B]) = World(f(revolvesAround).revolvesAround)
def get: A = revolvesAround
}
def lift[A](entity: A): World[A] = World(entity)
case class Person(name: String) {
def smite = DeadBody(name)
}
case class DeadBody(usedToBe: String)
case class Insect(species: String)
def kill(p: Person): DeadBody = p.smite
def reincarnate(b: DeadBody): World[Insect] = World(Insect("ant"))
val dave: Person = Person("Dave")
val antDave: Insect = lift(dave) map kill flatMap reincarnate get
println(antDave)
</pre>Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com4tag:blogger.com,1999:blog-5400765672405463592.post-47975177841117000402012-03-14T08:22:00.000+00:002012-03-14T08:30:48.667+00:00Done and Dusted<p>In a recent retrospective our team dug out our <b>Definition of Done</b> and gave it a good review. We ended up removing a number of unnecessary items. This got me thinking about what is Done?</p>
<p>In previous places that I have worked the <b>Definition of Done</b> was a long check-list of things that had to have been completed and verified. Is this the best that a philosophy that claims to value "individuals and interactions over processes and tools" can come up with? This got me thinking...</p>
<p>Isn’t Done actually just the state when the team agree that the story is complete? Why should it need to be any more than that?</P>
<div style="border: solid #888888 1px; background-color: #CCCCCC; width: 80%; margin-left: auto; margin-right: auto; color: black; padding: 10px">
Done: The state when all team members agree that the story is complete
</div>
<p>We could expand this slightly to define the roles that the different team members play in this definition:</p>
<ul>
<li>Developers agree that the code is complete, tested and refactored to an acceptable standard</li>
<li>QAs agree that the acceptance criteria for the story has been met, that the functionality works and that they have not found any defects introduced by the new feature</li>
<li>BAs and POs agree that the feature meets the business objectives for the story</li>
<li>(we could also add points for DBAs, Ops etc, as needed)</li>
</ul>
<p>So, if we accept that the team can decide for themselves when something is Done, why do we end up with long checklists, often including a whole range of metrics? It could be because 'management' aren’t able to release their need to monitor everything and produce nice reports. However, it’s more likely a matter of trust.</p>
<p>Teams are just not trusted to self-organise and make good decisions. If the team say that a story is Done then for many managers there is an immediate suspicion that the team is not telling the truth and is trying to cut corners. They then require that there is concrete proof that this is not the case. This is despite the fact that a group of professionals all happen to agree that it is Done. It just doesn’t make sense.</p>
<p>If the team has never released a story as Done when it clearly wasn’t then why doubt they will suddenly stop doing this in the future? Show trust in the team that is delivering completed work.</p>
<p>If the team are marking stories as Done when they clearly aren’t then there is a much more serious problem with the team and they probably either need replacing, mentoring or a firm guiding hand. Adding a requirement to satisfy a long checklist and provide associated metrics will never improve a bad team.</p>
<p>As with many things agile, it’s not about the process: it’s about changing attitudes; it’s about trusting teams to deliver the best solutions possible; it’s about not bogging the teams down with unnecessary controls.</p>Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com0tag:blogger.com,1999:blog-5400765672405463592.post-53150718910678933752012-02-13T18:01:00.000+00:002012-02-13T18:01:22.364+00:00Sometimes you just have to be lazy...<p>...or, solving problems with lazy evaluation.</p>
<p>I’ve recently been working on a simple template system, which also has the potential to grow into a full-blown document generation system. The basic principle is that a document is composed of a number of fragments. Each fragment is called in turn to generate its piece of the document and these pieces are concatenated together to product the final result.</p>
<p>My first implementation was a super simple one. We just fold over the list of fragments, accumulating the output from each into a string builder. Each fragment may alternatively generate an error, which we collect along the way as well. At the end of the fold we return either the string document or a list of errors.</p>
<pre class="brush:scala">
def render(fragments: List[Fragment]): Either[List[String], String] = {
fragments.foldLeft(RenderResult()) { (acc, fragment) =>
fragment.generate fold (
{ err => acc.copy(errors = err :: acc.errors) },
{ text => acc.doc.append(text); acc }
)
}.toEither
}
case class RenderResult(doc: StringBuilder = new StringBuilder(), errors: List[String] = List()) {
def toEither: Either[List[String], String] = ...
}
</pre>
<p>Now, this works great for documents of only a few kilobytes in size. However, as the documents grow to multi-megabyte sizes this approach becomes infeasible. The amount of memory required to hold the document being generated becomes huge. Multiple concurrent generations become impossible. We need a better solution.</p>
<p>The typical suggestion at this point is to stream output from each fragment straight into an output writer of some form rather than collect and build a string.</p>
<pre class="brush:scala">
def render(fragments: List[Fragment], writer: Writer): Option[List[String]] = {
fragments.foldLeft(List[String]()) { (acc, fragment) =>
fragment.generate fold (
{ err => err :: acc.errors },
{ text => writer.write(text); acc }
)
}.getErrors
}
</pre>
<p>Unfortunately this doesn’t work here because of the possibility that fragments may generate errors. We don’t want to stream out partially complete documents. We could output to a temporary file and then copy this to the ultimate output on success, but this seems like a less than ideal solution.</p>
<p>A common approach in an imperative style would be to first run a check phase to see if there are any errors and then run a second render phase to produce the output.</p>
<pre class="brush:scala">
def check(fragments: List[Fragment]): List[String] =
fragmens.foldLeft(List[String]()) { (acc, fragment) =>
fragment.checkForErrors map (_ :: acc) getOrElse acc
}
def render(fragments: List[Fragment], writer: Writer): Unit =
fragments foreach { fragment => writer.write(fragment.render) }
</pre>
<p>However, in this case the building of fragments might be quite costly and we don’t want to have to process them once to see if there is an error and then again to render.</p>
<p>However, as we are using Scala and programming in a functional style, we have an alternative approach. Instead of returning a string or writing direct to a writer we can return a lazily evaluated function. This function can encapsulate all the side-effect generating logic of doing the actual write - which is also nice from the point of view of being able to reason about our rendering code.</p>
<pre class="brush:scala">
def render(fragments: List[Fragment]): Either[List[String], (Writer) => Unit] = {
fragments.foldLeft(RenderResult()) { (acc, element) =>
fragment.generate fold (
{ err => acc.copy(errors = err :: acc.errors) },
{ func => acc.copy(fragmentFuncs = func :: fragmentFuncs }
)
}.toEither
}
case class RenderResult(fragmentFuncs: List[() => String] = List(), errors: List[String] = List()) {
def toEither: Either[List[String], List[(Writer) => Unit] = {
...
(writer: Writer) => fragmentFuncs foreach { f => writer.write(f()) }
...
}
}
</pre>
<p>The way this works is that the render method folds over each fragment, asking the fragment to do enough work to be certain that no errors will occur (e.g. validate data, take a temporary copy of any volatile values and so on). However, it doesn’t need to do the actual string generation. Instead, each fragment returns a function that will be called at some time in the future (or an error). The render code then accumulates all these small functions into one larger function and returns this.</p>
<p>The returned function, when called, takes a writer and iterates over all the small string generating functions, writing the output of each straight into the writer. Job done and in a mere fraction of the memory required by the original solution.</p>
<p>In cases where you need to generate large amounts of data or where you want to delay side effecting execution to a well-defined location in your code then lazy evaluation is a great approach to follow.</p>
<p>NOTE: All above code example are <b>significantly</b> simplified from the real code in order to demonstrate the approach.</p>Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com2tag:blogger.com,1999:blog-5400765672405463592.post-80797801561278256292012-02-04T21:17:00.001+00:002012-02-04T21:17:52.716+00:00Some TDD Advice<p>I’ve been doing some interviewing of new developers. Our company approach is to undertake a small pairing exercise with the candidate. We pick a relatively simple code kata and ask them to complete this. They drive the keyboard and do most of the work, while we, the interviewer, observe, ask questions, help point out obvious problems and provide assistance if necessary.</p>
<p>The only main requirement for these pairing sessions is that all work is done using a Test-Driven Development (TDD) approach. We do this for two reasons:</p>
<ol>
<li>We TDD all our code and therefore want to be comfortable that the candidate can work in this way.</li>
<li>Building a system gradually using tests gives us a good insight into how the person thinks, how they go about solving problems and how they evolve their code.</li>
</ol>
<p>Unfortunately, we are finding that many of the candidates are struggling with these sessions. They tend to start out strongly and then a few tests into the problem they become bogged down or stuck on furthering the solution.</p>
<p>I found this slightly puzzling, so I decided to undertake one of the katas following the exact same approach as we expect the candidates to follow. I picked a kata from one of the other devs that I had not seen before, so as to not prejudice my result. I started small with a simple test and gradually evolved the solution one test at a time. I was able to complete the kata easily within the allotted interview time and the code I produced was (IMHO) very clean and well structured.</p>
<p>While doing this I noticed one major difference between my approach and that of the people I have been interviewing. I follow a cycle as follows: write the test -> implement the code to make the test pass -> refactor the code to make it better -> move on to the next test. The devs that I have been interviewing who struggle miss out one of these key steps. Namely, they do: write the test -> implement the code to make the test pass -> move on to the next test. They miss the refactoring step at the end of each cycle.</p>
<p>What this tends to result in is that they easily build a first pass at some code to make a test pass. Then they build the code to pass the next test on top of this. This works for three of four iterations - but by this time their code is becoming quite complex and messy. Then as the requirements get more complex they struggle to change their code, understand how it actually works and how to change it to pass the next test.</p>
<p>At this point they either become so bogged down that they fail to make any progress, or they start a massive refactoring exercise that breaks loads of code and most of their previously working tests. Either way, they pretty much fail the interview at this point.</p>
<p>By adding a small refactor step into every cycle we ensure that the code we implement to pass the next test is always built on a solid foundation of well structured, clean code from the previous one. Doing this would make so much difference to those interview candidates.<p>
<p>Let’s finish by looking at an example of the difference this approach makes:</p>
<p>In this example we are working on a kata to build a speaking clock. For example: 0:00 speaks 'midnight', 15:30 speaks 'half past three', 11:45 speaks 'quarter to twelve' and so on. After writing code to pass the first couple of tests we have something like:</p>
<pre class="brush:scala">
class SpeakingClock {
def speak(time: DateTime): String ={
val hour = time.hourOfDay().get()
val minute = time.minuteOfDay().get()
if ( hour == 0 && minute == 0 ) "midnight"
else if ( hour == 12 && minute == 0) "noon"
else "unknown"
}
}
</pre>
<p>The next tests would be to cover a case for speaking the o'clock and half past times. Our candidate would most likelky go on to just add to the above code, something like:</p>
<pre class="brush:scala">
class SpeakingClock {
def speak(time: DateTime): String ={
val hour = time.hourOfDay().get()
val minute = time.minuteOfDay().get()
if ( hour == 0 && minute == 0 ) "midnight"
else if ( hour == 12 && minute == 0) "noon"
else if ( minute == 0 ) hourToText(hour) + " o'clock"
else if ( minute == 30 ) "half past" + hourToText(hour)
else "unknown"
}
}
</pre>
<p>Then we start adding tests covering quarter past and quarter to. At this point the algorithm starts to get more complex and the if/else block starts growing out of control. Refactoring this large if/else block becomes a major exercise under interview conditions and it is at this point that the solution invariably goes wrong and the candidate starts to struggle.</p>
<p>Now, by doing some refactoring after each step, we can change the first piece of code to something like:</p>
<pre class="brush:scala">
class SpeakingClock {
private val FixedTimes = Map[(Int, Int), String]((0, 0) -> "midnight",
(12, 0) -> "noon")
def speak(time: DateTime): String = {
val hour = time.hourOfDay().get()
val minute = time.minuteOfHour().get()
FixedTimes get ((hour, minute)) getOrElse {
"unknown"
}
}
}
</pre>
<p>Now we don't need to worry about the first case any more. It's nice and clean and out of the way. Also, if there are any new fixed times (say 17:00 is 'dinnertime') then we can easily add these to the solution with minimal additional code. Next, we add in the support for speaking the o'clock times:</p>
<pre class="brush:scala">
class SpeakingClock {
private val FixedTimes = Map((0, 0) -> "midnight",
(12, 0) -> "noon")
private val HourNames = "twelve" :: "one" :: "two" :: "three" :: "four" :: "five" :: "six" ::
"seven" :: "eight" :: "nine" :: "ten" :: "eleven" :: Nil
private val HoursToNames = (0 to 11) zip HourNames toMap
def speak(time: DateTime): String = {
val hour = time.hourOfDay().get()
val minute = time.minuteOfHour().get()
FixedTimes get ((hour, minute)) getOrElse {
val adjustedHour = if ( hour > 11 ) hour - 12 else hour
val hourName = HoursToNames(adjustedHour)
"%s o'clock".format(hourName)
}
}
}
</pre>
<p>Then we again go through the refactoring step and end up with something like:
<pre class="brush:scala">
class SpeakingClock {
private val FixedTimes = Map((0, 0) -> "midnight",
(12, 0) -> "noon")
private val HourNames = "twelve" :: "one" :: "two" :: "three" :: "four" :: "five" :: "six" ::
"seven" :: "eight" :: "nine" :: "ten" :: "eleven" :: Nil
private val HoursToNames = (0 to 11) zip HourNames toMap
def speak(time: DateTime): String = {
val hour = time.hourOfDay().get()
val minute = time.minuteOfHour().get()
FixedTimes get ((hour, minute)) getOrElse {
"%s o'clock".format(nameForHour(hour))
}
}
private def nameForHour(hour: Int) = {
val adjustedHour = if ( hour > 11 ) hour - 12 else hour
HoursToNames(adjustedHour)
}
}
</pre>
<p>By pulling out the nameForHour method we leave code that is much better structured and cleaner. It's then so much easier to see what changes to make in order to pass the next test. By growing code in this way we end up with a working solution that is also always in a clean and tidy state.<p>
<p>So, the moral of this story is that if you are doing TDD, in an interview or not, don't forget to refactor after each test passes to ensure your code is well written before moving on to the next test.</p>Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com0tag:blogger.com,1999:blog-5400765672405463592.post-90613574999277886132012-01-11T17:31:00.000+00:002012-01-11T17:31:25.130+00:00Thoughts on the UK Government's New Approach To ICT<p>The UK Government today announced a new approach to teaching computing in schools. The current sylabus is based on teaching how to use applications like Word, Excel and PowerPoint. This is, quite rightly, seen as highly boring and unecessary for today's computer literate pupils. The new strategy is to switch to a more 'computer science' based sylabus. This would include more on how computers actually work and would also teach programming. The Government's aim is to create a new generation of highly skilled graduates with excellent computing skills.</p>
<p>While this new approach sounds very interesting and laudable, I have some serious doubts about whether it will actually work. My concern can been seen if you break down the types of pupils that will be studying this new computing sylabus into three broad categories:</p>
<h2>The Uninterested</h2>
<p>There is likely to be a (fairly large?) category of pupils who, like with any subject at school, aren't really interested. These will be the pupils for whom a computer is about Facebook, playing games and downloading music and videos. It's pretty likely the these pupils won't really be interested in programming and will thus find it very challenging and probably quite boring. However, the sylabus won't be allowed to leave them behind, so it will have to run at a fairly slow pace in order to keep them on board.</p>
<h2>The Swayable</h2>
<p>This is the category of pupils who might have some interest in computing but whom have never really attempted any programming before. It's quite likely that a good portion of these might find the subject really enjoyable and will thus become the future genration of computer skilled graduates that the Government is hoping for. Unfortunately I think this category is MUCH smaller that those advocating the new sylabus believe that it is. In any given class I would think that it will be just one or two pupils - hardly a revolution.</p>
<h2>The Hooked</h2>
<p>Finally, there will be a group of pupils who will really love computing and programming. They will be the people likely to become the hard-core programmers of the future. Unfortunately, it's probably likely that these pupils will already have computing as a hobby at home. They will likely already be building websites and writing software. Even if they aren't at the start, then they soon will be once they get hooked. However, a sylabus created to cater for the slower pace of the uninterested will likely soon become very boring for those able and interested pupils. My fear would therefore be that they become unmotivated, unchallenged and bored by doing work that is way below their ability. This could then have a negative effect and turn those potential developers away from computing forever.</p>
<p>In fact, I know this will likely be the case as it happened to me. I was at school 28 years ago when computing first started being introduced. I got hooked early and my (very supportive parents) ensured that I was one of the first people in school to have a home computer. I taught myself to program at age 10 and never looked back. Unfortunately I found I was always way ahead of the curve when it came to computer studies lesson, talking to teachers about computers and so on. I got bored through not being stimulated. I had a few years of just playing games and word processing essays. I even toyed with a career in accountancy! Luckily I rediscovered my interest, changed tact, and went on to study Computer Science at University. It could have turned out so different.</p>
<br/>
<p>My real fear of the proposed strategy is that rather than producing a new generation of highly skilled computer graduates it will instead produce a large number of moderately skilled people while turing the highly skilled pupils with the best potential away from the subject. Hey, let's build a large army of mediocracy!</p>
<p>So, what do I think needs to be done. I agree with the general goal of the new strategy, but I think it needs to be covered by a dual-sylabus approach. A sylabus for the uninterested and swayable that teaches general computer skills, applications, effective use of the Internet, basic programming and so on. To be taught by ordinary teachers with standard levels of IT skills. There also needs to be an Advanced Computing sylabus for the hooked and highly able that teaches programming and computing to a much greater depth. It would need to be taught by highly skilled teachers with a heavy input from universities and industry. Its aim would be to stretch the most able who could become the highly skilled people that our industry really needs.</p>Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com1tag:blogger.com,1999:blog-5400765672405463592.post-65610249277025508402011-12-01T07:49:00.001+00:002011-12-01T08:42:39.682+00:00Increasing Agile Capacity<p>A scenario: you have a successful agile project. It’s made up of a number of small teams, each working on a separate component of the application suite. This has been working extremely well. You are capacity planning for the coming six months and you identify that one of these teams is likely to become a bottleneck. There are just too many stories that fit into their component area. How do you deal with this?</p>
<p>The above is something that looks likely on one of the projects that I am working on. I’ve been putting in some thinking on how this might pan out. This post documents my thoughts.</p>
<h3>Traditional Reaction</h3>
<p>There are typically two standard reactions to above problem: grow the team that is likely to be the bottleneck, or add an additional team to work on the component. In my experience neither of these works. Adding more people is rarely ever a good way to make a project go faster!</p>
<h2>Growing the team</h2>
<p>The best agile teams I have worked on have had one thing in common. They have all had between 4 and 6 devs, a QA, and a Scrum Master. As soon as the team gets any bigger than this I’ve found that the following occurs:</p>
<ul>
<li>The performance gain of adding each additional dev is less each time.</li>
<li>It becomes much more difficult to gain consensus.</li>
<li>Team meetings become more long-winded and less productive.</li>
<li>Communication becomes more difficult.</li>
<li>It becomes difficult to maintain team values.</li>
<li>Code cruft (technical debt) increases.</li>
</ul>
<p>Thus, adding additional devs and QAs to a team, either permanently or by borrowing from another team, just doesn’t work. Typically I’ve even found that the team goes slower as it has to spend more time cleaning up after itself.</p>
<p>Adding temporary devs (perhaps for a day or two) is, I’ve found, often worse than adding a permanent dev as there is less sense of ownership and shared team values. Working with good developers makes this less so, but I still believe it occurs more frequently than we like to admit.</p>
<h2>Adding more teams</h2>
<p>Having multiple teams working on a single component’s code base just doesn’t work well. I worked on one project where we had four teams working on the same code. We were tripping over each other all the time. Particular problems I have seen working in this way:</p>
<ul>
<li>Communications between the teams becomes a big overhead.</li>
<li>More difficult to gain consensus on major technical decisions.</li>
<li>Scheduling work to avoid multiple pairs changing the same code is very difficult.</li>
<li>Very difficult to maintain shared values across multiple teams.</li>
<li>One team’s nice code is another team’s cruft.</li>
<li>Code cruft increases because one team is less willing to refactor another team’s code.</li>
<li>Different approaches to achieving the same implementation become more prevalent.</li>
<li>One team undoes another team’s work because they disagree with how it was done.</li>
<li>Important aspects like performance tend to slip between the cracks.</li>
<li>Productivity of the combined teams is far less than the sum of the productivity of each team working independently.</li>
</ul>
<p>Multiple teams working on the same code base is a recipe for disaster. It results in more overhead, lower code quality and more cruft. Also, you get far less productivity gain that you would had you allowed each team to work independently on their own component.</p>
<h3>Other Approaches</h3>
<p>So, given that there is more work than the measured velocity of the team shows they can achieve in the time available. And, given that we don’t want to grow the team or add more teams. How do we solve the problem?</p>
<p>The aim is to keep small teams who can each work on an area of code that they have sole ownership of. This will give them the incentive to ensure that this component is clean, simple, maintainable and has a low level of cruft. It will also maximise the productivity of each individual team by minimising communication overhead.</p>
<p>I think a combination of four specific solutions are the best approach. These being: reducing waste, simplifying requirements, moving complexity and splitting components.</p>
<h2>Reduce Waste</h2>
<p>A good agile team should already be pretty good at reducing their waste. Retrospectives that generate actions which are then implemented go a long way towards this. A good Scrum Master who removes impediments is vital. The team will be provided with the best tools for the job.</p>
<p>Still, there is likely to be further waste that can be eliminated to allow the team to go faster. The first set comes from the category of unnecessary process waste. This includes things like:</p>
<ul>
<li>Having to attend full meetings where there is very little need for members from the team to be present.</li>
<li>Long-winded design sessions that could be cut down with a bit more decisiveness.</li>
<li>Unnecessarily revisiting architecture and design decisions over and over again.</li>
<li>Re-enacting the same debate repeatedly because consensus wasn’t achieved.</li>
</ul>
<p>The other set comes from giving the bottlenecked team stories to play that aren’t essential to what they are trying to deliver. This can be things like:</p>
<ul>
<li>Helping ops with tracking down a problem</li>
<li>Adding additional logging of messages to an external system</li>
<li>Trying to reproduce an infrequent defect</li>
<li>Updating a build pipeline</li>
</ul>
<p>In an ideal world, all of the above should be done by the team. However, if that team has a high volume of work to get through then these tasks can easily be divested out to teams that are perhaps not running at full capacity, allowing the team that is to maximise their throughput of important functional stories.</p>
<h2>Simplify Requirements</h2>
<p>Often performance of teams can be greatly hampered by having requirements that are either too complex or not clearly thought out. Ensure the backlog is groomed to contain only the most essential stories. Make sure these have been paired down to the absolute minimum set of requirements needed to fulfill the business needs.</p>
<p>The worst scenario is having a heavily loaded team working on features that are overly complex and time consuming. I’ve seen many cases where teams have worked on a fairly complex story only to later find that the clients uses just the core 80% of the feature, which could have been built in 20% of the time. The extra 80% of the time spent delivering that last 20% of non-essential requirement is just waste.</p>
<p>Analysts and the business need to scrutinise the backlog, squeezing every unnecessary piece of work from the stories. If the business doesn’t need some small aspect right now then create a separate story and play it later when the team has less critical path work.</p>
<h2>Move complexity</h2>
<p>In most large software systems it is possible to locate complexity in different components. For example table paging could be implemented in the server code or in javascript on the browser. If complexity in one place is causing capacity problems then it may be possible to move elements of this complexity to another team that has more spare capacity.</p>
<p>This may not always be the most technically pure solution, but it may be the best trade-off in order to deliver a product on time and budget. The complexity can always be relocated in the future.</p>
<p>As an example, consider a story where one component has 8 points of work do to delivery a feature and it’s two clients each have 2 points of work to consume the feature. Total 12 points. However, it might be possible to split the work differently, so that the core component implements something much simpler for 3 points, while the two clients take on some complexity and implement 5 points each. Total 13 points. So, the two approaches are roughly equivalent in size. The second might result in some duplication on behalf of two teams, but this allows the team with a large backlog to complete more stories. A pragmatic trade-off.</p>
<h2>Split Components</h2>
<p>If a team has multiple components, then the logical option is to relocate one or more of those components to another team. What if they only have one component?</p>
<p>Ideally, buy reducing waste, simplifying requirements, and moving complexity it should be possible to shrink the size of the backlog for a component. Hopefully the smaller backlog should be sufficiently sized that the team can complete it. What if they still can’t?</p>
<p>To me this seems to be a problem of decomposition. A component that is so large and complex that a team of six devs can’t maintain and enhance it is a component that is too big. In this case it should be split into two or more smaller components, with the current team keeping one part and a different team taking the other part(s).</p>
<p>Making the split is always technically challenging, but provided it is made across a clear interface boundary then my experience is that it is usually possible divide a component up into smaller constituent parts.</p>
<p>This is probably the only way that I would recommend going about adding more people to a project - introduce a new team (or repurpose an existing team) and give them their own code base and component(s) to work on.</p>Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com0tag:blogger.com,1999:blog-5400765672405463592.post-90156481412702394412011-11-25T10:41:00.000+00:002011-11-28T20:29:36.723+00:00Is Scala Like EJB2?<p>In <a href="blog.joda.org/2011/11scala-feels-like-ejb-2-and-other.html">this post</a> and then in <a href="blog.joda.org/2011/11scala-ejb-2-feedback.html">this followup</a>, Stephen Colebourne (@jodastephen) states his criticisms of the Scala language and how it feels to him similar to the same feeling of when he was working with EJB2. A colleague of mine also tweeted the following:</p>
<p><b>"This is why I will never use Scala: def ++ [B >: A, That] (that: TraversableOnce[B])(implicit bf: CanBuildFrom[List[A], B, That]) : That"</b></p>
<p>Here's my thoughts and experience with the above.</p>
<p>I started out as a C and C++ developer, but joined the Java set right at the very beginning. I've been fortunate enough to have been working in Java since 1998. Or unfortunate, if you include the early days of the Servlet API & JSP, EJB1 and then EJB2! About 2007 I really started to feel the need for a different language, one that would allow me to express my ideas more concisely and without writing so much boilerplate code. I looked at Groovy, but found it too easy to keep dropping back and writing Java. I then learned Ruby. This taught me how expressive a good language can be and how powerful closures can be. I dabbled with a few more languages before finally finding Scala at the beginning of 2009. These days I mostly code a mix of Scala, Java and Groovy. Next challenge is Clojure.</p>
<h2>Changing Mindset</h2>
<p>Now, I'm an engineer at heart and my typical approach is to not only understand how to use something but also (and usually at the same time) understand how and why it works. This approach always served me well when learning languages like Java and Ruby. These languages have a fairly small surface area and a relatively simple type system. It's easy to read up on them, understand how to use them and understand exactly how they work. It's easy to look at the standard and third-party libraries and see how these work as well. For me, Scala was different and involved me making a fundamental shift in mindset about how I learned the language.</p>
<p>When I started out I approached Scala in the same way I did other languages: I read a book and tried to build some test applications while at the same time delving into the library and language to see how they worked. This was daunting! I can fully appreciate Stephen's position that it feels a bit like EJB2. Let's put that in context. EJB2 was a horrid mess of unnecessary complexity and complication. It aimed to simplify building enterprise apps involving persistent entities and business services, but it just added way more complexity and mind effort than it saved.</p>
<p>Now, back to Scala. It's a big language with a very extensive and powerful type system. If you attack trying to understand it as a whole I can see how it could be mistaken as another EJB2: there seems to be loads of complexity, a lot of which you can't see the reason for and a lot of stuff just requires so much mind effort to understand. Much of it initially seems unecessary.</p>
<p>It was after my initial failure to understand how the Scala language and libraries actually were built and worked that I took a step back and reconsidered my learning style. I could already see some value to the language and I wanted to explore more, but how? I then made a fundamental change in my approach. I decided that it was okay for me to start using the language without necessarily understanding exactly how all the details worked. As I used the language more I would gradually try to understand the how and why, one small step at a time.</p>
<p>I started learning to write Scala code. When I wanted to achieve something I researched how to do it until I was familiar with the use of a particular language feature or method. If there was some syntax, implementation detail or concept that I didn't get at that point I just noted it down as something to come back to. Now after a year or so writing Scala I look back at what I didn't initially understand and it mostly makes sense. I still occasionally encounter things that don't make total sense initially, so I just learn enough to use them and then come back later to learn the why or how. It's a system that seems to work incredibly well.</p>
<p>When you attack looking at and learning Scala in this more gradual way, first understanding how to use it then gradually, and incrementally, delving more deeply into the type system, you begin to realised that it's not so bad. Yes, there is complexity there, but it's not the unnecessary complexity of EJB2, it's moderate complexity that helps you be expressive, write safe code and get things done. As my knowledge of Scala has grown I've yet to find any features that I haven't been able to see the value of. It's a very well thought out language, even if it doesn't quite seems so at first.</p>
<p>Switching from a mindset where I have to have a full understanding or how something works before I use it to one where I'm comfortable with how to use it even if I don't know why it works was a very disconcerting change in my thinking. I'm not going to say it was an easy change to make, because it wasn't. However the effort was worth it for me because it got me using a language that I now find to be many more times more productive than Java. I think I also understand the language more deeply because I have put it to use and then come back later to understand more deeply exactly how and why it worked. I would never have got to this depth of understanding with my approach to learning other languages.</p>
<h2>An Example</h2>
<p>Consider the method:</p>
<pre class="brush:scala">
def ++ [B >: A, That] (that: TraversableOnce[B])
(implicit bf: CanBuildFrom[List[A], B, That]) : That
</pre>
<p>There's a lot of implementation detail in there. However, in order to use this method all I really need to understand is that it allows me to concatenate two immutable collections, returning a third immutable collection containing the elements of the other two. That's why the Scala api docs also include a use case signature for this method:</p>
<pre class="brush:scala">
def ++ [B](that: GenTraversableOnce[B]): List[B]
</pre>
<p>From this I understand that on a List I can concatenate anything that is GenTraversibleOnce (Scala's equivalent of a Java Iterable) and that I will get a new list returned. Initially I need understand nothing more in order to use this method. I can go away and write some pretty decent Scala code.</p>
<p>Some time later I learn that the >: symbol means 'same or super type of' and understand this concept. I can then refine my understanding of the method so that I now know that the collection that I want to concatenate must have elements that are the same (or a supertype) as the collection I am concatenating onto.</p>
<p>As I later progress and learn type classes I can see that CanBuildFrom is a type class that has a default implementation for concatenating lists. I can then take this further and create my own CanBuildFrom implementations to vary the behaviour and allow different types to be returned.</p>
<p>I'm first learning how to use the code, then gradually refining my understanding as my knowledge of the language grows. Trying to start out by learning exactly what the first signature meant would have been futile and frustrating.</p>
<h2>Conclusion</h2>
<p>If you start using Scala and you take the approach of trying to see it all, understand it all and know how and why it all works then it's a scary starting place. The size and complexity of the language (particularly its type system) can seem daunting. You could easily be confused in feeling it similar to EJB2, where reams of needless complexity just made it plain unusable.</p>
<p>However, by switching mindset, learning to use the language and then growing understanding of how it works over time it becomes much more manageable. As you do this you gradually realise that the perceived initial complexity is not as bad as it first seemed and that it all has great value. That initial hurdle from wanting to understand it all to being happy to understand just enough is very hard to get over, but if you do then a very powerful, concise and productive language awaits you.</p>Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com4tag:blogger.com,1999:blog-5400765672405463592.post-55773981077905042512011-11-21T11:23:00.001+00:002011-11-21T11:28:59.726+00:00The Luhny Bin Challenge<p>Last week Bob Lee (<a href="https://twitter.com/#!/crazybob">@crazybob</a>), from <a href="https://squareup.com">Square</a>, set an interesting coding challenge. All the details can be found here: <a href="http://corner.squareup.com/2011/11/luhny-bin.html">http://corner.squareup.com/2011/11/luhny-bin.html</a>. I decided this sounded like a bit of fun a set to work on a Scala implementation. I'm pleased with my final result, but also learnt something interesting along the way. My full solution is available on my github account: <a href="https://github.com/skipoleschris/luhnybin">https://github.com/skipoleschris/luhnybin</a>.</p>
<p>The challenge was an interesting one, and more difficult that initial reading would suggest. In particular, the ability to deal with overlapping card number and the need to mask them both added significantly to the difficulty of the problem.</p>
<h2>My Algorithm</h2>
<p>The approach I selected was fairly simple. Work through the input string until a digit character was encountered. Walk forward from there consuming digit, space of hyphen characters until either the end of the string was reached, an different character was encountered or 16 digits had been collected. Next try to mach the LUHN, first on 16 digits, then 15 and finally 14. Record an object describing the index and how many characters to mask. Repeat for the next digit character and so on. After visiting all characters in the string, apply all the masks to the input.</p>
<p>There were some optimisations along the way. For example, if encountering 14 or less characters then there was no chance of an overlapping card number so you could safely skip over these remaining digits after the first one had been evaluated.</p>
<h2>My Implementation</h2>
<p>I decided to implement in Scala as this is my language of choice at the moment. I considered both a traditional imperative approach as well as a more functional programming solution. In the end I went for a very functional implementation, based around immutable data. My main reason for this choice was because I'm working on improving my functional programming skills and I though it looked like an interesting problem to try and solve in this way.</p>
<p>The implementation consist of for main pieces of code. Each implements a transformation of input into output, is fairly self contained. I fell that by writing them in this way I have created code that is possible to reason about and also reuse in other scenarios. I tried for elegant code rather then optimising for the most efficient and minimal code base. I think I achieved my goals.</p>
<h2>Conclusion</h2>
<p>While I am very happy with my solution and had great fun building it, it was very interesting to look at other solutions that had been attempted. In particular it's quite clear that the ~900ms that I was able to active on my mid-2010 MBP was a number of times slower than the fastest solutions. The inevitable conclusion being that in some cases adopting a more mutable and imperative approach may be preferable. This is a typical example of this case, when we are in effect creating a logging filter, and speed of operation could easily be argued as more important than adhering to the principles of immutability. An interesting conclusion.</p>Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com0tag:blogger.com,1999:blog-5400765672405463592.post-57290780660395303102011-11-01T17:41:00.000+00:002011-11-01T17:41:14.940+00:00Concise, Elegant and Readable Unit Tests<p>Just recently I was pair programming with another developer and we were writing some unit tests for the code we were about to write (yes, we were doing real TDD!). During this exercise we came to a point where I didn't feel that the changes being made to the code met my measure of what makes a good test. As in all the best teams we had a quick discussion and decided to leave the code as it was (I save my real objections for times when it is more valuable). However, this got me thinking about what actually I was uncomfortable with, and hence the content of this post.</p>
<p>As a rule I like my code to have three major facets. I like it to be concise: there should be no more code than necessary and that code should be clear and understandable with minimum contextual information required to read it. It should be elegant: the code should solve the problem at hand in the cleanest and most idiomatic way. Also, it must be readable: clearly laid out, using appropriate names, following the style and conventions of the language it is written in and avoiding excessive complexity.</p>
<p>Now, the test in question was written in Groovy, but it could just as well be in any language as it was testing a fairly simple class that maps error information into a map that will be converted to Json. The test is:</p>
<pre class="brush:java">
@Test
void shouldGenerateAMapOfErrorInformation() {
String item = "TestField"
String message = "There was an error with the test field"
Exception cause = new IllegalStateException()
ErrorInfo info = new ErrorInfo()
info.addError(item, message, cause)
Map content = info.toJsonMap()
assert content.errors.size() == 1
assert content.errors[0].item == item
assert content.errors[0].message == message
assert content.errors[0].cause == cause.toString()
}
</pre>
<p>Now, to my mind this test is at least three lines too long - the three lines declaring variables that are then used to seed the test and in the asserts. These break my measure of conciseness by being unnecessary and adding to the amount of context needed in order to follow the test case. Also, the assertions of the map contents are less than elegant (and also not very concise), which further reduces the readability of the code. I'd much rather see this test case look more like this:</p>
<pre class="brush:java">
@Test
void shouldGenerateAMapOfErrorInformation() {
ErrorInfo = new ErrorInfo()
info.add("TestField", "There was an error with the test field",
new IllegalStateException())
Map content = info.toJsonMap()
assert content.errors.size() == 1
assert hasMatchingElements(contents.errors[0],
[ item : "TestField",
message : "There was an error with the test field",
cause : "class java.lang.IllegalStateException" ])
}
</pre>
<p>This test implementation is more concise. There are less lines of code and less context to keep track of. To my eye I find this much more elegant and far more readable. </p>
<p>"But, you've added duplication of the values used in the test", I hear you cry! That is true and it was totally intentional. Let me explain why…</p>
<p>Firstly, as we have already mentioned, duplicating some test values actually makes the code more concise and readable. You can look at either the test setup/execution or the assertions in isolation without needing any other context. Generally, I would agree that avoiding duplication in code is a good idea, but in cases such at this, the clarity and readability far out way the duplication of a couple of values!</p>
<p>Secondly, I like my tests to be very explicit about what they are expecting the result to be. For example, in the original test, we asserted that the cause string returned was the same as the result of calling toString() on the cause exception object. However, what would happen if the underlying implementation of that toString() method changed. Suddenly my code would be returning different Json and my tests would know nothing about it. This might event break my clients - not a good situation.</p>
<p>Thirdly, I like the intention of my code to be very clear, especially so in my tests. For example, I might create some specific, but strange, test data value to exercise a certain edge case. If I just assigned it to a variable, then it would be very easy for someone to come along, think it was an error and correct it. However, if the value was explicitly used to both configure/run the code and in the assertion then I would hope another developer might think more carefully as to my intent before making a correction in multiple places.</p>
<p>My final, and possibly most important reason for not using variables to both run the test and assert against is that this can mask errors. Consider the following test case:</p>
<pre class="brush:java">
@Test
void shouldCalculateBasketTotals() {
Basket basket = new Basket()
LineItem item1 = new LineItem("Some item", Price(23.88))
LineItem item2 = new LineItem("Another item", Price(46.78))
basket.add(item1)
basket.add(item2)
assert basket.vatExclusivePrice ==
item1.vatExclusivePrice + item2.vatExclusivePrice
assert basket.vat == item1.vat + item2.vat
assert basket.totalPrice == item1.price + item2.price
}
</pre>
<p>In this test, what would happen if the LineItem class had an unnoticed rounding error in its VAT calculation that was then replicated into the basket code? As we are asserting the basket values based on the fixture test data we may never notice as both sides of the assertion would report the same incorrectly rounded value. By being more explicit in our test code we not only create code that is more concise, elegant and readable but we also find more errors:</p>
<pre class="brush:java">
@Test
void shouldCalculateBasketTotals() {
Basket basket = new Basket()
LineItem item1 = new LineItem("Some item", Price(23.88))
LineItem item2 = new LineItem("Another item", Price(46.78))
basket.add(item1)
basket.add(item2)
assert basket.vatExclusivePrice == 58.88
assert basket.vat == 11.78
assert basket.totalPrice == 70.66
}
</pre>
<p>Finally, a couple more points that people might raise and my answers to them (just for completeness!):</p>
<p>Q: Can't we move the constant variables up to the class level and initialise them in a setup to keep the tests methods concise?<br/>
A: This makes an individual method contain less lines of code but fails my consiseness test as there is context outside of the test method that is required to understand what it does. You actually have to go and look elsewhere to find out what values your test is executing against. This is also less readable!</p>
<p>Q: Isn't is better to share test fixtures across test methods in order to keep the test class as a whole more concise?<br/>
A: This might be appropriate for some facets, such as any mocks that are used in the test or if there is a particularly large data set required for the test which is not part of the direct subject of the test. However, I'd rather see slightly longer individual test methods that are explicit in their test data and assertions, even if this means some duplication or a slightly longer overall test class.</p>Chris Turner (he/him)http://www.blogger.com/profile/09141108967081921802noreply@blogger.com3