Saturday, 12 March 2022

The Value of a Safe and Open Interview Process

I’ve recently been interviewing for a new role. During this time I went through the interview process with three different companies. Two of these I already had in progress before finding the third role (which was the one I really wanted). Each offered a very different approach to the way they structured and conducted their interviews.

At the end of these processes I had very different emotional feelings about each of the three companies and roles. This got me thinking about what factors resulted in these different emotional responses.

Company A - Traditional Interview Process

This company had a fairly traditional interview process divided into four stages:

  • An initial short chat with a member of the Talent Team to make sure that the role was a good fit. There’s no point in pursuing an expensive and time consuming series of interviews if the role is not a good fit for the candidate or visa-versa.
  • A technical interview with two Engineering Team members talking through experience, knowledge and approach, with a few technical and design questions along the way. This also acted as a chance to discuss and ask about engineering culture.
  • A problem interview, which for this company was conducting a code review on an example pull request. This was carried out in a low-pressure way where I was given time to do the review without the interviewers being present, followed by a short discussion of the points I had identified at the end. (Kudos to the person who wrote the code to review, it was a masterpiece of how to cram as much bad coding and design practice as possible onto a single page!).
  • A final chat with the Engineering Manager (which I didn’t complete as I had already accepted one of the other roles by this time).

Overall I came away from this interview process feeling fairly neutral about both the company and the role. There was nothing that made me feel uncomfortable or unsettled, but at the same time I didn’t come away from any of the interviews feeling super excited or energised by the process.

They created an environment where I was able to feel fairly open to answering their questions, but still feeling that there were certain questions that they wanted to hear me give model answers to and others where the answers had to be correct.

I expect that this is the default experience for many people as they go through the process of finding their next role, but is this really the best we can do?

Company B - Toxic Interview Process

The second company I interviewed with had an interview process that I can only describe as toxic. I made it to almost the final stage of these interviews, but after my experience, there was no way I would have wanted to go and work for them anyway (even though they were the most high profile and highest paying of the three roles).

First off, before you could even enter their interview process you had to complete an online, timed, coding exercise. This involved two tasks where you are presented with a short problem statement, an empty code template and a set of unit tests (most of which you can’t see the inputs or expected outputs for). You then have to code up a solution that solves the problem and makes all the unit tests pass. All this takes place in an environment with a big countdown clock in the corner that ticks away your remaining time.

In my experience, there are very few people who can just sit down and pick up coding kata problems like these and produce great results. In order to be successful you have to have practised many times on similar tasks before actually taking the test. These practices also get you familiar with how the site doing the testing works, how it describes its problems, what sort of things it tests for and so forth. The net result is that to be able to pass the 90 minute test you also have to additionally invest a number of hours of practice time as well, and this is before you even know if the company or role is a good fit or not.

I also have some very specific complaints about these sort of logic/coding puzzle timed tests:

  • All they prove is that you are good at logic/coding puzzles - they bear little resemblance to most of the engineering you will actually be doing in your job.
  • A lot of them require you to just ‘see’ the solution, and, if for that particular problem, you don’t have that immediate insight, then it’s almost impossible to complete a solution, as you just don’t have enough time to work through it and create working quality code.
  • Some people who are great engineers just don’t thrive in situations where there’s a countdown timer or someone watching their every keystroke.
  • These tests may also exclude people who feel less confident or a bit anxious from even being able to access the interview process at all.
  • They can create stress and discomfort, especially if the candidate is struggling with the particular problem they have been presented with and the clock is draining away.
  • Unit tests where you cannot see the input or the expected output, only that they are failing provide minimal value or oppertunity to improve your solution and just add additional stress and pressure.

So, I did manage to pass this initial screening test. Even though my first impressions weren’t great, I decided to continue with the process just to find out more about the role, team and company - you never know, it may just have been some poorly considered HR requirement.

The next steps were multiple technical interviews with various Engineering Team members that all followed a similar pattern:

  • A brief discussion about the role or some aspects of the project or engineering culture.
  • A bunch of fairly detailed technical questions covering skills and previous experience.
  • Yet more logic/coding exercises carried out live with the interviewer(s) watching and commenting on every line you wrote and expecting you to both code and narrate your thought processes as you did so.

There were a number of aspects to these interviews that turned them into a hostile and stressful environment:

  • When discussing projects or processes the interviewing engineers came across as acting in an incredibly superior way, sometimes sneering if your answer wasn’t perfect or you previously haven’t been working to the specific ‘variation’ of scrum that their teams have adopted.
  • They presented a demeanour of making you feel small or unworthy if you didn’t immediately know the Big O notation for a specific sort algorithm or couldn’t immediately code out a fully working, super efficient implementation of their chosen logic/coding problem.
  • I got the feeling that often they were trying to catch me out and just waiting for the moment to pounce: especially if I gave a slightly wrong or incomplete answer to one of their questions.

The result of this sort of interview approach is that I started to feel stressed and pressured. Every question became a panic decision between do I give the answer I thought was right vs the risk that the answer might be wrong vs admitting that was something that I didn’t fully know. Rather than opening up to them I was being forced to close down. Ultimately I just didn’t feel safe or supported in that interview environment, started to doubt my own abilities and began to become flustered and unable to answer or write code any more.

At the end of each interview, the overwhelming emotion that I experienced was one of relief that the ordeal was over. No excitement about the role or the company was generated: just relief. After the final interview with them I was left visibly shaking from the release of stress tension built up over the duration of the interview.

Edit: Having thought about this more, I'm sure that this company didn't set out to create an interview process that created these emotions. I believe they probably have the goal of selecting only the very best technical candidates. Their selected solution for doing this was to focus heavily on processes that test primarily for technical skills. The unintended side-effect of this being that they have less time to focus on cultural fit and empathy for the candidates. /end

If that’s their process for finding the ‘right’ people, what would they be like to work for? A culture that encourages that sort of pressure on interview candidates surely isn’t an enjoyable, safe and supportive place to work? Would the type of people who were actually able to make it through the recruitment process be the kind of people I would want as my future colleagues?

In the end they decided not to continue with the final stages of the interview process with me. I wouldn’t have gone for another interview with them anyway. I’ve never been happier to drop out of the running for a new job!

Company C - Safe and Open Interview Process

The final company I interviewed with followed a process that was a world apart from my other two experiences. This process was probably the most detailed and time consuming of the three, but every aspect of it was thoroughly enjoyable, engaging and enlightening.

First off was a chat with the Engineering Manager who I would be working with. This was about as far away from a technical interview as you can get. We just chatted. He spent a lot of time telling me about the company, the product and the culture. Then I was asked a simple question (I can’t remember the exact words, but it was something along the lines of): “What’s your first impression, what excites you about this role?”. What a great way to assess a candidate. If they aren’t excited by what they’ve heard then why would you want them in your team? As a candidate, if the role and culture aren’t exciting then would you really enjoy working there? Would you grow? After that we just chatted generally about experiences, background and so on.

Particularly interesting was that the chat was conducted in a way that felt safe. It was easy to open up, admit answers I didn’t know; ask questions that might seem strange; talk about times when things I did went wrong. The overall feeling was that the interview was about getting to know each other rather than trying to judge. I finished this step excited and already knowing that I really wanted this job.

The next stage of their process was a take-home exercise. I know some people aren’t a fan of these, but I’ve always found them to be quite an enjoyable way to demonstrate my abilities. If they are structured right then they allow you to show problem solving, technical approach and code that is realistic to what you would typically produce (as opposed to trivial/funky/clever code that solves a specific logic problem).

In this particular case the exercise was closely related to their product, so it almost felt like you were already working on something related to the job. The exercise was timeboxed to three hours (which is often a criticism of take home exercises that are open ended and thus candidates feel they have to dedicate a huge chunk of their life to completing them). I spent a day mulling over the problem in my head then sat down over a couple of sessions and produced my submission. I was really enjoying the problem so I even carried on for a few more hours after submission to improve my solution and solve a couple of little areas that I hadn’t quite got right.

My submission was reviewed and I was invited back to the next stage of the interview, which was to talk about what I had done. This was carried out with two members of the Engineering Team that I would be working with if successful. Again this was conducted in a way that was structured to be entirely safe and open. They seemed really interested in what I had done and how I had done it - not some kind of faked interest, but genuinely interested. At no point was there ever any judgement on the approach taken, it was always a discussion on the reasons and the thinking behind a particular approach or technique.

Feeling comfortable and secure, I found it easy to be open and honest about where I had made good or bad decisions and how I would do things differently as I iterated over the solution. This enabled much more discussion rather than it being a question/answer driven session. I was not only able to get a sense of how they worked as a team and how strong their engineering practices were, but also who they were as people - something that’s generally really hard to get during a typical interview.

Next up was another session, this time on talking through how I would go about solving a particular engineering problem. Again this was a problem relevant to their domain and product so it felt like I was actually working with the team rather than being assessed. This was also another great example of creating a safe interview space where it was easy to be open and explore different avenues. No suggestion was wrong or dismissed, I was free to explore avenues that didn’t go anywhere or solutions that might be a bit unusual. It was easy to ask questions about what might or might not be possible. The discussions were all about the directions taken to solve the problem, why they might work or not and how my thinking process worked to get there. Again this was very much a two-way process of how I would fit in with the team but also how the team would work with me.

I left both of the above sessions feeling even more excited about the role, excited about the people I would be working with and with a sense of comfort that I would be able to bring skills and ideas that would enrich the team and that I would get the same in return. That’s a really positive feeling to leave a technical interview with!

The final interview stage was back with the Engineering Manager again, this time with the Product Owner involved as well. Again this worked as much more of a discussion process rather than an interview. The PO was able to enthusiastically talk about the product and the vision for the future. There were ample opportunities to ask detailed questions and discuss thoughts. Finally there were questions around how I think about products, user experiences, technology and so forth. Again this was all conducted in a safe space where I felt at ease opening up with my thoughts and where I felt comfortable giving detailed answers and my reasoning behind them.

Overall this interview process was amazing. At every point in the process I felt able to give open and truly honest answers. After each interview I felt energised and excited. Suffice to say, when they offered me the position I jumped at the opportunity and I can’t wait until I join the team. In fact, I already feel part of the team just from the interview process and that’s a great feeling.

Conclusions

Thinking over my interview experiences there’s a few good takeaways that should be part of every organisations’ interview process:

  • If an organisation can’t excite a candidate or the candidate can’t excite the team then the two may not be a good fit.
  • Creating a safe interview space will encourage candidates to be more open and honest with their answers, offer up more details and be more likely to ask deeper questions. Both parties learn so much more about each other.
  • Conversely, creating an interview environment that is hostile and stressful will discourage candidates sharing honest answers, will limit how much you can learn about them and ultimately will prevent them wanting to join your organisation
  • Some exploration of technical skills and experience is always relevant but ensuring the candidate is a great fit to the team culture personality wise is far more important,
  • During the interview, make the candidate part of the team and work with them as such. You will learn much more about who they are, how they work and whether they are a great fit than if you treat them as an outsider to interrogate.
  • Empathy with the candidate shows that the team they will be joining cares, and for great people that’s often a far bigger selling point than financial rewards or interesting technologies.

Tuesday, 4 March 2014

The Importance of a Good Shard Key

In MongoDB (and many other database solutions) scalability is achieved by dividing your database into a number of smaller portions. Each portion is stored on a separate set of database nodes. The theory is that this approach allows your database to scale horizontally beyond the capacity of a single node and allows load to be spread more evenly across multiple clusters of nodes. In MongoDB this approach is known as sharding (many other databases call a similar concept partitioning).

When you deploy shards in MongoDB you have to select a field (or fields) that are present in each document as the shard key. The value of this field determines which set of database nodes holds a specific document. If you get this key selection wrong then the impacts on your system can be huge.

As an example of how shard key selection is vital, consider this real-world, non-IT example, which clearly highlights how a poorly selected shard key can massively impact performance of a system:

At the weekend I participated in a large Half Marathon event, with about 20,000 other runners. The number of bags to be stored while the race was on was too large for a single baggage tent, so the organisers had wisely decided to introduce a sharded approach by having two tents, each holding half the bags. Additionally, each tent was further sub-divided into separate evenly-sized sharded collections of bags, each managed by its own team of helpers.

Now, as a shard key the organisers had decided to use race number: a seemingly sensible choice given that this would be the single unique piece of information that every runner would have. The bag tents were therefore arranged so that tent 1 was for numbers 1 to 10,000 and tent 2 was for numbers 10,001 to 20,000. The sub divisions inside the tents were further broken down into 1000 number blocks (i.e. 1 to 1000, 1001 to 2000 and so on).

For pre-race bag drop off this sharding approach worked really well (the write scenario). Runners dropping off bags were nicely distributed across both time and the full range of values, so each tent and sub-division could work in parallel for the maximum write performance. This was clearly an excellent shard key selection for the write case.

The problem occurred at the end of the race when runners returned to the tents to collect their bags (the read scenario). Now, the organisers of the race had issued the numbers based on predicted finish time (1 for the fastest predicted finisher, 20,000 for the slowest): the result being that runners finished roughly in number order.

So, what happened then is that there was a initially massive read queue at tent 1 for the 1 to 2000 numbers, while the other shard nodes were almost completely idle. Then the queue moved to the 2001 to 4000 shards, and so on. Towards the end of the race, tent 1 was idle while tent 2 now had the read queues as the later numbered runners all finished.

We have a perfect case of a shard key that seems quite sensible by design but is actually fundamentally broken by the usage scenario of the system, in this case the (near) sequential arrival of numbers for retrieving the bags.

So, how can we solve this? Issuing the race numbers in a different order is one option, but a sequential numbering system based on predicted finish time is quite sensible for many other race organisation requirements. A better option is to change the shard key used by the baggage tents.

Basing the shard key around race number is still a good approach as this is the one piece of information guaranteed to be unique for each runner and the range of possible values is clearly defined and well distributed. Bag drop off (write performance) is never going to be a problem as runners will arrive fairly well distributed across both time and the race number range.

The challenge is coming up with a way of better distributing the bag retrieval (read performance). Fortunately with a sequential numbering system this is pretty easy. Rather than shard by the whole number, just shard by the last digit of the number: tent 1 – numbers ending 0 to 4; tent 2 – numbers ending 5 to 9. Then in each tent further divide into five separate shards, one for each ending digit. If finer granularity is required then within each shard it should be easy enough to sort and index by full race number.

Selecting a good shard key is hard. Considering both the write and read scenarios, and how the shard key will be utilised within these scenarios is critical. It can make a huge difference between a well-functioning system and one that fails to gain the benefits of the sharded approach.

Wednesday, 4 December 2013

Scala's Maturing Community

I've been involved in the Scala community and attending Scala conferences for about four years now. I've just come back from 2013 Scala eXchange, hosted by Skillsmatter in London. It was a great conference, one of the best I’ve been to, and the organisers and speakers should be thoroughly pleased with their efforts. One thing that I did think quite interesting was a significant change in emphasis from all of the previous Scala conferences that I’ve attended.

When I first started going to Scala conferences four years ago, the emphasis was definitely on an introduction to Scala. The talks focused on basic language features, how to use the collections libraries and core functional programming concepts. There were also some side talks about interesting libraries built in Scala, like Akka and Lift.

Last year the focus moved to a higher level, with more time spent on talking about where Scala was going and on more advanced functional programming concepts. There were talks focusing on things like Lenses and Monads and lots of detail about highly functional libraries developed using Scala. Presentaions about Akka and Lift were still present and people were starting to talk about Futures. In all of these talks, however, functional programming was the primary focus.

At Scala eXchange this year the emphasis was almost entirely flipped around. Most talks were about reactive programming using asynchronous approaches. Loads of stuff about Akka, actors, futures, events and the like. Some talks focused on Scala features such as macros and using type classes. However, there was very little direct talk of functional programming: it was just assumed that everyone present was using immutable data and following a functional programming approach. A huge shift in perspective from just a year ago.

I believe that this represents a massive shift in the maturity of the Scala community. We have moved from an immature group learning about how to use this exciting new language, how to work with immutable data and how to blend functional approaches into our code. We have instead become a group who are using this fantasic language and a set of fairly mature products and reactive techniques to build highly scalable systems. A huge leap in just a couple of years.

I remember a similar transition in the Java community when conferences went from talking about basic Java concepts like POJOs, collections and design patterns to talking about building complex solutions using advanced libraries like Spring and JMS. The Scala community seems to have made this leap much more quickly than the Java community did.

The one thing that worries me slightly about this change is that using immutable data and functional programming has almost become an unwritten assumption. Each presentation just seemed to assume that you would be programming in this way by default. While this is great for those of us in the community who have made this transition, I think we need to be careful not to skip this step for those people just transitioning into the Scala world.

The worst case would be developers transitioning from a language like Java straight into an asynchronous reactive programming model without first going through the transformation of thinking in terms of immutable data and functional programming. Bringing the concepts of mutable state and imperative code straight into an async world is a recipe for disastrous software projects. These in turn could tarnish Scala's representation and significantly complicate the life of those of us trying to bring Scala into traditionally conservative organisations and companies.

It's great the the Scala community has come so far and matured so fast. But, let's not forget the core concepts that underly this transformation. We must ensure that we continue to emphasise their importance as we move forward into the brave new world of highly scalable, asynchronous, reactive systems that Scala seems to the targeting so directly and successfully.

Monday, 15 April 2013

Coaching: How Not What

Yesterday evening I was playing a game of 5-a-side football (soccer to my American readers) with some friends. One of the players on my team was a particularly competent player who was also very vocal in offering advice to his team mates on what to do.

For example, when a player from the opposing team was approaching me with the ball he would offer sage advice such as "Don't let them get past you!". This was rather stating the obvious, as I was quite clearly not planning on letting the opponent go past me and take a clear shot on goal.

In this particular situation my challenge was not in knowing what I should do, but in having the skill and experience necessary to carry out the task. No amount of shouted commands were suddenly going to increase my skill to a level beyond that which I currently possess.

This got me thinking about how we coach people in programming and agile practices. How often do we say to someone something like "that needs to be refactored" or "you need to keep your functions small"? Or, we might say to an agile team "don't overcommit yourselves this sprint". All of these are instructions to do something, made with the assumption that the people receiving them have the skill and experience necessary to carry out the actions.

What if, like me on the football field, the recipients of these instructions knows what they should do, but not how to do it. Clearly we are not being successful coaches in these cases. We need to focus much more on the how rather than the what. Advice offered should be enabling, providing the recipient with a way of learning and gaining experience.

For example, "if you pull this bit of code out into a separate function then this loop becomes much simpler" is much more helpful than "that needs to be refactored".

As coaches we need to be mindful of how we communicate in order to improve the people under our guidance. It's very easy to let it slip and just become another player shouting advice to those who haven't yet gained the skill necessary to implement that advice.

Wednesday, 6 February 2013

Fast Track to Haskell Course

I've spent the last two days on a Fast Track to Haskell course. It was a very interesting two days and I learnt a huge amount. The course was taught by Andres Löh from Well-Typed and hosted by the excellent people at Skillsmatter.

I went into the course having read the superb Learn You A Haskell book and having built a couple of experimental Haskell programs. From that starting point there was not a huge amount of content in the course that I was not aware of. However, it was great to have my learning verified and my understanding in many areas significantly improved. There were also a few things that changed my thinking on certain topics. All in all, a well spent two days.

The first day started with a whistle stop tour of the Haskell language and how to build things in it. There were plenty of excellent exercises to help cement learning and understanding. Then we looked in more detail at types and how to reason about software using types. The day concluded with looking at some more advanced type concepts including higher-order functions and type classes. The second day followed on with some more about types and then went on to look at how Haskell deals with I/O (which is very different to most other languages I have encountered). The rest of the day was then spent looking at common patterns that can be found in Haskell code and how these can be generalised into concepts such as Functors and Monads. Plenty more exercises followed.

I will now be going away and working on some private projects in Haskell, with the hope of attending the advanced course some time later this year. In the mean time, this blog post looks at some of the key things that I came away from the course with and how they relate back to the Scala and Java that I tend to do for my day job.

Focus on the Types, they are the API

When using an OO language, such as Java (or even Scala), we tend to think about classes, their interfaces and what behaviour they have, and then use this as a starting point to develop from. In Haskell you tend to think first about the types that are involved and then work forward from there. This is a very interesting approach as it makes it possible to think about a problem in a very general and quite abstract way without getting too bogged down in detail too quickly. The types become an API to your system (rather than the interfaces, methods and domain model in an OO solution).

My experience from the course was that this tends to lead to many more types and type aliases in Haskell than I would typically define in Java or Scala. However, this soon proves to be a huge advantage as it then makes it much easier to reason about what each function does/should do and makes its behaviour very clear via the type signature. I will certainly be looking at following this practice in any Haskell programs I write and also trying to introduce more types into my Scala code.

It's Pattern Matching and Recursion All The Way Down

In a language like Java there is nothing like pattern matching and recursion has to be used sparingly if you want to avoid blowing up the stack and writing poorly performing code. In Scala there is much more scope for both, although writing tail recursive functions is still very important to ensure that you get the best optimisations from the compiler. In Haskell, pattern matching and recursion are the bread and butter of the language. Almost every single function tends to be a pattern match on arguments and the language supports a naturally recursive style across the board. Given the lazy semantics of Haskell and the optimisation in the runtime for recursive algorithms this is the best approach for building programs in Haskell.

One area for improvement that I can see in my Scala code is to make more use of pattern matching, especially partial functions.

Thinking Curried Helps a Lot

In the Scala world we typically only curry functions when there are very obvious places to partially apply them. More often than not we just partially apply a function using the underscore for one of its parameters. This is needed to ensure compatibility with Java and the object-oriented model. However, in Haskell every function is completely curried. There are no multi-argument functions. Everything is a single argument function that returns either a result or another function.

Initially I found that this made sense in the model that I'm used to: partially applying functions. However, once it came to defining new types as actually being functions as well I found that this started to fry my mind slightly. The conclusion that I came to is that it's best to just think of everything from a curried mindset and throughout the course things gradually became clearer. This thinking is essential to understand things like the State monad (which I'm still not 100% confident about to be totally honest).

Parametric Polymorphism is Very Useful

One of my biggest take-aways from the course was how useful and important parametric polymorphism is. As an (overly) simple example, what does a function from Int -> Int do? It could return the same number, add something to it, multiply it, factor it, modulus it, ignore the input and return a constant - the possibilities are nearly endless. However, what does a function from a -> a do? Given no other constraints it can only do one thing - return itself (this is called the identity function - id in Haskell).

The ability to define functions in this polymorphic way makes it much easier to reason about what a function might do and also to restrict what a function actually can do. This second point greatly reduces the possibility of introducing unexpected defects or creating functions that have multiple interweaved concerns. As another example, consider the function: Ord a => [a] -> [a]. What does this do? Well, it could just return the same input list, but because it takes the Ord type class as a constraint it's quite likely to be some form of sorting or filtering by order function. As the writer of this function I'm restricted to working with lists and just using equality and ordering functionality so the possibility of doing anything unexpected is greatly reduced.

Parametric Polymorphism is possible in Scala, but not quite as powerful as there aren't type classes for may common concepts (e.g. equality, ordering, numbers, functor, applicative, monoid, monad etc.). Introducing Scalaz fixes this for quite a lot of cases, but that's not to the taste of many projects and mixed Scala/Java teams may find this a step too far. However, there's certainly more scope for using parametric polymorphism in everyday Scala code and this is something I'm going to work on ove the coming weeks (expect some more blog posts).

Don't Try to Over Generalise

Something that I've seen in every language is that once developers discover the power of generalising concepts they tend to go overboard with this. Remember the time when every piece of code was a combination of seven billion design patterns? The power of Haskell's type classes and parametric polymorphism makes it very easy to see patterns that may not actually hold. I can see that it would be very easy to waste a lot of time and effort trying to make a data type fit into various different type classes or trying to spot type class patterns in your code and pull them out into generic concepts.

As with any kind of design pattern, generalisation or similar, the best approach (in any language) is to just build what you need to solve a problem and then when it becomes apparent that it either fits an existing pattern or represents a general pattern then this change can be made as part of a refactoring process.

Try to Keep Pure and I/O Code Separate

Haskell's I/O model at first seems very different, unusual and restrictive. Once you get your head around it, it makes so much sense. Separating pure code from code that talks about doing an I/O action from code that actually does the I/O makes for software that is much more easy to reason about, test and reuse. Very powerful and something I'm looking forward to exploring more during my journey into Haskell.

In the Scala world, I'm not sure about the need to go to such lengths as using an I/O Monad or similar. However, I can really see the advantage of separating pure code from code that performs I/O. Having pure functions that transform data and then wiring them together with the I/O performing functions at a higher level seems a natural way to create software that is better structured and easier to test. I'll certainly be playing with some approaches in Scala to experiment with this concept further.

Functors, Applicatives, Monoids and Monads are Nothing Special

A final observation from the course is that many people learning and talking about functional programming get a bit obsessed with concepts like Functors, Applicatives, Monoids and Monads (especially Monads). What I learnt is that most of these are just simple patterns that can be represented by one or two basic functions and a few simple rules. Nothing more to them really. Certainly nothing to get all worked up about!

Yes, there's a great deal of skill involved in knowing when to use these patterns, when to make your own code follow a pattern and so on, but that's true of any programming language or design pattern. It's called experience.

Conclusion

If you are a Scala developer who is enjoying the functional programming aspects of the language I would certainly recommend taking a look at Haskell. I found that it clarified lots of things that I thought I understood from Scala but still had some doubts about. Haskell is, in my opinion, a more elegant language than Scala and once you start using it you realise just how many hoops the Scala team had to jump through to maintain compatibility with the JVM, Java code and object-oriented features. Over the two days I learnt a lot about Haskell, a lot about functional programming and a lot about how to get even more value from Scala.