I’ve recently been working as an associate for Equal Experts. I’ve been part of a successful project to build a digital marketing back-end solution. The project started out life as a JVM project with no particular language choice. We started out with Java and some Groovy. However, the nature of the domain was one that leant itself to functional transformations of data so we started pulling in some Scala code. In the end we were fully implementing in Scala. This post describes the learning gained on this project.
The Project
The project team consisted of five experienced, full-time developers and one part-time developer/scrum master. One of the developers (me) was proficient in Scala development (18 months, part-time) while the others were all new to the language.
The initial project configuration (sprint zero) was based on a previous Java/Groovy project undertaken by Equal Experts. The initial tooling, build and library stack was taken directly from this project with some examples moved from Groovy to Scala. The initial project stack included:
- Gradle (groovy based build system) with the Scala plugin bolted in
- IntelliJ IDEA with the Scala plugin
- Jersey for RESTful web service support
- Jackson for Json/Domain Object mapping
- Spring Framework
- MongoDB with the Java driver
Tooling
The biggest hurdle faced by the project was tooling, specifically tool support for the Scala language. While Scala tooling has come on a huge amount over the last year, it is still far from perfect. The project faced two specific challenges: build tool support and IDE support.
Build Tool Support
The Gradle build tool is primarily a build tool for Java and Groovy. It supports Scala via a plugin. Similar plugins exist for other build tools like Maven and Buildr.
What we experienced on this project was that this was not the ideal scenario for building Scala projects. Build times for loading the Scala compiler were rather slow and support for incremental compilation was not always as good as we would have liked. More often than not we just had to undertake a clean and build each time. This really slowed down test-driven development.
Currently (in my opinion) the only really workable build solution for Scala is the Simple Build Tool, also known as SBT (https://github.com/harrah/xsbt/wiki). This is a Scala specific build tool and has great support for fast and incremental Scala compilation plus incremental testing.
With hindsight it would probably have been better to switch over from Gradle to SBT as soon as the decision was made to use Scala for the primary implementation language.
IDE Support
The Scala plugin for IntellJ IDEA is one of the best editors for Scala. In particular its code analysis and highlighting has come on in leaps and bounds over the last few months. However, the support for compilation and running tests within the IDE is still less than ideal. Even on core i7 laptops we were experiencing long build times due to slow compiler startup. Using the Fast Scala Compiler (FSC) made a difference, but we found it somewhat unstable.
What was more annoying was that after failed compilations in IDEA, we often found that incremental compilation would no longer work and that a full rebuild of the project was required. This was VERY slow and really hampered development.
One proven way of doing Scala development within IDEA is to use the SBT plugin and run the compilation and tests within SBT inside an IDEA window. This has proven very successful on other Scala projects I have been involved with and again with hindsight we should have investigated this further.
Some developers on the team also found the lack of refactoring options, in IDEA, when working with Scala somewhat limiting. This is a major challenge for the IDE vendors given the nature of the language and its ability to nest function and method definitions and support implicits. Personally this doesn’t bother me quite as much as my early days were spent writing C and C++ code in vi!
Libraries
Scala has excellent integration with the JVM and can make good use of the extensive range of Java libraries and frameworks. However, what we experienced in a number of cases was that integrating with Java libraries required us to implement more code than we would have liked to bridge between the Java and Scala world. Also, much of this code was less than idiomatic Scala.
What we found as the project progressed was that it became increasingly useful to switch out the Java libraries and replace them with Scala written alternatives. For example, very early we stopped using the MongoDB Java driver and introduced the Scala driver, called Casbah. Then we added the Salat library for mapping Scala case classes to and from MongoDB. This greatly simplified our persistence layer and allowed us to use much less and more idiomatic Scala code.
We never got a chance to swap out some of the other major libraries that the original project structure was built on. However with more hindsight we should perhaps have made the effort to make these swaps as they would have resulted in much less and much cleaner code. Some specific swaps that we could have done:
- Jersey for Unfiltered or Bowler
- Jackson for sjson or lift-json
- Spring for just plain Scala code
Scala projects work well with Java libraries. However when a project is being fully implemented in Scala then it’s much better to swap these Java libraries for their Scala implemented alternatives. This results in much less and more idiomatic Scala code being required.
Team Members
Having highly experienced team members makes a big difference on any project and this one was no exception. The other big advantage on this project was that all of the developers were skilled in multiple languages, even though most had not known Scala before. One thing I have found over many years is that developers who know multiple languages can learn new ones much more easily than those who only know one. I’d certainly recommend recruiting polygot developers for new Scala projects.
It also helped having at least one experienced Scala developer on the team. The knowledge of the Scala eco-system was helpful as was the ability to more quickly solve any challenging Scala type errors. However, the main benefit was knowledge transfer. All of the team members learned the Scala language, libraries and techniques much more quickly by pairing with someone experienced in the language.
Additionally, as the experienced Scala developer I found it very valuable working with good developers who were still learning the language. I found that explaining concepts really clarified my understanding of the language and how to pass this on to others. This will be of great benefit for future projects.
Any new project that is planning on using Scala should always bring on at least one developer who already has experience in the language. It creates a better product and helps the rest of the team become proficient in the language much more quickly.
Object-Oriented/Functional Mix
We found Scala’s ability to mix both object-oriented and functional approaches very valuable. All team members came from an object-oriented background and some also had experience with functional approaches. Initially development started mainly using object-oriented techniques, but leaning towards immutable data wherever possible.
Functional approaches were then introduced gradually where this was really obvious, such as implementing transformations across diverse data sets using map and filter functions. As the team got more familiar with the functional programming techniques more functionality was refactored into functional style code. We found this made the code smaller, easier to test and more reliable.
Perhaps the only real disadvantage of moving to more functional leaning Scala code is that some code became less readable for external developer with less exposure to the functional style. It became necessary to explain some of the approaches taken to external developers: something that would probably have not been necessary with more procedural style code. This is worth bearing in mind for projects were there will be a handover to a team not skilled in Scala and functional programming techniques.
Conclusions
The project was very successful. We delivered on time and more features than the customer originally expected. I don’t think we would have achieved this if the implementation was in Java, so from that point the smaller, more functional code base produced by developing in Scala was a real winner. Using Scala with MongoDB was a major benefit. Tools was the real let-down on the project. Some problems could be alleviated by moving from Java libraries and solutions to their Scala equivalents. However, there is still some way to go before Scala tools (expecially IDEs) match the speed an power of their Java equivalents.
No comments:
Post a Comment