Unifying client and server side form validation with Clojure(Script)

At uSwitch we recently built a Clojure/ClojureScript service to handle the specification of questions in online forms and the validation of the user's answers. This combination of client and server side code has been an effective solution to some of the problem we had. In this post we talk about the sweet spot we found for applying Clojure/ClojureScript in combination and the lessons learned along the way.

The high cost of changes across applications and tiers

uSwitch is the number one energy supplier switching service in the UK. As part of the switching process we ask the customer a set of supplier specific questions (things such as credit check consent, previous addresses etc.)

An example of the type of questions we ask

Updates to questions occur frequently and were a time consuming development task. Why? The main problem was adding questions required changing code in several places: we had a service that specified the questions to ask and we had two web applications (one for our main website and one for our call centre) that displayed and validated the questions. Worse, when a new question was added, code in both the client (Javascript) and server (Ruby on Rails) layers of these applications had to be modified.

The old architecture

A Clojure Service and a ClojureScript generated Javascript library

To solve these problem we created a new Clojure service that provided both a RESTful API and a ClojureScript generated Javascript library. At the core of both of these was a set of shared Clojure data structures (a vector of maps) and a rules engine. This data represented the set of questions that needed to be asked and the validation and display rules for each question.

Example question

The RESTful API offered two endpoints 1) validation of answers to questions and 2) the details of the questions to display. The Javascript libraries API offered two similar functions 1) validation of answers to questions and 2) which questions to display based on the answers given.

Sharing of code via CLJX and where it runs

The shared data structures and the rules engine were written in ‘plain’ Clojure, enabling us to share this code in both the client side and the server side. The ClojureScript code is compiled into a single JavaScript file that is served from the RESTful API service and loaded by the browser. The JavaScript library acts as a cached version of the core service, enabling instance validation of the user’s answers to questions without the need to communicate back to the server side.

Using the serialised form as consistent input data

In order to use the core rules engine in both places the input data passed to both the RESTful API and JavaScript library need to be consistent. To achieve this we use a map/hash of the form values as the consistent data structure. This works by default for the RESTful API (as the Ring middleware converts the POST parameters into a map) but for the Javascript API we serialise the form into a Javascript hash that is in turn converted into a Clojure map. This allows the input data to be processed consistently by both Clojure and ClojureScript.

The client applications still handle rendering the questions. This is ok as it varies by application anyway. Other concerns, such as validation and what to show, are passed on to the JavaScript library on the client and the RESTful API on the server side.

From a maintenance perspective this means questions can be added, removed and updated in one place, (the Clojure/ClojureScript service) without touching the two applications. Only occasionally do we need to change the HTML layers of the applications (when adding a totally new question type for or applying custom styling).

The new architecture - changes to questions now happen only to the Clojure/ClojureScript code

Using Prismatic Schema for validation

We use Prismatic Schema to handle the validation itself. Prismatic Schema allows you to validate clojure data structures and supports both Clojure and ClojureScript.

You use Schema by first declaring a schema data structure:

and then validating a clojure data structure against this schema:

Check out their github page for more details.

Sharing code with CLJX

How CLJX works (taken from the CLJX documentation)

To share code between the Clojure and ClojureScript we use the Leiningen plugin CLJX. It automatically copies the shared code into each project at compile time. In practice there are subtle differences between the Java and Javascript platform that are targeted. You can write preprocessor rules with CLJX to vary portions of your code for each target platform. In our case this turned out to be a handful of functions for data parsing/arithmetic and logging. A gotcha we encountered was around regular expressions - be careful not to use any platform specific tokens (such as \A and \Z) and test your regular expressions on both platforms.

An example of a function that varies between the Java and Javascript platforms

Keeping the Javascript API library small

Although we weren’t obsessed with Javascript file size we were keen to keep it small to ensures responsive page load times. As such we monitored the size of the Javascript file during development and did some optimisation to avoid file size bloat:

  • We only included libraries of which we were using large parts. At first we included a ClojureScript version of cljs-time to utilise the functions it offered. This added several KB to the final file size so we replaced it with two of our own custom function.
  • We only included the namespace we actually used! We experimented with schema.coerce early in the project. In the end we didn’t actually use it, but forgot to remove the require namespace in our code which increased the final file size.
  • We minimised our data. Some of the data at the core of the API was only used by the RESTful API and was unused by the Javascript library. We wrote a custom macro to remove these keys and values from the maps used by ClojureScript.

Our macro to optimize maps

These optimisations reduced the minified and gzipped file size from 69KB to to 45KB.

The sweet spot

Overall we are happy with using Clojure/ClojureScript in combination for this use case. It has given us the ability to make changes in a single place that are reflected not just across applications but across both the client and server tiers of those applications.