Simplify & automate how you manage cybersecurity risk

Putting the Viz in Dataviz

Creating a custom visualisation framework using D3.js

When considering data visualisation in web applications there might only be a few underlying technologies to choose from, but the landscape of frameworks and libraries utilising HTML5 and JavaScript can quickly become confusing.
At the start of our journey to create visually appealing, but most importantly user-centric visualisations for effective communication of information, we decided on a small set of options:

  1. Premium or open source frameworks that offer a set number of pre-built visualisation types and sometimes additional options for customising design and interaction.
  2. Building our own framework, perhaps by utilising libraries for better data manipulation and handling.

The appeal with the first option is obvious: You’ll hit the ground running by simply making sure your data is in the framework’s expected format and by passing some configuration telling it what type of visualisation and surrounding ink to throw back at you.

Certainly, simple use cases will benefit from this option. But when settling for a framework, do make sure it meets your long-term requirements:

  • Will it support the types of plots you need?
  • Is the data structure it expects easy enough for you to generate?
  • Can you modify or configure the design and interactions to suit your needs?

Evaluating our options and use cases, we soon realised no out-of-the-box solution would quite cut it. And whilst making alterations or customisations to existing frameworks could be considered an option, it also means taking on some risk. Incorporating future updates to the underlying framework with your own adjustments added whilst keeping code maintainable can be a big challenge. So instead of ending up with a hybrid of framework and hack code that is hard to manage the clear choice was building something bespoke.

During our research, it was impossible to avoid mention of D3.js (which stands for Data Driven Documents) when reading of flexible data visualisation on the web. So it quickly became a very obvious choice and presented itself as one of the most mature viz libraries available.

However, do not mistake D3 for just another viz framework. It is a toolkit that enables you to manipulate data and create highly custom data visualisation from scratch. Although the core is well-documented and there are plenty of examples of different use cases, getting started will require some more in-depth tech savvy.

So, how did we go about planning and ultimately building a bespoke visualisation framework? Firstly, we gathered our requirements, which at a high-level looked something like this:

  • We need a framework allowing us to render an extensible number of visualisation types from a common data structure.
  • Visualisations should be able to interpret a single, standard data structure independently and render this model after a chart type’s minimum data requirements have been met.
  • Additional configuration needs to allow alternative interpretation of the data as well as custom display and interaction settings.
  • The framework needs to be easily extensible to new visualisation types and interactions.

Important in achieving this flexibility was getting the basic structure of our framework right. So we decided to structure the framework into components as follows:

  • Dependencies: These are libraries or applications our framework depends on.
  • Core: This is the core code base to our framework, all visualisations have access to. It is used to enable common functionality shared across the framework.
  • Modules: These are the pluggable components – mainly the different visualisation types themselves. Removing any of these will still leave the framework fully functional and only reduce that particular extension.

So let’s have a look at these three areas in a little more detail.

Dependencies

As I touched on earlier D3 is not really just another viz framework. It is much more a library helping us to firstly work with data in JavaScript and ultimately display it visually. To do that D3 supplies a whole range of helper functions we can use to structure and sort data, but also to apply common mathematical computations.
Further the underlying idea of D3 is not to give you out of the box viz functionality, but to help you “bind arbitrary data to a Document Object Model (DOM), and then apply data-driven transformations to the document” (from: Introduction to D3.js).

In a nutshell that means creating corresponding DOM elements to your data and being able to then manipulate them visually and interactively using CSS and JavaScript to create data-visualisations. I hope by now you can imagine we are great fans of D3.js and it is really our only main dependency.

Core

At the core of our framework sits a thin service layer and a common controller. Both are available to all modular components and supply a whole range of methods and options to use and handle all the groundwork required to set up a visualisation instance. That includes for example generating the basic common markup, retrieving data from APIs, supplying colour schemes and offering methods for number formatting and binding common interactions.

For illustration purposes here an example of a skeleton with potential core functions of such a framework. Variables are prefixed with $.

Modules


We extend the core of the framework by setting up modules for each of our custom visualisations (e.g. there is a histogram module).


One approach to help make the code more maintainable would be to adhere to a fixed structure for these modules. By using a naming convention for frequently used methods we would be able to invoke these from the core of the framework rather than within the modules themselves. We can then handle event chaining and running order in one place, the core, and avoid introducing repetition of this process within the modules (see code samples above lines #38 -> #58).
One downside of this approach is the slightly tighter coupling between the core and the modules. In this setup, the automatic chains do not run if the module methods do not follow the naming convention. However, as they are conditional, they fail gracefully and modules can invoke their own methods if an alternative approach is needed.

On a high level the convention for methods within a module could look like this:

  • Process Data: A method that interprets our data structure in the way needed for the type of visualisation. This is an optional method as not all modules will require it.
  • Setup Viz: A method setting up all basic functions and generating all necessary base DOM elements required to paint this type of viz.
  • Paint Viz: A method actually painting the visualisation. This function needs to be set up in a way that it can be repeated in case of data change for example. This then needs to redraw the viz without adding unnecessary repetition.

We believe that this general architecture and approach allows for the framework to be extended quickly through additional modular components, or by rolling out common features through the core layer of the framework.

Our main application is build using AngularJS so naturally we also make use of some of its features in our visualisation framework. More on integrating D3.js and AngularJS in a future blog post!

No comments yet. Be the first one?

Leave a Reply