Feature: Add datavis (chart-builder) app (initial version)

[ababic]

Includes a baseline implementation of chart-builder functionality for Wagtail

• Data sources can be defined manually or by uploading an existing CSV to populate them.
• Visualisations of different types can be defined from existing data sources
• Once defined, a visualisation (or multiple visualisations) can be added to an Information page, and highcharts will render them as a basic chart (not the final representation).
• Implementations for all chart-types earmarked for inclusion for day one release are included (the 'additional data souces' option means we do not require a separate type for each current optiom)
• Annotions are implemented from a BE perspective, but require more FE work to support fully.
• Whilst we don't have a concept of 'Template charts' (as outlined in the original requirements), the 'Copy' function means that any chart can be used as a template to create another.
• Whilst you can't freely change the 'type' of an existing chart (as outlined in the original requirements), you CAN choose a new type when 'Copying' an existing chart, which I believe is a good trade-off.
• Data sources and Visualisations are accessed in Wagtail via the 'Datavis' menu item, which provides access to both indexes

How to review

Data sources

The data source index/listing

The listing for data sources has been customised to include additional information about each item, including:

• How many columns it has
• Who created it
• When it was created
• When it was last updated

The listing includes a search option, which will filter the list to include matches on 'Title' or 'UUID' (if known), and also includes a number of filter options to help identify items of interest:

Adding a data source

• When creating a data source, a "Populate from CSV" field allows a CSV to be uploaded in order to populate the "Data" field. When used, the uploaded CSV is parsed on save, and any data present in the 'Data' field will be replaced with the contents of the CSV.
• After initial creation, the "Data" field is used to make edits, or the "Populate from CSV" option can be used again to upload a fresh copy of the data.
• Uploaded CSVs exist in temporary storage only and aren't saved to permanent storage. After they have been used to populate the 'Data' field, they are forgotten about and should be garbage-collected from the filesystem automatically in due course.

NOTE: The "Data" field (particularly the custom streamfield block that provides the table-like interface) works well from a UX perspective, but doesn't do a great job of storing data in a way that can be easily used by visualisations. So, when a data source is saved, and the 'Data' value has changed, the value is parsed to extract the row and column values, and convert anything that looks like a number to int or float values. This extracted data is then saved to a hidden data field on the model as JSON, which can be efficiently pulled out of the database with all of the numbers values in-tact, so that no on-the-fly reformatting of data is needed.

Previewing data sources as different visualisation types

When creating or editing a data source, the preview tab in Wagtail allows you to easily see what the data could look like as different types of chart (using the default settings for each)

The available preview options are hardcoded on the Datasource model, and do not update automatically to reflect newly-added visualisation types.

Copying a data source

A data source can be easily created from an existing one using the 'Copy' feature, which can be accessed via the item-specific option menu in the listing:

For data sources, this uses the stock 'Copy view' that Wagtail provides for snippets.

Visualisations

The visualisation index/listing

The listing for visualisation has been customised to include additional information about each item, including:

• The type of visualisation
• Who created it
• When it was created
• When it was last updated

The listing includes a search option, which will filter the list to include matches on 'Name', 'Type', 'Title', 'Subtitle' or 'Caption', and also includes a number of filter options to help identify items of interest:

Adding a visualisation

When adding a visualisation, the first step of the process is to select which type of visualisation is required.

This step is required to allow the second step to be able to identify the relevant model class and render a UI that makes sense for the selected type. Currently, only the 'Bar chart' type makes any significant changes (to relabel the X axis and Y axis options), but this is likely to change in future.

In the second step, the new visualisation can be previewed without saving, as soon as all of the required fields have been populated. Like with pages, any changes made to the available options will trigger an automatic refresh of the preview (provided the form has no validation errors to resolve).

All visualisation types require a "Primary data source" to be selected, which provides the data used when rendering. Because we're using Wagtail's native snippet chooser here, the chooser is permission-aware, and only presents datasources for selection that are visible to the current user.

All included chart types currently render using a common template and JS. This will change to use one or more official Design System component when they becomes available.

Adding annotations to a visualisation

Annotations can be added via the "Advanced" tab in the edit interface for all current visualisation types. For each item, the editor must provide the label text, X position and Y position. The positional fields should correspond to "values" on the X and Y axis, but no validation is currently applied.

Annotations are not rendering reliably yet, because Highcharts doesn't support defining axis position values that use strings out-of-the-box. Further JS customisation is needed to evaluate annotation values when the chart is loaded and convert string values to coordinates that Highcharts will understand.

Adding additional data sources to a visualisation

These can also be added via the "Advanced" tab in the edit interface for all current visualisation types. For each item, the editor must select the data source and choose how the data should be rendered:

The "Include in downloadables?" option is ignored currently, but will come into play when we eventually come to generating download links for the various sets of data featured in a chart.

Copying a visualisation

A visualisation can easily be created from an existing one using the 'Copy' feature, which can be accessed via the item-specific menu in the listing:

Rather than taking you to a fully-populated form like the Wagtail's built-in version, this custom version only has two fields, allowing you to specify a name for the new visualisation, and also choose a different visualisation type to created it as:

The new object is created using the same data source and configuration options as the original, but is saved as an instance of the selected type.

Currently, all visualisation types are similar enough that the values are compatible across all types. However, in future, this is less likely to be the case.

Featuring charts on a page

The DataVisBlock is available as an option on the "Information page" only for now. The block allows you to choose from any visualisations visible to the current user, and forces the editor to provide accompanying 'Title' and 'Subtitle" text, which replace the values set on the selected visualisation when the block is rendered.

When the page is rendered, any additional media required for rendering visualisations on the page is collected together, deduplicated, and included in the <head> section of the page.

Developer notes

Permissioning

For now, both the Datasource and Visualisation models use the CollectionOwnershipPermissionPolicy from Wagtail, which is the same approach taken for images and documents, and is outlined here. More information about collections in general can be found here.

This can be reviewed / amended at a later time as necessary.

Follow-up Actions

• It's very important that we get this work in front of people (including the Datavis, Publishing and Design System teams) to start gathering feedback, so that we can figure out where to go from there.
• Permission-model refinement.
• Iterarate the current solution based on feedback from various teams.
• Current templates / front-end JS to be replaced with Design System components once available (and configuration-preparation logic on visualisation classes revised accordingly to match the API of each component).
• Datavis Block to be made available on all relevant page types.
• Unit/regression tests to be added later (before launch) once features / models / options are considered stable.