Part 2: Analysing the data
Intro
If you haven't seen it yet check out Part 1 to see where all this data is coming from. There is also a Demo Dashboard if you want to see the final result.
Now all the data is being collected and stored in a Postgres database on AWS the next task is to use & analyse the data. For that I needed an API to calculate the information I am interested in and a pretty dashboard to present it in a useful way.
My First thought when thinking about how to do this was something like: React, Redux, Babel, Storybook, Jest, React Developer Tools, UI Component Library, … and more. I was making the mistake made by many devs before me (and indeed i'd made before), over-engineering.
All I need to do is call the API and display the values on the page, no interactivity, no collaboration, no customization. This can be done easily with plain JavaScript in a fraction of the time!
Overview
The design was simple. A JavaScript function would call a series of APIs in turn, then pick out the data and write the values into the appropriate HTML divs on the page. With the format of the data defining the type of chart widget to use.
It is secured by simply storing the API key in localStorage and passing that into the appropriate header of the request, if the API key is wrong or missing then the API will error and the data is safe. No need to even program that AWS API gateway supports it out of the box.
Creating each widget
With the data pulled in from the API it can then be placed into the appropriate type of widget, each designed to optimally show that type of data.
Single values
For single values that show current metrics such as the number of users on the site now or the percentage of visitors who are new over the last week there is a simple number widget that shows just that value. Nothing much to say about this one so lets move on.
Table
Next I created a table widget that shows a list of single values. The top 8 are shown by default but it's scrollable so all data can be accessed. The table widget is used for a few things such as the number of page views and average time spent and average scroll distance for each page. This allows me to check which content is of most interest to visitors based on the various metrics. The widget is also used to display OS and browser stats so the system capabilities of visitors can be investigated.
Bar Chart
An important metric that I was interested in was the scroll distance. A measure of how far down the page each visitor got. I decided to show this with a bar chart, showing how many visitors scrolled to each 10% increment down the page, averaged over all pages. Helping to give an general overview of how much of each article people are actually reading (or at least scrolling over).
Trend
The final and most complex was the trend widget, used to show how important metrics such as how the number of visitors and number of page views have changed over time. With the help of ChartJS this wasn't actually too challenging and just involved simply morphing the data from the api into the right shape and letting the library do the hard work.
Conclusion:
The result? Check out the demo here.
Overall the time spent on this project was only around 6 hours, significantly less than it would have been if I'd been tempted to over-engineer the system. Ultimately this just highlights the need to pick the right tool for the job based on the complexity of the project, the time restraints and importantly the number of developers working on it. Often, if the number of developers is one, a huge chunk of the tooling can be skipped to save time with no real impact on the outcomes success.