Time Series Data | Five Engine Requisites for Real-time Analytics

James Corcoran

23 February 2023, 07.00am

Tapping into data’s true power as a business can be difficult, considering the sheer amount that’s available. But in this piece, James Corcoran, KX’s Chief Growth Officer, provides a list of five analytics engine must-haves, so that time series data can be translated into impactful, real-time business analytics.

Data overload isn’t just a 21st century problem.

When Thomas Jefferson became US president in 1801, he would receive around 150 letters every 30 days. Skip ahead 100 years, Theodore Roosevelt needed a separate team of staff to handle the sheer weight of letters he received. By the time Harry Truman was in power, 144 years later, letters arrived at a rate of three truckloads per day.

In 2023, it’s estimated that Joe Biden receives a massive 65,000 letters per week – not to mention all the emails and DMs he receives, and the social media posts about him that people across the globe are sharing on a daily basis. It’s a great demonstration of the data volume challenge every business now faces.

But volume isn’t the only element that’s increasing. Data has become more valuable, and as a result, the decisions made using it are more critical and growing in number by the day.

Take predictive healthcare, anomaly detection, predictive maintenance and operational equipment efficiency, and pre- and post-trade analytics as examples: these are all systems which have come from ever-increasing sources of data.

And that’s before businesses even start to understand what machine learning (ML) may demonstrate when it comes to detecting trends or patterns. Depending on how fast a system can identify them, they could either flag exciting new opportunities or, on the flip side, highlight potential future concerns the business should know.

However, many companies, regardless of industry sector, simply aren’t harnessing the power of their data to generate these insights. They’re too focused on solving technical problems with their data, leading them to become blind to how they could be using it to uncover valuable information.

A key reason for this is that legacy databases and analytics software aren’t able to handle the increased demand that comes with capturing and analysing data in real time.

Below are five must-haves for a modern real-time analytics engine using time series data, so that organisations can begin using such data to meet their overarching business goals.

Putting time series first

While it isn’t widely known, most data today is time series-based, as it’s been generated by processes and machines rather than humans. Any business looking to make use of data’s potential should ensure they have an analytics database that’s optimised for time series-specific characteristics, such as append-only, fast, and timestamped. A high-class system should be able to quickly digest diverse data sets and perform in-line calculations, as well as execute fast reads and provide efficient storage.

Resisting the urge to narrow down systems

Alongside this, the data estate of a typical modern enterprise is large and growing every day. This means any analytics engine a business deploys must interface with a wide variety of messaging protocols and support a range of data formats along with inter-process communication (IPC) and REST APIs for quick, easy connectivity to multiple sources. It should also cater for reference data, like sensor or bond IDs, that will enable it to add context and meaning to streaming data sets, giving the ability to combine them in advanced analytics and share them to make strategic business decisions.

Don’t forget historical data

By combining real-time data for up-to-date insights, along with historical data for past context, organisations can make quicker and better in-the-moment responses to events as they happen. Plus, this can eliminate the development and maintenance overhead of replicated queries and analytics across different systems. The ability to rapidly process increased volumes of data using fewer computing resources is also incredibly helpful for ML initiatives, not to mention reducing TCO and helping businesses to hit any sustainability goals they might have.

Taking on the cloud to create smooth processes

The top thing businesses need to add to their data toolkit is an analytics software built with microservices that allows developers and data scientists to quickly ingest, transform and publish valuable insights on datasets – all without needing to develop complex access, tracking, and location mechanisms.

Complications like data tiering, aging, archiving, and migration can take up important time and resources which could be better used to concentrate on creating insights which business leaders can utilise. Additionally, natively integrating with major cloud vendors and making this available as a fully-managed service is an important consideration if businesses are seeking to onboard a new system with ease.

Trying before buying

Time series databases are not new to the market. However, the ever-growing volume, velocity, and variety of data, alongside the pressure to create rapid insights and actions from it, means many technologies have yet to be properly tested within the wider market. But business leaders need to remain vigilant and be looking for software where there are robust use cases and clear examples of return on investment.

Data is a constantly moving target, and businesses need to recognise this and evolve to keep up. It’s now an independent asset with its own C-Level owners, allowing businesses to automate decisions in fields including, but not limited to trading, production, and network.

Plus, it holds more economic value than ever before, as businesses are now finding they pay more for it, but the insights it can generate can contribute greater freedom to businesses.

There are a wide variety of positives businesses can reap from continuous, context-rich data analytics-driven insights based on time series data, such as delivering greater business decisions, enabling enterprises to adapt faster to market changes, increasing customer satisfaction, and ultimately improving the bottom line.

However, this can only be the case if business leaders ensure their teams have the right technology in place at the right time – or they risk losing out to competitors who recognise this new data must-have first.