Data lakes are dead. Long live data in motion

By Nick Gordon, CEO and co-founder of Adnitio Insight

Financial institutions are facing a myriad of operational risks to such an extent that regulators are becoming increasingly interested in operational resilience practices. The reality is that beyond just being a headache, failed transactions and processes can often lead to customer dissatisfaction and reduced profits. One major issue faced by these institutions is the speed to which processes happen internally vs the capabilities of tracking these data points as they move through the systems. Without adequate tracking, companies can fail to catch lost payments, liquidity issues and other disruptions, leaving them open to risks and loses that cannot be fixed in time. These issues almost always have to be dealt with long after the fact.

Financial transactions now happen in real time, but antithetically the data capture engines used by these institutions are not. Instead, they remain static and often the data they collect remains trapped in a highly complex, often difficult to maintain database. A transaction, by nature, is not static, but dynamic, involving data in motion as the transaction traverses different systems. Today, processing and monitoring needs are intra-day, not end-of-day as they once were. The current end-of-day process leaves businesses struggling to understand where their data is, where it has been, and which processes it has gone through; often companies are forced to rely on a large team to search for this information. At the moment, when a transaction goes missing the team needs to manually check the database many times over. In this circumstance, the time and money lost by the failed transaction is multiplied exponentially when one considers the time lost looking for the transaction in the first place. The complexity of these transaction processes means the giving a simple tracking number does not work either, as the data moves through multiple business types such as trade, settlement, and payment, requiring many transformations.

This can only be solved with innovative technology. Data at rest has its uses, but data in motion is the remedy for many of today’s business woes. It is a solution which can bring operational resilience, increased profits and unlock the true potential of a business. This means having a system in place that can truly — finally — capture data in motion in real time. By doing so, a business has the potential to gain real-time end-to-end visibility, thereby supporting compliance with operational resilience regulations. Real-time business alerts, real-time data lineage collection and statistical business dashboards, with full transaction detail available at the click of a button, give companies the ability to respond immediately to any disruptions.

Not only that, but for businesses to react in real time to issues, an easy-to-use visualisation tool aimed at the businessperson is vital. Complete visibility into financial processes allows for financial products to become intra-day in their execution, enabling the company to increase profitability. Imagine what you could do with a real-time view of your clients’ data as it moves across your business environment:

• Identify failed transactions in real time, allowing for rectification intra-day.
• Identify lost opportunities through the analysis of rejected transactions.
• Proactive insight into risk and liquidity in real time.
• Identify trends in client behaviour and track it in real time.
• Provide data to allow settlement breaks to be reduced over time, thereby reducing costs.

Without this capability, a business today will always be behind the game. To understand why, an easy comparison can be made to a F1 team. Today, the monitoring and alerting on these high-tech machines rely on real-time capabilities. Knowing after the fact that an engine is overheating, a tyre is losing pressure, or the fuel is being used too fast would result in the car not making the end of the race. In this circumstance, one would expect the team to fail. Through the introduction of real-time data, the F1 team has instead the capability to build in automatic engine alerts: to cool the car, set the engine to a more economical fuel usage, change the engine management system to optimise performance, or take action before the tyre explodes. Relying solely on people, such as the driver, to understand and feedback is not a winning strategy anymore. Financial services companies are today running blind, without “eyes” on their core transactional systems, data and failures in real time. This lack of visibility impacts business. They have grown used to rely on out-of-date data to make business decisions, which without a real-time view are only made reactively, not proactively.

With over 20 years of transaction tracking experience, we can see that the volumes and complexity of transactions have increased exponentially. With increased transaction volumes come more and more steps that must be performed in a fraction of a second across different systems. The Faster Payments initiative is a recent example, which facilitates real-time payments of up to £1 million for most people or businesses with a bank account. The fact is, data lakes will not be able to perform, and inevitably they will turn into data swamps.

The only way for financial companies to stay competitive will be through real-time visibility of business operations, which is not possible if data is not analysed proactively. The good news is that there is a new breed of technologies emerging that can be rolled out to give this visibility today. Data lakes have been dying a slow death, it’s time we finally laid them to rest and embraced the next generation of data in motion.