Painting a complete picture with unstructured data

Published : 5 years ago, on 30/04/2019

By Tim Pullan, CEO and Founder of ThoughtRiver

The Financial Conduct Authority(FCA) has asked for input on how technology can help achieve smarter regulatory reporting[i]. This is unsurprising given the IFRS9 accounting rules that came into force in January and the introduction of the GDPR on the horizon. There will also be plenty of regulatory uncertainty around Brexit. Most firms in the finance industry rely on a suite of technology to help meet regulatory obligations but many tools do not provide sufficient legal context around commercial data. Artificial intelligence (AI) tools, combined with intelligent schema,can supplement traditional tools, helping extract useful data from unstructured documents such as contracts.

Traditional software tools work with structured data to meet regulatory obligations.However, the process of populating these tools is not efficient because much data, the so-called ‘known unknowns’, has to be extracted manually from unstructured documents. And too often the database does not capture legal restrictions and permissions related to commercial data.

For example, when dealing with revenue recognition in accounting, traditional tools might only report revenue figures aggregated from data that is easily populated. However, if the system does not take into account the warranty clause in the contract that specifies the circumstances when that revenue is not recognized, it provides only a partial picture. To obtain the full legal context and complete the picture, firms must find data-across contracts or other unstructured documents. Currently, many firms are doing this manually and most firms are not uncovering all the relevant data.

Potential applications of AI tools with intelligent schemas

AI tools that combine machine learning with natural language processing can analyse and summarize contracts and unstructured documents efficiently. It is easy to extract and organise data in a structured manner almost entirely without the need for human supervision.

AI tools that run on machine learning require users to input samples of relevant documents containing the issues they are looking for. These issues must be labelled for the intelligent software to build algorithms to extract those types of issues accurately in new documents. The best tools allow users from various organisations to pool AI training for the issues that they all share. This way, users do not need to reinvent the wheel developing their own AI in silos. When AI training data is pooled, accuracy and applicability improve for all users.The latest intelligent tools have even achieved a level of sophistication that enables them to summarise the meaning of data extracted.

This sophistication is important because it allows the AI to establish relationship between data fields, providing a layer of context and meaning.It converts a simple database into an intelligent schema. The advantage of using an intelligent schema is that users may field complex queries to the AI, which the AI then answers using both extracted data and the hierarchy of relationships between those data.

Two potential applications for AI tools coupled with intelligent schema are:

• Asset Classification: Under IFRS9, firms are required to perform the ‘contractual cashflow’ test or SPPI test for asset classification. An AI tool can trawl through contracts of varying complexity to determine whether cashflows are limited to principles and interests or if there are other types of compensations involved[ii].The AI can quickly determine whether something passes the test by pulling out the exact provisions and looking at how they relate to the two outcomes. In this case, that would involve determining whether exact provisions involving non-principle and non-interest cashflows exist. The meaning of relevant provisions is understood by the AI with regards to the intelligent schema even if no exact wording is found. In this case, the intelligent schema can understand when cashflows occur, whether payment terms occur only under rare circumstances (“not genuine”), whether non-payment results in bankruptcy rights or what contingent payment terms exist. The AI relates answers from each of these features to determine the outcomes to this binary test.This information also makes the process of modifying future instruments to fall under desired asset classes much easier.

• Stress Testing: Stress testing has become a fact of life in the industry since the financial crisis of 2008[iii]. Insurance firms use stress and scenario testing to consider the potential impact of adverse circumstances on their business. Banks must hold capital against several types of risks and must regularly prove that their capital is sufficient. How firms fare during stress testing – and in real life – depends heavily on which legal obligations are triggered across thousands of contractual relationships. Many firms utilize various statistical sampling methods to deduce non-credit risk because reviewing all contractual obligations manually is prohibitively expensive. This does not always capture scenarios of “Black Swan events” well. AI tools can be trained to unearth relevant obligations and their temporal or geographical contexts. This allows a more accurate picture to be painted during stress testing and speeds up the process.

The types of insights obtained from AI tools coupled with intelligent schema go beyond simply helping firms meet regulatory obligations more quickly and cheaply. Commercial data with legal context is extremely valuable because of the visibility it provides of the firm’s commercial relationships. This visibility can be used to identify and mitigate previously unknown risks or to develop new business opportunities in the increasingly competitive banking and finance sector..