Adoption of Open Source Technologies Is Increasing in Financial Data Management – but What Are the Challenges?

By Mark Hermeling, CTO, Asset Control.

Open source software is software for which the original source code is made freely available and may be redistributed and modified according to the requirement of the user. The roots of the approach lie in the individual passion and enthusiasm of groups of software developers. Yet, open source has now become a mainstream commercial business model and we are seeing a rapid uptake of the software by businesses. According to Markets and Markets, the global open source services market is expected to be worth $32.95 billion by 2022.

Financial services is among those key markets seeing strong growth in open source software take-up. Over the past 12 months, adoption has reached a turning point in this sector. Institutions are now using more open source not just to help them reduce costs as they scale but also because they are seeing productivity benefits from open source that allow them to deliver projects more quickly.  At the same time, an increasingly broad set of community-maintained functionality has become available and there is a growing ecosystem and skill base that both solution providers and internal IT departments can leverage, making it easier for them to attract and retain talent.

Mark Hermeling

As well as helping clients reduce software and infrastructure costs, the increasing adoption of open source is fuelling innovation across the sector and has become increasingly key for firms across financial services. Open source is ideal to use in sandbox environments and laboratory environments, often leap-frogging existing legacy technology stacks. The psychology of engineers also often fuels further innovation. Typically, they like to work on cutting-edge projects. They also often appreciate the peer recognition they will get from contributing to open source projects. It is also the case that using the software makes it easier for firms to start small on new projects without having to go through a protracted procurement or tendering process.

This in turn can deliver crucial time to market advantages, accelerate the development process and – through lower cost – quickly build a portfolio of innovative projects. In addition, the use of open source helps firms avoid all the pitfalls and dangers of ‘lock-in’ associated with proprietary tech.

Moreover, adopting open source typically means deploying cloud native apps and migrating workloads to public or private cloud built on open source infrastructure. Open source often provides foundational technology, including languages, libraries and database technologies that can provide a rich foundation to quickly develop applications. That, coupled with an increase in the uptake of managed services options, is making open source still more attractive to financial services businesses – and is further driving innovation within these organizations.

The use of open source also helps to enhance systems security, including cyber-security, for financial services firms. That’s because if an organization is using open source technology that is being leveraged by a large community, it is unlikely, statistically speaking, that it will be the first to catch a bug.

Gauging the Challenge

That said, firms will also have to navigate a host of challenges with regards to open source such as choice of database technologies, support requirements and potential implications of using open source licenses with IP they are developing. Firms also need to ensure they are picking the right open source projects where they will attain optimum value and also ensure they are using the right open source tools. Just as in the world of commercial software, there will often be a range of competing tools available to them which could potentially be used to tackle a problem and choosing the right one is critical. Some technologies, such as Python, Spark and Cassandra, have caught enormous momentum. Others may have lost it. So it is important that firms do their normal sourcing homework.

Aside from these more general challenges, financial services firms will be likely to have more specific data management issues that they need to address. They may well want to use NoSQL database technology that came out of open source for data management purposes. Cassandra is good for time series data modelling, while Spark is effective as a data processing framework. As financial services firms look to get more out of their data and source more data, data scientists need to be properly equipped both with the requisite data preparation and data quality solutions as well as with the tools they would need to analyse the data and test their data models.

In addressing a move to open source, firms should look to leverage the help of curated, open source solution providers that use open source themselves and therefore benefit from some of the advancements that have been made in order to deliver cost-effective scalable solutions. By partnering with a commercial provider in such a way, firms will also be able to access support from providers. In other words instead of taking on the onus for leveraging the technology alone, the onus will be on the provider to deliver the underlying technology. In short, it will enable financial services organizations to optimise their deployment of open source and get the most they can out of the technology today.