How Banking on the right database (s) will accelerate your post-COVID Recovery

By Martin Gaffney, Vice President, EMEA, Yugabyte, the leader in open source distributed SQL databases

It’s just too risky for a bank to rely on one data engine for all your applications and needs in the post-COVID digital economy, warns database expert Martin Gaffney.

At many global banking and finance organisations, IT is charged with looking after a complex mix of cloud and on-premise technology.  And while you had probably learned to live with all that complexity until COVID, those older, monolithic databases you rely on are struggling with the new challenges of having to support high resilience, greater scale, security and geo-distribution.

This is leading to the break-up of the old, reliable way of working with banking processes—at least, at the most forward-looking organisations. The days of one platform for all use cases are starting to draw to a close, and a new wave of embracing a best-of-breed approach to meet the opportunities digital transformation is opening up.

I say ‘digital transformation’. But to be honest this really kicked off when we all started to move to the cloud. You can call it digital transformation, or you can call it modernisation, but it boils down to the same thing: moving more to online, adding new services, extending into new territories, which also means taking on a further regulatory and compliance load. And luckily, cloud opened up the possibility, after decades of reliance on legacy systems, of true banking back end modernisation—especially with the rise of so-called ‘microservices’.

The constancy of change

There’s a lot of technical detail involved that I am going to simplify here. But what happened next can be summed up in one word: change. You need to be able to change, and quickly. Customers who spent months locked up in their flats and houses learning how to buy and live and work online have certainly embraced it: and they expect you have, too. (And if you haven’t, your competitor is literally one click away, so it’s not great to be you.)

So digital banking and constantly improving the online CX has to be part of the difference between you and irrelevance now. But every amazing little bit of app functionality has to be backed up by solid, mainframe-class and mainframe-scale accuracy, security and reliability, unfortunately.

It would be ideal if you could do all this new whizzy digital payments and banking stuff with just all your beloved, expensive Big Tin, but you can’t.  So, the main attraction of a microservices architecture for the banking community is the simplicity with which banks can add exciting new widgets that pop up to give the user a great new enhancement or feature.  However, much of the heavy lifting to make this a reality, will all happen in the data layer at the back.  And there were initially two kinds of databases here: they’ll either be an SQL database or, perversely enough perhaps, a NoSQL one.

An SQL database is always amazing at doing transactions, making sure that if I update this thing here and that thing there, they’ll either both succeed or they’ll both fail, which is vitally important in the world of banking, where everything is pretty much effectively transactional. SQL databases have been around for decades and are the mainstay of the industry and are what we have been happy for years to buy from the Oracles, Microsofts and IBMs of the world.

Buying expensive machines that sat half-idle for most of the year

So why would you have anything else? The NoSQL database is a lot younger and came about because of that shift to the cloud. Because having these big Oracle databases that are good at transactions meant having to run it on these huge, superfast clusters of Sparc processes or other specialist in-house hardware to get the best out of them. That meant that if you wanted to grow more customers or you wanted more data to play around with or wanted to do more transactions more quickly, then you had to fork out for a bigger machine in your datacentre.

The emphasis here, note, is always machine singular: the kind of SQL databases a bank needs runs on one machine, or a kind of cluster of a close couple of machines, and 99 times out of 100 in the one same physical space, too. The NoSQL guys came along and said that doesn’t work in cyberspace, though: how do you flex up and down, say, so if you need a lot of processing for short periods of the year, around like Christmas or less in the summer holidays when you don’t need that capacity, you end up buying these expensive machines but have them sitting half-idle for most of the year. (There’s another angle here about resilience and business continuity that I won’t get into, apart from to say doing that in this model basically meant having a shadow big system and all the expense that goes with that.)

Plus, to make cloud processing work for banks, you can’t get those great big machines on the cloud—by definition, for the cloud you need virtual machines that work out of lots of standard commodity boxes chained together. If you want more power, you add more of them, but your trusted SQL databases don’t like to run like that; they like one big, dedicated machine.

NoSQL databases came along and they cracked that. You could run them on 1, 3, 10, a hundred small cloud processors; you could scatter those in different datacentres in different regions of the world, you could scale up or down as your business liked and also you become more resilient, as there isn’t one machine to fail or one datacentre to conk out/be deprived by week-long power cuts.

Database victory… at a cost?

This was (and is to some extent) all great; you increase efficiency but also lower your risk. And as a lot of them were open source as well, you were also escaping from those huge license fees you had to pay for those nice but pricey proprietary relational databases. So, great: you’ve cracked a load of problems; you fixed scaling up, scaling down, you’ve moved all of that capex off the IT balance sheet and escaped vendor lock-In.

But… you knew there’d be a but: this is life, after all. To do all this, NoSQL, be it from standalone vendors or big cloud providers, threw away a rather important thing to do all this. There is now a space between old legacy, monolithic, relational SQL transactional databases and then more modern, more flexible, distributed, cloud-native NoSQL ones. And it’s a chasm marked by the critical transaction bit. The ability to do transactions—that rather important bit we were able to do so easily in the old days of if I move this money from my account into your account both ends of that really need to happen—got left behind.

The NoSQL database guys say they can do it. But they do it by cutting corners and they cannot guarantee robustness here. So, your adoption of NoSQL has to be restricted to use cases where that kind of transactional rigour is not needed: notification apps, great, no problem; a system where if I change my profile picture on my account and somebody in Singapore looks at it and sees the old picture and then hits refresh and sees a new one, no big deal; it doesn’t matter.

But if you’re moving money, it could have mattered a lot. So the NoSQL database has great penetration in new use cases, but for serious work it’s a lot of horribly complicated programming by the developers to try and enforce the old rigorous transactional consistency. Luckily, a new player has entered the game which solves all these issues: the distributed SQL database, which has a similar architecture to Oracle and SQL Server but is horizontally scalable across multiple commodity boxes and capable of being geo-distributed.

Distributed SQL is great because it is the best of both worlds: it has the rigour and ability to support transactions at scale like the old ‘monolithic’ database, but also offers something just as good: support for resilience. That’s incredibly important in a cloud scenario for a bank. I mentioned the old business continuity approach earlier, which was all about those twin systems; it’s actually a real pain having these mirror machines, which are complex to run, can even fail themselves, but most importantly can lose data at switchover time at a pinch point.

A synthesis of old and new

So to sum up my little history lesson of the last 20 years of banking IT: we had old monolithic databases that were excellent at their job but weren’t going to cut it at all in the cloud; we then had NoSQL databases which were really cloud-friendly, but lost transactions—and then you’ve got a response to that in the form of Distributed SQL, which is an SQL database that can do transactions, is really flexible and can do nice things for you too like sharding (spreading the same database across multiple machines).

My contention, then, is that to move to the next stage of digital banking can’t be done with either monolithic relational SQL or NoSQL. A synthesis is the way forward, which is an open source, distributed SQL technology approach that can bridge the two ends of the banking data spectrum nicely and neatly, allowing you to make the most of things like microservices and cloud equally and satisfactorily. Even better: given security is a concern for banks, some distributed SQL providers will prioritise security, so you should also be able to address tokenisation, encryption at/rest encryption in motion, authorisation and roles and access control.

If you stick to one of these two basic ways of working with your bank’s data, (Monolithic relational SQL or No SQL) the cost, complexity, potential vendor lock, and the price of maintaining your infrastructure and resources is going to be huge. You could get to a fully digital and agile future, but it’s going to be slower and cost a lot more.

I say ‘synthesis’ of the main banking database architectures. But I really mean synergy: a state of the sum adding up to way more than the parts—and which is the way your organisation will absolutely triumph as we exit COVID into a new, if unpredictable, banking world.