The Challenges of Cloud Native Development

Jon Payne, database engineer, InterSystems

Months have passed since Devoxx 2018, but some of the conversations I had with the developers visiting our stand there still stick in my mind, probably because I found one or two of these talks so surprising.

One of the main topics for discussion was building cloud native applications. Of course, I had known that a build-your-own approach here was becoming popular, but I hadn’t appreciated the level at which it had permeated the entire development ecosystem.

Many of the developers I spoke to had a couple of things in common; they all came from organisations that owned huge volumes of data and they all needed to create a process involving both analytical and transactional workloads with that data using a variety of different technologies.

These analytical workflows were varied in nature. Many were finding that being able to run SQL, for example, wasn’t enough to solve the functional and non-functional requirements of queries in an adequate manner.

Consequently, a number of developers were building their own cloud native data management platform. What I found interesting was the number of different organisations feeling the need to do this given there is such a wide variety of cloud native – and also on premise platforms – out there that are well-known and are in the SQL and NoSQL space. Yet, they can find nothing on the market to suit their needs.

I find this remarkable because they may see it as the most cost-effective option to begin with, but, it is likely to turn out a much less economic option in the long-term. What these developers are building is very specific to a particular problem – and as far as addressing their challenge it is likely to be an initial success. However, there is considerable risk inherent in this approach.

All will be well if those developers building the solution remain with their organisation. However, if they decide to leave – and let’s face it the competition for developers couldn’t be stronger – then their existing employer either has to offer them more money, or face a knowledge vacuum surrounding the platform, possibly having to bring in expensive consultants to the rescue.

The other issue is a matter of functionality. Once the organisation wants to do something extra with the platform they will need to set up a data pipeline and replicate it in another data store, reconstructing it along the way. Before they know where they are, they have the same data replicated in four or five different structures. Suddenly, what started out as a cost-effective platform developed for a particular purpose has become both expensive and complex.

Interestingly, this was one of the reasons several developers told me they are not going cloud native. This ramping up of cost and complexity is not easy to manage. Besides, if you consider Amazon S3, AWS or any of the other low-cost cloud platforms, they are not fast mechanisms. It might take 20 seconds to do a read and there is no performance guarantee. If a business is data-intensive then it does beg the question as to whether cloud native is the right route.

This is especially the case if an organisation needs specific hosting requirements. For example, a healthcare company may need to be connected to N3 or HS3N. If so, the costs will rise dramatically as the data can’t be accommodated in large-scale AWS racks but must be kept apart.

Of course, there is a plus side, especially if a developer makes use of all available services offered by the cloud provider. This can significantly reduce the time to build and deploy a solution – and ensures it is highly scalable. However, this does tie the organisation to a particular cloud provider as, if a solution is built, for example, in AWS, it can’t be moved. Then as transactional values increase, data volumes grow and complexity intensifies, the costs can increase again quite dramatically.

Traditionally in the database market we used to talk about ‘asset compliance’. In the cloud world, because of the way the infrastructure works, providers are unable to offer this as we used to know it. Instead the big providers have redefined the concept so they can comply. While in some cases this may not be important, it can bring a whole host of issues to the fore when building apps that are critically dependent on the consistency and quality of data such as real-time clinical applications.

Yet despite all these drawbacks, developers are still building cloud native applications because they can’t find what they want on the market. This bodes well for solutions such as InterSystems’ IRIS Data Platform which, with its flexible architecture can meet varied transactional and analytical workloads and interrogate data in a number of different modes, not just in SQL or object or document-based.

What could also make IRIS so valuable in these cases is its interoperability; in particular, its ability to integrate data and applications into seamless, real-time business processes. It can also cut through the complexity, collapsing the tech stack and managing multiple open source components, all in a highly cost-effective manner.

Perhaps I shouldn’t have been so surprised at the number of developers at Devoxx building their own cloud native applications. After all, they are rather a self-selecting band, given the nature of the event.

However, the most exciting aspect here is not that that such a technically inquisitive and inventive group are doing it themselves, more that they are being forced to do so despite the shortcomings of cloud native because of a gap in the market. Which all means current new market developments are certainly moving in the right direction.