Technology
Dealing with the data deluge: Big Data skills most in-demand
Published by Gbaf News
Posted on July 31, 2018
9 min readLast updated: January 21, 2026
Global Banking and Finance Review
Company
Published by Gbaf News
Posted on July 31, 2018
9 min readLast updated: January 21, 2026
Martin Ewings, Director of Specialist Markets, Experis
Businesses across every industry are facing unprecedented disruption; with tech giants such as Netflix, Amazon and Uber transforming how services are delivered and uncovering new patterns in consumer habits in the process.
Business models are being rewritten, and efficiencies are being found through data-driven operations, as customers increasingly demand personalised experiences.
At the same time, the much talked about General Data Protection Regulation (GDPR) is shaking up how businesses collect, process, manage and secure their data. Designed to give EU citizens more control over their personal information and cut through the complexities that organisations face when complying with opposing local laws, the regulation is having a significant impact on businesses’ internal processes. Against this backdrop, it’s hardly surprising that Big Data professionals are highly sought after by employers. In fact, demand for Big Data expertise has skyrocketed by 78% in the last 12 months, with organisations under increasing pressure to make better use of their data to both maintain a competitive edge and remain compliant with the new regulation.
It’s the insight that can be gleaned from datasets that makes Big Data so powerful. To achieve this, employers must engage with the right professionals − those who can turn raw data into business gold. Here, I consider the reasons why employer demand for Big Data talent is on the rise, as well as the specific skillsets that are currently needed in enterprises.
Turning data into actionable insights
With the steady rise in adoption of sensor technologies, including the Internet of Things (IoT), we can expect datasets to grow and, similarly, the demand for Big Data specialists. More specifically, contractor demand has grown by 128% in the past year and permanent roles by 68% over the same period. This increase outstrips demand across other important skills markets, including Mobile (26%) and Cloud (30%).
Sought after skills
As business leaders continue to grapple with the complexity of running increasingly data-led organisations, the most in-demand skills for specific Big Data roles include AWS, Python, Hadoop, Spark, Cloudera, MongoDB, Hive, Tableau and Java. And, given the specialist nature of these skills, Big Data professionals command some of the highest salaries in the tech industry – earning an average of £67,464 – this is more than the other four tech disciplines tracked in our Tech Cities Job Watch report – Cloud (£60,494), IT Security (£59,342), Mobile (£53,552) and Web Development (£43,198).
Stagnating salaries
When demand for certain skillsets grows, most people would expect the wages to be pushed up, as a result of increased competition. Whilst Big Data salaries remain the highest of the five core disciplines, remuneration has actually stalled in recent months. Permanent salaries increased by just 0.1% in the past year, and contractor day rates were down 5% over the same period.
This could be due to the fact that some Big Data projects have been put on ice in the run up to the GDPR launch, and those with data skills have been reassigned to ensuring that the necessary data processing changes are successful. Thus, those capable of working on higher value – and, by default, higher paying – Big Data projects are making do with ‘GDPR gigs’. Though one would have expected the demand to have peaked some time prior to the GDPR launch, organisations cannot risk being understaffed should their GDPR implementation not perform as intended.
It could also be because of a rebalancing in the market, and the fact that Big Data experts have seen the potential for high earnings and are upskilling themselves with the qualifications they need to secure those lucrative positions. In years gone by, Big Data skills were scarcer and employers had to put their hand in their pocket to secure the best. Now, candidate supply more closely meets demand, which could explain this pay stagnation.
Keeping top talent energised and engaged
Hiring Big Data specialists will enable organisations to compete in the digital era, and businesses should look for candidates who demonstrate an aptitude and enthusiasm to learn new skills, rather than someone who simply has the right credentials on their CV. Once these individuals are through the door, employers must also work to ensure they have processes in place to allow for career development and continuous opportunities to upskill. Team leaders should encourage staff to get involved with different projects and assignments around the business to widen their understanding and provide opportunities to develop their skillsets by completing the latest industry-certified courses.
In this data-driven world, organisations that can turn information into insight, better business decisions and personalised customer experiences are the ones that will truly thrive in the future of work. Big Data skills remain in high demand and businesses must keep a close eye on the market − to ensure they stay one step ahead of the competition and continue to position themselves as an employer of choice.
Martin Ewings, Director of Specialist Markets, Experis
Businesses across every industry are facing unprecedented disruption; with tech giants such as Netflix, Amazon and Uber transforming how services are delivered and uncovering new patterns in consumer habits in the process.
Business models are being rewritten, and efficiencies are being found through data-driven operations, as customers increasingly demand personalised experiences.
At the same time, the much talked about General Data Protection Regulation (GDPR) is shaking up how businesses collect, process, manage and secure their data. Designed to give EU citizens more control over their personal information and cut through the complexities that organisations face when complying with opposing local laws, the regulation is having a significant impact on businesses’ internal processes. Against this backdrop, it’s hardly surprising that Big Data professionals are highly sought after by employers. In fact, demand for Big Data expertise has skyrocketed by 78% in the last 12 months, with organisations under increasing pressure to make better use of their data to both maintain a competitive edge and remain compliant with the new regulation.
It’s the insight that can be gleaned from datasets that makes Big Data so powerful. To achieve this, employers must engage with the right professionals − those who can turn raw data into business gold. Here, I consider the reasons why employer demand for Big Data talent is on the rise, as well as the specific skillsets that are currently needed in enterprises.
Turning data into actionable insights
With the steady rise in adoption of sensor technologies, including the Internet of Things (IoT), we can expect datasets to grow and, similarly, the demand for Big Data specialists. More specifically, contractor demand has grown by 128% in the past year and permanent roles by 68% over the same period. This increase outstrips demand across other important skills markets, including Mobile (26%) and Cloud (30%).
Sought after skills
As business leaders continue to grapple with the complexity of running increasingly data-led organisations, the most in-demand skills for specific Big Data roles include AWS, Python, Hadoop, Spark, Cloudera, MongoDB, Hive, Tableau and Java. And, given the specialist nature of these skills, Big Data professionals command some of the highest salaries in the tech industry – earning an average of £67,464 – this is more than the other four tech disciplines tracked in our Tech Cities Job Watch report – Cloud (£60,494), IT Security (£59,342), Mobile (£53,552) and Web Development (£43,198).
Stagnating salaries
When demand for certain skillsets grows, most people would expect the wages to be pushed up, as a result of increased competition. Whilst Big Data salaries remain the highest of the five core disciplines, remuneration has actually stalled in recent months. Permanent salaries increased by just 0.1% in the past year, and contractor day rates were down 5% over the same period.
This could be due to the fact that some Big Data projects have been put on ice in the run up to the GDPR launch, and those with data skills have been reassigned to ensuring that the necessary data processing changes are successful. Thus, those capable of working on higher value – and, by default, higher paying – Big Data projects are making do with ‘GDPR gigs’. Though one would have expected the demand to have peaked some time prior to the GDPR launch, organisations cannot risk being understaffed should their GDPR implementation not perform as intended.
It could also be because of a rebalancing in the market, and the fact that Big Data experts have seen the potential for high earnings and are upskilling themselves with the qualifications they need to secure those lucrative positions. In years gone by, Big Data skills were scarcer and employers had to put their hand in their pocket to secure the best. Now, candidate supply more closely meets demand, which could explain this pay stagnation.
Keeping top talent energised and engaged
Hiring Big Data specialists will enable organisations to compete in the digital era, and businesses should look for candidates who demonstrate an aptitude and enthusiasm to learn new skills, rather than someone who simply has the right credentials on their CV. Once these individuals are through the door, employers must also work to ensure they have processes in place to allow for career development and continuous opportunities to upskill. Team leaders should encourage staff to get involved with different projects and assignments around the business to widen their understanding and provide opportunities to develop their skillsets by completing the latest industry-certified courses.
In this data-driven world, organisations that can turn information into insight, better business decisions and personalised customer experiences are the ones that will truly thrive in the future of work. Big Data skills remain in high demand and businesses must keep a close eye on the market − to ensure they stay one step ahead of the competition and continue to position themselves as an employer of choice.
Explore more articles in the Technology category
