Editorial & Advertiser Disclosure Global Banking And Finance Review is an independent publisher which offers News, information, Analysis, Opinion, Press Releases, Reviews, Research reports covering various economies, industries, products, services and companies. The content available on globalbankingandfinance.com is sourced by a mixture of different methods which is not limited to content produced and supplied by various staff writers, journalists, freelancers, individuals, organizations, companies, PR agencies Sponsored Posts etc. The information available on this website is purely for educational and informational purposes only. We cannot guarantee the accuracy or applicability of any of the information provided at globalbankingandfinance.com with respect to your individual or personal circumstances. Please seek professional advice from a qualified professional before making any financial decisions. Globalbankingandfinance.com also links to various third party websites and we cannot guarantee the accuracy or applicability of the information provided by third party websites. Links from various articles on our site to third party websites are a mixture of non-sponsored links and sponsored links. Only a very small fraction of the links which point to external websites are affiliate links. Some of the links which you may click on our website may link to various products and services from our partners who may compensate us if you buy a service or product or fill a form or install an app. This will not incur additional cost to you. A very few articles on our website are sponsored posts or paid advertorials. These are marked as sponsored posts at the bottom of each post. For avoidance of any doubts and to make it easier for you to differentiate sponsored or non-sponsored articles or links, you may consider all articles on our site or all links to external websites as sponsored . Please note that some of the services or products which we talk about carry a high level of risk and may not be suitable for everyone. These may be complex services or products and we request the readers to consider this purely from an educational standpoint. The information provided on this website is general in nature. Global Banking & Finance Review expressly disclaims any liability without any limitation which may arise directly or indirectly from the use of such information.

FIVE DATA CHALLENGES EVERY INDEX PROVIDER FACES

Partha Sen, CEO and co-founder, Fuzzy Logix
Partha Sen, CEO and co-founder, Fuzzy Logix

by Partha Sen, CEO and co-founder, Fuzzy Logix

As the popularity of index-based investing grows due to transparency and lower fees, pressure among index providers is increasing. And for good reason. Index-based funds that follow the S&P 500, Dow Jones and Russell 2000 for instance, tend to beat 80 percent of actively managed funds. This lower-fee style of investing makes a meaningful difference to an investor’s return. It should come as no surprise that last year, according to Morningstar passively managed funds attracted almost $505 billion, compared with outflows of $340 billion from active funds.

So what does this mean for index providers? The valuation and movement of an index can have an even greater impact on financial markets and investors. Index providers are under more pressure to meet the highest levels of data quality and precision in their calculations. But it turns out developing and managing an index is a complex and very time-consuming process that involves equal amounts of industry knowledge, mathematics and data analysis.

With all of the progress we’ve made in machine learning, algorithms and big data analytics, the industry has been surprisingly slow to adopt new technologies to streamline and accelerate the management of indexes and their time to market. Why? Historically it’s so operationally-intensive and the financial markets – and indexes – never take a vacation.

Processes across the industry are largely manual, and in some cases even date back 60-100 years. There are many different types of indexes, each with their own unique set of calculations. An index might be weighted, for example, based on a stock’s price or its market capitalization. Depending on the how an index is calculated, there are various factors to be considered such as the liquidity of constituent components, or the relative weightage of each component. Some key attributes and terms associated with an index include:

In order to achieve the high degree of accuracy required, it can take several days and countless employee hours to manage an index. Processes include Index Review, Index Rebalancing, Index Back-testing, Ad-hoc Enquiries and Factsheets, Industry-sector Classification, Calculation of the index itself, after having worked out the FAF and Cap Factor. On any given day, challenges include:

  1. Consolidating data from multiple internal and external sources (Reuters, Bloomberg, MarkIt, FactSet, Exchanges, etc.)
  2. Ensuring data quality and eliminating data duplication
  3. Analyzing large volumes of structured and non-structured data including securities trading data, market data, newswires, social media, annual reports, etc.
  4. Extensive reviews and re-balancing of indexes
  5. Back-testing and creating new indexes through a series of streamlined tasks such as universe selection, benchmarking against historical data and a maker-checker review process.

The key to managing a successful index today? The ability to sift through an enormous amount of data from multiple sources and to calculate it accurately and quickly. The good news is that progress is being made and some global market index providers are adopting new approaches.

Advanced data analytics and parallelized, machine learning algorithms that incorporate various index needs can automate processes and drastically alleviate bottlenecks to reduce employee hours and resources. This enables the index provider to automate many of its previously manually intensive functions – such as bringing rebalancing down to seconds instead of days – and eliminating any previous human error or data quality issues.

More importantly, by saving countless employee hours in operating a portfolio of indexes, modern day index management can help index providers shift resources to what will drive new revenue:  expansion and diversification of their portfolio by bringing new products to market quicker.