Market Data in Flux: Real Time, AI, and Hybrid Infrastructures as Determining Factors

Do you know how uncoordinated and dangerous Formula One races were in the 1950s? In those days, a pit stop was a tedious process that took a good minute to complete, and the pit crew never even knew exactly when the racecar would pull into the pit lane. The data on that were sparse, and decisions were based on experience and gut instincts.

Today, a pit stop is a highly orchestrated process. Sensors deliver live data, artificial intelligence (AI) predicts the ideal timing, and man and machine work closely together because every millisecond counts.

The financial industry, too, has evolved from an “analog workshop” into a digital high-performance center where decisions are increasingly made on the basis of real-time information with the aid of AI and flexible hybrid architectures. And new ways of utilizing existing data are constantly emerging.

This evolution continues to progress. In today’s volatile financial markets, the buy-side has to react to global events in real time. Speed has long since become a basic requirement. In place of end-of-day reports, market participants are increasingly relying on real-time data to continually monitor risk, compliance, and portfolios to thus stay competitive.

The market data study by Greenwich Coalition and SIX found that 65% of the buy-side firms surveyed now use real-time data throughout the trading day, marking a substantial increase from the previous year (54%). It discovered that use of real-time data is moving far beyond trade execution and is getting deeply embedded into processes conducted by risk management and compliance teams and into portfolio analytics.

As real-time data gain importance, requirements placed on data sources are also mounting. The buy-side is increasingly using historical tick data for market and trade surveillance. 85% of the survey respondents resort to it, according to the study. That’s 26 percentage points more than in the previous year. Moreover, they rely on multiple sources for dependable high-frequency data.

The volume of processed tick data therefore looks set to increase because quality is and will continue to be the crucial criterion: clean, accurate data is needed in times of mounting regulatory requirements and specialized data needs, and market participants are willing to invest more to obtain it.

If requirements and needs are on the rise, it’s not surprising then that the survey respondents foresee growing budgets for market data. That expectation has risen 12 percentage points compared to 2024, and almost 70% of the respondents anticipate a 1% to 5% increase in spending on market data.

A disproportionately large slice of budgets is reportedly earmarked for spending on index, risk, regulatory, and crypto data. In both front and middle offices, those data categories are especially pertinent to compliance, innovation, or new market requirements, for instance.

Risk management and compliance teams, for example, are increasingly resorting to using tick data to monitor for market abuse, for instance. At the same time, crypto data requirements are increasing substantially because more and more traditional institutions are trading crypto assets and are developing investment products based on them.

However, it’s important for data providers to continue to constantly supply all kinds of data because although demand is increasing across all data types from year to year, priorities shift according to prevailing market conditions, volatility, the regulatory environment, and evolving risk management and compliance requirements.

Market data purchasing and licensing decisions are increasingly being made in the business units that actually utilize the data. This may lead to more flexible budgets in some circumstances and underscores that the market is still open to stepping up spending.

The way in which market data get distributed and utilized also continues to evolve rapidly. Technologies like AI and machine learning (ML) in particular, but also cloud infrastructure and application programing interfaces (APIs), are shaping this change. Approximately 80% of the survey respondents now see AI and ML as the biggest drivers over the next two to three years.

This jibes with a recent report from Swiss Banking that describes what potential, risks, and implementation strategies exist for generative AI (GenAI) in the banking sector. For the buy side, GenAI unlocks potential that extends beyond trading signals or algorithmic trading. Wealth managers and private banks see in GenAI an opportunity to automate their internal business processes, to enhance customer experiences with personalized insights, and to optimize data-based decisions by leveraging predictive analytics.

A recent report from Forrester also describes how financial institutions are employing AI agents that even go a step further and completely automate processes.

Just as there’s a consensus that AI is a driving force in the market for financial data, there is also increasingly general agreement on the role that AI is to play. Whether it be customer service, investment ideas, or trade execution and settlement: AI suggests, humans decide.

The findings of the study imply that AI is being used more intensively and confidently, which is an important step for survival in a market data ecosystem that is becoming more and more complex.

The findings also reveal an increase in trust in automated analytics: concerns about erroneous data are subsiding. Users are more concerned with finding ways of meeting market requirements and regulations that at the same time enable them to work efficiently. And in order to allow AI to flex its strengths, a qualitatively sound data base and good data governance are needed.

Data delivery channels also continue to evolve. 63% of the survey respondents source market data today via the public cloud, more than twice as many as in 2023. Earlier security concerns about the public cloud appear to be receding. It also appears that use of APIs is on the way to becoming the standard. APIs guarantee efficiency while maintaining a consistent quality and user experience and thus offer a good return on investment for market participants.

Direct point-to-point connectivity nevertheless remains relevant in the market data ecosystem, particularly wherever security and control are required. Moreover, direct access to primary sources is indispensable for specialized market participants like hedge funds, for example.

The future is clearly hybrid: market participants source data through various technical avenues such as APIs, the cloud, and dedicated point-to-point connections. At the same time, they resort to using a variety of data suppliers, as the study shows. Most buy-side firms, for instance, use redistributors, on-demand platforms, and direct-from-the-source offerings concurrently. They frequently source specialized data from four or five different providers.

The choice of data provider depends greatly on the use case. Redistributors provide multiple datasets through the same pipe, which simplifies technical integration. However, challenges exist here with regard to data quality, level of service, costs, and transparency. For example, throughput pricing is often opaque and access to the original source is limited.

Direct-from-the-source solutions, in contrast, offer maximum control over data quality, latency, and up-to-datedness. Whoever sources data directly from a supplier like SIX, for example, has full access to the source, but has to arrange integration, licensing, and contract management on its own. This direct access is often indispensable for hedge funds and other specialized market participants due to regulatory requirements or internal compliance rules, for instance. For smaller firms with limited resources, on the other hand, an SaaS approach – i.e. data plus technology – may be the most feasible solution.

There is no one-size-fits-all solution, which is why many market participants deliberately rely on a combination of provider models.