Navigating the New Energy Landscape: The Age of Uncomfortable Truths

First published on: 5/3/2026

Summary

The global energy industry is undergoing a profound and messy transformation. The simple, linear narratives of the past - where one fuel source neatly replaces another - are giving way to a more complex reality of “energy additions,” driven by soaring global demand and the insatiable energy needs of data centers and artificial intelligence. This new era is defined by interconnected markets, unprecedented volatility, and a deluge of data that promises insight but often delivers complexity.

This white paper confronts six uncomfortable truths facing the energy and data industries. We explore how the link between data and power is reshaping global leadership, why speculation poses a new threat, and how the industry can avoid past mistakes. Ultimately, navigating this new landscape requires a fundamental shift away from merely collecting data to building robust “decisioning infrastructure” that can turn a flood of information into a tangible competitive decisioning advantage.

Introduction: The Age of Uncomfortable Truths

The energy industry is experiencing unprecedented disruption. Geopolitics and policymaking are unpredictable, while rapid innovation creates new vulnerabilities. Decarbonization is necessary but challenging, and the Law of Accelerating Returns is in full force. All the while, the rate of change itself accelerates as new ideas and technologies compound on each other. Meanwhile, AI advancements continue to shift the dynamics of everything we do.

For centuries, the story of energy was told as a series of well-defined transitions: the 18th century was dominated by wood, which gave way to coal, which, in the 20th century, gave way to oil. However, modern reality has proven to be a story of addition, not substitution. Global energy consumption has quadrupled since 1960, and this rapid growth has meant that every energy source - even legacy fuels like coal - has seen consumption increase.

The energy industry, along with its leaders, must collectively get comfortable with being uncomfortable. By identifying and confronting these six modern-day, uncomfortable truths, there’s real possibility for significant change.

Truth One: Global leadership depends on data as much as power

A nation’s strength has always been dependent on the health of its energy economy and natural resources. With its energy reserves and track record of technology innovation, the US has held a leadership position in the global economy since the post-WW2 era. However, energy markets are volatile, and external factors are adding more layers of uncertainty, making the race for global energy leadership anyone’s game. AI is the game changer. It will transform individual industries and entire economies. The new gold rush is for power - access to energy will be the determining factor in winning the AI race. It won’t be the cost of building data centers, but rather the cost of running them that will be the real challenge.

Training AI models can cause huge, short-term spikes in energy use, something which existing energy grids find hard to manage. As Andrea Remyn Stone, CEO of Zema Global, warns, “The pace of innovation in technology is moving much faster than the pace of construction and innovation in the energy infrastructure.”

The correlation between data and power could be so strong that the marginal cost of data may eventually become indistinguishable from the cost of energy. In this new world, past success is no indicator of future leadership. Grid infrastructure will need investment and innovation to keep up with AI, with its almost infinite thirst for energy, for players to stay on track to maintain global leadership positions.

Furthermore, the industry faces a potential Jevons Paradox, which states that when something becomes more efficient, people use more of it, not less. As Stephen Broad, Head of Products and Solutions at Zema Global, puts it, “More performance and more efficiency is simply going to lead to more consumption.”

Industry stakeholders are starting to recognise the interconnection between energy and economy. When a focus group of Zema Global customers and prospects were asked what they thought the key driver of global economic leadership will be in the next decade, the majority said that it will be energy access and innovation, as well as AI and computing power.

Truth Two: If your speculative investments seem too good to be true, they are

A core question remains whether current market structures and regulations are causing capital to be invested in ways that best serve the energy transition. Deregulation has brought the emergence of new energy markets, such as intraday and capacity markets, where power suppliers are paid for their commitment to producing energy in case demand surges, as opposed to being paid for the electricity they actually produce. These markets are increasingly dynamic and help to create profitable opportunities, but the question is whether these short-term profits align with the long-term need for a sustainable energy system.

A staggering amount of capital is required to modernize the grid. As established, as energy markets become more efficient, we can.

Truth Three: The biggest risks are those you cannot see

The diverse blend of traditional and renewable energy resources, their locales, and their economic profiles - along with financial and technological innovation - are exposing new areas of risk in terms of reliability, resource adequacy, and price. The increased demand for power is a macro tailwind we expect for at least the next 25 years, driven by the electrification of everything in a quest for clean energy. Indeed, the electric power sector is the workhorse of global decarbonization, with significant medium to long term impacts on molecules, technology, transportation, mining, and numerous other commodities.

Technology needs to evolve to support more dynamic, multicommodity trading as well as demand-side energy management. This evolution is especially urgent as vulnerabilities in the supply chain are also revealing new risks amidst murky geopolitical dynamics.

The current geopolitical climate draws an eerie parallel with financial markets in the lead-up to the 2008 crisis due to complexity, lack of transparency, and outdated systems failing under new market pressures. Back then, it was difficult to see portfolio risk across desks trading distinct products with interdependencies; market risk and counterparty risk were fragmented and hard to measure. The energy industry cannot afford to make the same mistakes in the modern day.

New risks are coming from factors we cannot see or understand because we are living in a siloed world. We need data and analytics to bridge these silos across asset classes, industries, supply chains, and geopolitical boundaries.

When asked what system in their company they would modernize first, industry stakeholders were almost evenly split across:

• C/ETRM (Commodity/Energy Trading Risk Management)
• Data governance
• Grid analytics and forecasting
• Risk analysis
• Trading systems

This indicates companies are divided on where the biggest modernization gaps lie, reinforcing the point that the industry is still siloed. To prevent history from repeating itself, the industry must invest in integrated systems that provide real-time visibility across assets, geographies and markets.

Truth Four: Too often, data rich = insight poor

Innovation in data and analytics offers some hope for offsetting the systemic vulnerability seen in 2008. Cloud hyper-scalers provide flexible and affordable access to computing power offering IT resilience, while microservices architecture enables the creation of applications that operate at speed. Data is more affordable and accessible due to innovation in data storage, like data lakes, and AI offers the ability to derive insight from those huge data sets.

But data is flowing in as fast as analytics are racing to unlock insight. Fundamentally, data is easy to gather but difficult to manage. There is an increased variety of data thanks to new asset classes, sensor data, and more local sources appearing as markets deregulate. The speed at which data is created, collected and processed is fluctuating more than ever. Data granularity is also increasing, as is the volume and accessibility of unstructured data. All the while, vendors and regulators are putting in place more rules around data rights, data lineage, and data compliance, adding yet another layer of complexity.

This self-perpetuating dilemma coils back upon itself. Many companies which have collected masses of data then try to self-manage and self-govern these data sets. In turn, they become data rich and insight poor. Ungoverned data lakes often lead to a ‘mushrooming effect’ whereby business users, lacking context or understanding of data lineage, pull the wrong information for their analyses, resulting in a rapid, mushroom-esque proliferation of misunderstandings.

This is where decisioning infrastructure – which organises and prepares data specifically for analytics and AI – becomes invaluable and gives a decisioning advantage to those in the industry operating in this volatile space.

Evidence of the industry’s challenges with data comes from Zema Global customers and prospects, half of whom say that regulatory complexity is the number one blocker stopping their team from making confident energy decisions. Other reasons cited include inferior analytics and poor-quality data. This highlights the urgent need not just for better tools, but for smarter infrastructure that can operate within - and help to navigate - the increasingly dense data landscape.

Truth Five: Not all data is created equal

The energy industry has historically been very good at storing data and keeping it away from external eyes. Energy operators hold vast swathes of internal data yet struggle to find the tools and the talent to unlock insight.

For example, the extreme granularity of Internet of Things (IOT) data - interspersed with pricing or index data from smart meters and sensors - costs a lot in storage, while increasing the difficulty to find insight due to sheer volume. The energy industry needs truly excellent analytics across petabytes (1,000,000 gigabytes) of data, working in a cost-effective manner.

Another example is the transmission data from the Electric Reliability Council of Texas (ERCOT), where several different representations of the grid exist, none of which agree on which lines are in service. This lack of a single, reliable source of truth makes it extremely difficult to conduct accurate studies of the grid. Therefore, analysis has often been based on easily accessible data, rather than the most accurate or insightful.

Thus, energy analytics have not kept pace with technology innovation and the data explosion. Risk engines are outdated and based on incomplete data, and most long-term resource planning analysis is done deterministically with software that underrepresents the impact of uncertainty. Transmission planning typically relies on a small number of snapshot models, and economic planning often uses software that was built on the assumption that intermittent resources and energy storage would have very limited penetration.

Insight from industry leaders reinforces the argument that most decision-makers lack full confidence in the data on which they rely. Less than twenty percent feel completely confident that they always have access to the right energy and commodities data for each and every decision they make. 

Truth Six: Energy security, driven by the energy transition, is still a long way away

The industry is entering a phase of sobering realism that the energy transition isn’t just about switching to renewables. It will also need to account for complex factors like geopolitics, finance and policy. Meanwhile, alarm bells are sounding due to the slower-than-desired pace of progress on decarbonization goals. A significant amount of capital has been invested into the energy transition - $2.1 trillion last year alone - but this doesn’t address the amount is estimated to be required to upgrade the grid itself.

Incentive systems such as Renewable Energy Credits (RECs) have helped to fund renewable projects, but the credits themselves have received scrutiny over poor governance. Doubt over their legitimacy stunts demand in financial markets to fund projects, which in turn slows down the pace of decarbonization. The industry is also seeing gaps in the value of projects expressed to banks at financing versus real-world operation.

As well as this, the knowledge base of the energy industry has been skewed by both policy and an innovation premium. Government policies heavily favour renewable energies to meet decarbonization goals, shifting training and investment accordingly. This has led to not just a greater understanding of renewables, but also a simultaneous loss of knowledge regarding the maintenance of traditional fossil fuels and gases, which continue to make up a significant portion of our energy.

Traders likewise have little perspective due to a critical skill gap and lack of data-fluent natives in the workforce. Firms require a knowledge base that both supports a diverse energy economy and understands the modern technologies needed to power dynamic decisions.

While there are a variety of reasons for slower than-desired progress, many businesses cite an inability to make sense of their data at a speed that matches the pace of their operations. As well as this, data infrastructures are decoupled from analytics infrastructures, moving asynchronously, with inaccurate and incomplete data underpinning decision-making. Essentially, the industry is struggling to make the right decisions at the right time.

Despite this, it’s a fundamental fact that energy security is a key enabler of most other economic sectors and will dictate growth and policy for the foreseeable future. Notably, tech leaders from many of the world’s leading AI companies have explained a dire need for electric power to keep pace with AI technologies, highlighting the urgency of solving the energy sector’s underlying problems.

Over half of Zema Global customers find transmission infrastructure to be their biggest challenge in achieving energy security. This is in line with the IEA’s concerns that there is not enough investment into the grid. But this is only part of the problem. Others cited regulatory delays, government inaction, and market pricing design as core issues.

To read the key takeaways and conclusion on the uncomfortable truths facing the energy industry, download the full whitepaper below.