The Age of Uncomfortable Truths
6 unavoidable truths in the energy industry – and why they are the key to success
The energy industry is at a point of reckoning.
Geopolitics and policy-making are unpredictable. Market structures and financial innovation are creating new vulnerabilities. Decarbonization is necessary but tough. And the Law of Accelerating Returns is in full force – with compounding innovation accelerating the rate of change across industries and disciplines.
It’s time for the energy industry to face six fundamental truths – and in the process get comfortable with being uncomfortable as we work out the solutions.
Truth 1: Global leadership depends on data as much as power.
In the 21st century, global competitiveness is no longer determined solely by physical infrastructure or natural resource abundance. Leadership in the digital economy increasingly hinges on two interdependent forces: data and energy.
The explosive growth of AI, high-performance computing, and digital services is pushing electricity demand to unprecedented levels. It's not the cost of building datacenters that will define the next superpowers—but the cost and reliability of powering them.
As AI training and inference workloads scale, the marginal cost of data will converge to the marginal cost of energy. This will fundamentally reshape the economics of digital infrastructure. Nations and companies that possess a resilient, decarbonized, and affordable energy backbone will gain a decisioning advantage. Those who don’t, will fall behind in the AI race.
Yet much of the grid infrastructure in developed or developing national economies is ill-equipped for this future. Without targeted investment in grid modernization, flexible generation, demand-response technologies, and a clear view of the consequences of slow adoption thereof, we risk building digital castles on a foundation of unreliable energy.
The new geopolitics of AI will be defined as much by teraflops as by terawatts. To lead in data, you must lead in energy.
Truth 2: If your speculative investments seem too good to be true, they are.
Are financial and regulatory market structures misdirecting capital? Innovation has certainly created more dynamism in power markets, and that has created more opportunities for profitable investment. The industry needs that investment. The IEA has warned that last year’s spend of $390 billion in electricity grids is not enough to keep pace with the rate of electrification and renewable integration.
It’s important, however, that investment is going in the right places. The industry is already seeing price dislocation driven by correlations between local and global market trading.
The challenge here lies in the relationship between growing dynamism in energy markets and increasing innovation in financial markets. Dynamic markets draw both visionaries and opportunists — those driven by purpose, and those by profit.
After the drought of speculative trading in financial markets post Dodd Frank and oversupply, we are seeing a re-emergence of speculative trading as well as creative financing of new projects from new private financial institutions like hedge funds and private equity – unregulated institutions. The industry needs to attract visionaries without creating a paradise for opportunists.
Truth 3: The biggest risks are those you cannot see
The energy industry cannot afford to remain siloed. The roles of sub-sectors such as electric power and natural gas are increasingly entangled with the broader economy as the energy transition gains momentum. Siloed data and analytics lead to mounting invisible risks.
Data and analytics is needed to build bridges between asset classes, industries, supply chains and nations, to identify and manage risks and ensure the energy industry doesn’t repeat the mistakes of the finance industry 20 years ago.
But technology is falling woefully short. Many ETRMs are still living in the 1990s, and oriented towards day-ahead trading. Multiple E/CTRMs are common, because each is optimized for very specific asset classes.
The move to the cloud is happening – but it’s slow work. Some tech solutions are so old, it’s hard to source talent to operate them.
There’s an eerie parallel here with financial markets in 2008, when every fixed-income asset class was a vertically integrated, monolithic silo from front- to back office. Portfolio risk across distinct but interdependent desks was all but invisible. Market and counterparty risk was fragmented and disconnected. Many securities were opaque. Derivative values bore no relation to underlying assets.
Everyone knows what happened next. The energy industry cannot afford to make the same mistakes.
Truth 4: Too often, data rich = insight poor
Innovation in data and analytics offers some hope for offsetting the systemic vulnerability the energy industry saw in 2008.
Cloud hyper-scalers offer IT resilience, scalability and affordable access to computer power. Microservices architecture supports the creation of apps that operate at speed. Data is more affordable – and accessible. And AI offers the ability to derive insight from exactly those data sets.
But if data is getting easier to gather, it is harder to manage. Variety and velocity is growing at pace. Businesses who have spent millions of dollars on data acquisition are often data rich and insight poor due to insufficient data management and governance strategies.
Data granularity is up, as is the volume and accessibility and of unstructured data. At the same time, vendors and regulators are putting more rules around data rights, lineage and compliance in place.
The answer is to use proven, decision-ready data solutions and harness the promise of AI to leverage that data.
Truth 5: Not all data is created equal
The energy industry has historically been very good at storing data, and very good at keeping it from external eyes. Data security hampers analytical rigor when analysts take what data they can get rather than what is most accurate, complete, or current. High-quality data management can ensure that analysts securely and reliably find the high-quality data that they need.
In practice, analytics often consume stale or incomplete data to produce deterministic forecasts that are guaranteed to be wrong with no sense of margin of error. Business agility suffers when portfolio or risk analytics are based on last months’ views, or underrepresent the impact of uncertainty.
Transmission planning typically relies on a small number of snapshot models. Economic planning often uses software that was built on the assumption that intermittent resources and energy storage would have very limited penetration.
These models typically don’t account for major areas of concern today, such as sector-coupling, renewable curtailment, or negative pricing. They handle primary commodities, but struggle with GOs, resource adequate capacity, and carbon.
These are exactly the areas which need the greatest insight today.
Truth 6: Energy security, driven by the energy transition, is still a long way away
The energy industry is ringing a warning bell at the slower than desired pace of progress on decarbonization and energy transition.
This could be blamed on flawed incentive schemes that have directed capital towards renewables while neglecting grid upgrades; or slow adoption among consumers who remain highly sensitive to pump prices and energy bills. Or, like many organizations, it could be blamed on the inability to perform credible analytics at the pace of business.
Whatever the truth, energy is the key enabler of almost every other economic sector.
What’s next?
The energy industry must shrink the journey from question to decision, from days and months to minutes and hours.
Energy businesses that hamper their ability to make decisions because they are data rich and insight poor can transform themselves into nimble leaders by investing in data with a focus on management and governance, and in analytics with robust stochastic algorithms and orchestrated workflows.
Join us for a webinar on 18th September where Andrea Remyn Stone, CEO of Zema Global will be joined by Steven Broad, Head of Product & Solutions at cQuant, and John Kemp, Founder & Editor of J Kemp Energy, will discuss these challenges and how leaders in the energy industry can ensure they are able to manage them.
