LONDON (Sept 1): US crude prices jumped more than 27% in three trading days between Thursday and Monday, which should convince even the most die-hard believers the oil futures market is neither efficient nor rational.
The rally has left traders, analysts and journalists struggling to make sense of the sudden change in direction after prices had fallen steadily for two months, hitting their lowest level since 2009.
Possible explanations range from a bounce in global stock markets, pipeline problems in Nigeria, new data showing US oil production falling, and an editorial published in OPEC's monthly bulletin interpreted by some analysts as a call for a coordinated approach to cutting output ("Cooperation holds the key to oil's future", August 2015).
But it should be obvious none of these factors, singly or in combination, is sufficient to explain an increase in the price of US crude from $38 to $49 in the space of less than a week.
Far more important was the unusually large concentration of short derivatives positions in US crude still held by hedge funds betting prices would fall even further.
Large concentrations of hedge fund long or short positions have often preceded a sharp reversal in prices, as happened in March this year, when an unusual concentration of short positions preceded a sharp $18 rally.
Hedge funds and other money managers held short futures and options positions in the main US crude equivalent to 157 million barrels of oil at the end of trading on Tuesday, Aug 25, two days before the rally began.
Hedge fund short positions had barely been scaled back from a peak of 163 million barrels two weeks earlier and were almost three times greater than they had been two months previously, when they stood at just 56 million barrels.
The number of hedge funds with reportable short positions stood at 61 (down by just four from the peak) and the average position was 2.575 million barrels (down by just 287,000 barrels from the peak).
With so many hedge funds so heavily committed to a strategy which relied on a further decline in prices, any factor that caused the market to start rising, however insignificant, had the potential to start a race to cover short positions.
BEHAVIOURAL FACTORS
The oil futures market is a voting machine in which countless individuals register their views about the future of prices based partly on reason and partly on emotion. It is not a weighing machine in which supply-and-demand fundamentals are recorded by an exact and impersonal mechanism ("Security Analysis", Graham and Dodd, 1934).
New data about supply and demand is filtered through the minds and decisions of buyers and sellers ensuring psychological factors play a decisive role in price formation. Traders' expectations are based partly on their own evaluation of supply and demand, and partly influenced by talking to colleagues, customers and counterparties ("Why stock markets crash", Sornette, 2003).
The oil market is a community, subject to the same fads and fashions as any other community, rather than a mechanical system governed by the laws of physics. Fundamentals always reassert themselves eventually, but psychological factors are more important in the short term, and the short term can last for a surprisingly long time, stretching from days into weeks and months.
Between the middle of June and end of August, hedge funds applied an increasingly bearish filter to market news, emphasising bearish developments in supply and demand while minimising bullish ones. As more and more hedge funds established larger and larger short positions, prices fell, seeming to validate their bearish expectations, a classic example of what billionaire hedge fund manager George Soros calls reflexive trading ("The Alchemy of Finance", 1987).
But this strategy depended on the news flow getting progressively more bearish and all the hedge funds remaining short. As soon as prices stopped falling and started rising there was likely to be a rush for the exits as everyone tried to buy back short positions before everyone else ("Predatory trading and crowded exits", Clunie, 2010).
In some sense, the precise trigger for the short-covering rally is irrelevant. It was bound to happen at some point, whether sparked by reported revisions to US oil production, Nigerian pipelines or any other factors ("What might prompt a short-covering rally in US oil price?", Aug 17).
There is no need for a big cause to explain a large movement in prices such as happened over the last few days. In a complex system, a small trigger can result in an outsized movement in prices through positive feedback and a cascade effect.
TELLING STORIES
Many market fundamentalists find this hard to accept. They are constantly searching for a logical and tidy explanation for price movements.
The need to answer the question "why?" and as a prelude to asking "what happens next?" are among the oldest and most basic human instincts.
For thousands of years, humans have told stories to make sense of the world around them.
Five thousand years ago it was stories told around a fire about gods and heroes. Now it is more likely to be about asset prices and feature complicated mathematics.
In the most satisfying stories, causes and consequences are usually proportional and proceed logically according to a predestined plan. Randomness, chaos and accidents are unsettling and have no place in a good narrative.
But because the oil market, like other financial markets, is a community made up of human traders it doesn't have to behave rationally and logically. There are all sorts of price spikes and crashes which are only very loosely if it all connected with fundamentals.
This chaotic behaviour is what makes markets so interesting and unpredictable. The most successful traders embrace it, even if it horrifies the tidy minds of the fundamentalists.
John Kemp is a Reuters market analyst. The views expressed are his own.