This is the first part of the main article from New Scientist, written by Mark Buchanan, who is one of the most interesting scientists out there today, and a very accomplished science writer. He has a Ph.D. in physics, from Yale I believe.

Crazy Money: How Can We Stop The Financial Markets Creating So Much Misery? Forget Textbook Economics, The Answer Lies Elsewhere, Says Mark Buchanan

When you next sit down to watch the TV news, listen out for a telling phrase. At some point the newscaster will say something like: "The financial markets reacted to the report with a sharp fall..." Don't believe a word of it. The markets rarely react to news in this way.

Earlier this year, physicist Jean-Phillipe Bouchaud and colleagues at Capital Fund Management in Paris studied the news feeds produced by Dow Jones and Reuters that provide real-time reports of items of potential interest to investors. Looking at more than 90,000 news items relevant to hundreds of stocks over a two-year period, they studied how 'jumps' in stock prices---sudden, large movements---were linked to news items.

They weren't. Most such jumps weren't directly related to any news at all, and most news items didn't cause any jumps. "Jumps seem to occur for no identifiable reason," Bouchaud says (www.arxiv.org/abs/0803.1769).

This finding flies in the face of traditional economic theory, which insists that markets are mostly in equilibrium, reflecting an overall balance of economic forces. Markets change, the theory says, when those forces change: for example, when good news about a company increases demand for its stock, making its price go up. In this view, dramatic changes can only follow from correspondingly dramatic causes. Bouchaud's evidence says that, in fact, markets have unruly internal dynamics all their own, with rallies and crashes emerging seemingly from nowhere.

Evidence against the simple 'equilibrium' view of economics is piling up from other sources too. Take the recent worldwide credit crisis. Its main cause, the most sophisticated computer models now suggest, may be a fundamental tendency for markets to evolve, like an uncooled nuclear reactor, towards a dangerously unstable state. Everything from observations of irrationality in traders to the statistics of market fluctuations is telling us something is wrong with received wisdom, and a growing band of researchers has formed the view that we desperately need to develop a new theory of economics. "If we don't address the problems, there's absolutely no doubt that other extreme crises will occur in future," says Didier Sornette, an econophysicist at the Swiss Federal Institute of Technology in Zurich.

So what might this new economics look like? The standard theory of financial markets, shaped in large part by American economists Milton Friedman and Eugene Fama in the 1950s, is founded on the idea that the prices of stocks and other securities should tend toward their proper values. There are two reasons for this. First, investors have a strong incentive---the potential loss of their own money---to work out how much an investment is really worth. As rational people, they shouldn't be willing to pay too much for a stock, or to sell it for too little.

Second, the information gathered by millions of investors should in effect be pooled by the buying and selling in the market, making the market price an even better match to the true value of the stock than any individual can arrive at alone. [LeRoy: This is the supposed 'wisdom of crowds' effect...] Any temporary mis-pricing, the theory claims, should quickly get wiped out as some clever investor jumps on it with an eye to an easy profit. In this way, market forces should tend to iron out any problems long before they get unduly large. An unexpected rise or plunge in values just cannot happen unless there has been some correspondingly good or bad news.

This tells us straight away that something about the model is flawed. We are currently experiencing what may be the worst financial crisis since the 1930s. [LeRoy: Nouriel Roubini and I both agree on this.] Wall Street firms have already lost billions, and the U.S. government has had to save at least one [Bear Stearns] from outright collapse. Some analysts forecast that losses could ultimately exceed a trillion dollars. [LeRoy: Closer to two trillion, I would think.]

The crisis was triggered by the bursting of a bubble in the U.S. mortgage market that had grown to grotesque proportions, thanks to lax banking regulations and complex financial instruments that hid risks in what appeared to be safe packages.

On top of that, there was the issue of 'moral hazard'. As economist have been pointing out for some years, many common financial incentives induce people to act for their own short-term benefit, while saddling someone else---often their clients or the firm they work for---with longer-term risks. In the case of the sub-prime mortgage market, for instance, brokers were collecting commissions on mortgages that required no deposit and no proof of income. Since the brokers were not lending their own money, it was for them a risk-free business. Meanwhile, investment banks took on these risky loans and lumped them together into 'collateralized debt obligations' (CDOs). Once the risks were safely blurred, the banks were able to sell the CDOs on at a healthy profit.

Alarming as this sounds, it should be fine if you really believe in individuals' good sense and equilibrium economics. Investors will simply factor the risks of the sub-prime mortgages into the value of mortgage-backed securities, and adjust their expectations, putting realistic prices on everything.

Unfortunately, equilibrium thinking has hit the wall. In 2007 a global panic saw stock markets plunge. "A striking feature of the crisis is that the situation appeared to be driven by emotion," says physicist and former hedge-fund manager Doyne Farmer, now at the Santa Fe Institute in New Mexico. "The word 'fear', which is not an equilibrium concept, appeared in almost every newspaper article covering these events."

The crisis also illustrates another shortcoming of equilibrium thinking: a tendency to underestimate the likelihood of sudden large events. Compared with the normal distribution of random events represented by the bell curve, the statistics of financial fluctuations have fat tails. In other words, large price fluctuations are more likely than one might at first sight expect.

Failure to appreciate this has led to a number of big losses by 'quant' hedge funds, which use complex mathematical algorithms to analyze the markets. While analysts insisted this was just bad luck, they had in fact based their calculations on an incorrect understanding of the statistics of the markets, according to economist Brad DeLong of the University of California, Berkeley. "They said things like, 'Our strategy was fine, we were just hit by a 16-standard-deviation event,'" he says. This reflects erroneous equilibrium thinking that assumes the tail of the curve is slender. "Tails are fat," says DeLong.

Perhaps we should have seen this coming. Some economists have long argued that the movement of opinions and information between people tends to amplify market movements, leading inevitably to fat tails. Bouchaud and colleague Olivier Guedj found strong evidence for the idea four years ago. Using data on analysts' forecasts of U.S., European, U.K. and Japanese stock earnings over the period 1987 to 2004, they looked at how well their predictions had turned out. The data showed, for starters, that they were generally over-optimistic---so much so that a more successful strategy would have been simply to assume that the following year's earnings would be the same as the current year's. Tellingly, Bouchaud and Guedj found that the analysts tended to make forecasts that were similar to those other analysts had already announced, even when this went against available information (www.arxiv.org/abs/cond-mat/04­10079). They flock like sheep in Prada shoes.

LeRoy: I will post the last third or so of this article as a reply to this post. For the moment, let me announce that from now on I think I am going to describe myself as an econophysicist, to distinguish myself from all the 'orthodox' economists who simply don't (or even won't) bother themselves with stuff like this.

Virtual Markets

A new generation of financial market simulations is starting to take this flocking behavior into account. The idea is to include more detail on what makes people buy and sell, and how the opinions or actions of one investor can influence others. "Traditional economic models really don't even try to capture these dynamics," says Stefan Thurner, head of the Complex Systems Research Group at the Medical University of Vienna in Austria. [LeRoy: Where it would seem that their interests extend far beyond medicine...]

While this may seem like a considerable omission, it's not an easy one to put right. Market dynamics can be bewilderingly complicated, with thousands or even millions of participants---ranging from banks and investment funds down to individual punters---all interacting with one another. One of the most practical ways to get a handle on how these elements interact is to build computer models populated by artificially intelligent 'agents' that buy and sell among themselves, mimicking the activity of real markets. According to economist Blake LeBaron of Brandeis University in Waltham, Massachusetts, such models have already had some impressive success at reproducing stock histories. "These models seem to fit real markets---not only the fat tails, but trading volume and other measures too," LeBaron says. "Traditional models just don't go very far in reproducing any of this."

Meanwhile, Sornette has been investigating the effect of herding behavior on the rallies and crashes that seem to be inherent in financial markets. His models have shown that news from the real world does have an effect on markets---though not in the informed and rational way one might expect from the classical equilibrium view. "Paradoxically, it is investors trying to learn the relevance of new information---often by watching others---that amplifies price swings," Sornette says.

He developed the model with his student Georges Harras. Each day their agents decide what to do using three types of information: public news; what they hear from friends or others in their social network; and any private information they may have themselves. Over time, the agents gauge how effective each kind of information is in helping them make good decisions, and they adjust their behavior accordingly.

Sornette and Harras found that, as in any real market, prices in their artificial market never completely stabilize but continue to move up and down more or less chaotically. The researchers could, however, do something that is impossible in a real market: look at how individual players' decisions were linked to those ups and downs. This showed that the public and private information tends to keep prices around realistic values, as the classical equilibrium model says it should. The joker in the pack is information that flows through social networks, and gets spread by word of mouth. This, it turns out, creates groups of people coordinated in their actions, which in turn leads to bubbles---stocks that become priced too high or too low. Curiously, these bubbles can be triggered by nothing more than a random streak of news, which then becomes amplified by social feedback.

The model is also providing insights into the origin of market crashes, suggesting that here too received wisdom is wrong. Most financial analysts look for the origins of a crash in specific events immediately beforehand. Sornette and Harras's model, by contrast, indicates that it has more to do with a progressive linking together of investors' decisions and expectations over months or years. This reinforces any problems---which in turn leads to general market instability. Eventually, Sornette reckons, the markets reach a state like an avalanche waiting to happen. "Anything can trigger the avalanche once the system is ripe," he says. [LeRoy: I would say the 'system' is pretty 'ripe' these days...]

This kind of instability may have a lot to do with the events that triggered the current credit crisis. In recent unpublished work, Thurner, Farmer, and Yale University economist John Geanokoplos have developed an agent model of the securities market that includes hedge funds, banks, and ordinary investors. The model's hedge funds try to identify momentarily mis-priced securities, and make a profit by buying or selling in the expectation that the price will return to a realistic value in the future. As in the real world, they 'leverage' their investments by borrowing from the banks.

The simulations have revealed some alarming consequences of this kind of activity. [LeRoy: No! You don't say!] With no leverage, a hedge fund can only lose its own investors' money, but as leverage increases it can also lose money it has borrowed from a bank, possibly putting that bank into difficulties. "Lots of leverage begins to pose the threat of failures cascading through the market," says Thurner.

Intriguingly, the risk of cascades like this occurring doesn't increase gradually. Things go smoothly until the amount of leverage reaches a certain threshold, at which point the model shows the market undergoing a sudden change, loosely akin to a physical phase transition, like water freezing into ice. [LeRoy: There is a good reason why some observers describe the credit markets as 'freezing up'!] Increasing levels of credit create stronger links between market players, heightening the chance that the failure of one can put an unsustainable burden on others, triggering further failures. In the simulations, once the level of leverage passes a certain threshold, it becomes overwhelmingly likely that a single chance failure will send waves of trouble through the entire market. Avoiding future crises will mean identifying where the real-world market's 'freezing point' is---and keeping levels of leverage low enough to steer clear of it.

Geanokoplos cautions that this work remains speculative, but the idea of increasing leverage bringing disaster corresponds well with history, says Sornette, who has studied the dynamics of a number of market crashes. [LeRoy: Indeed, this can hardly be claimed to be a new idea, as it has been expounded on by authors such as Robert Shiller in his book Irrational Exuberance and Charles Kindleburger in his book on Financial Manias, Panics, and Crashes. And others as well.] "All bubbles I have studies have been associated with increasing access to easy money, whether it's lower margin requirements, lower interest rates, more foreign investments, and so on," [Sornette] says. Whether this idea can be put to work depends not only on identifying the threshold for trouble, but also on the regulatory authorities' willingness to try new approaches. They are certainly needed, Sornette reckons. "The natural reaction to a crisis is to update and upscale regulation and supervision," he says, "but this has repeatedly failed to ensure even medium-term stability in the past." Now could be the time for a move away from equilibrium thinking.

LeRoy: The article has an interesting graph which represents the outcome of one of these agent-based market simulations. The captions says: "Boom and Bust: Markets Become Unstable When Traders Follow The Herd." The smaller note with the graph says: "In a mathematical model of market behavior, a newsfeed of equally positive and negative stories has little or no effect on the price of a stock being bought and sold by traders acting independently of each other. Small spikes appear when the traders start to copy each other, and at a certain threshold level of copying these turn into massive bubbles, each followed by a catastrophic crash.

Traditional economists have, as a rule, made little or no attempt to model the effects of information flow through these social networks, a failing which has rendered our old models of financial markets completely unreliable. As Buchanan argues, time for a change. This article could be viewed as a useful companion piece to the article from an earlier issue of New Scientist I posted a couple of months ago (at the beginning of summer), titled "Managing the Collapse of Civilization". That article argued that our civilization is much more fragile than is commonly understood, and prone to a sudden collapse. Although this article deals much more specifically with the state of our financial markets, the underlying message is the same.

Nothing about the article really surprises me. I would have thought that the several mentions of "fat tails" would have led to the introduction of such terms as "skew" and "kurtosis". I think I see my message about isolating this market so that it can't affect the "real economy" appearing here. It should be remarked that a poker game with considerable volatility and excitement can stil be played without damaging either the card or the chips. It is when a losing player successively bets his watch, his truck, his farm, and his wife that this overflow is noted.

Many ideas in these two parts of your citations...most of which revolve around herd behavior. (See my reference to Sentiment Analysis in your first posting). But I'd like to take issue with the following excerpt:

The crisis was triggered by the bursting of a bubble in the U.S. mortgage market that had grown to grotesque proportions, thanks to lax banking regulations and complex financial instruments that hid risks in what appeared to be safe packages.

Besides lax banking regulations and complex financial instruments (including derivatives), it should be emphasized that it was the public policy to encourage the Man in the Street to purchase homes, to deduct interest and other costs from his taxes, and to exclude any future capital gains from taxation. The housing bubble really got started after Clinton signed the change to the capital gains exclusion in 1997--raising the exclusion from $125,000 once a lifetime to $500,000 once every five years, and after living in a principal dwelling for only two years. Here we see that overgenerous tax loopholes impelled millions into housing--which became a virtuous circle with the deregulation of banks and the slicing and dicing of mortgages into CDO sausages for consumption by third parties starved for yield in Greenspan's artificially low interest rate environment. I'd say that the tax policy was the primary motivator, and the subsequent change in lending terms a secondary accommodator.

The sad fact is that the people (and the banks) wanted the goddam bubble to get rich quick, and the shortsighted politicians accommodated them. As Carlin said, "It's not the politicians, it's the people who are screwed up". Specifically, I'd say it's mainly the pandering Republicans who are the problem today--unlike during the 1960s, when it was the pandering Democrats who first got the federal budget off the rails.

1920s: pandering GOP

1960s: pandering Democrats

2000s: pandering GOP

Notice a pattern here?

When Republicans pander, watch out, as they are supposed to be the "fiscal conservatives". When nobody's minding the store, depressions happen.

Economic academics tend to ignore or deny the obvious. They get rewarded somehow for obscurity.

Current conditions have obvious causes.

Houses were costing 1-2% teaser interest for 2 years while prices were rising 20%-40% a year. The price rises were fed in part by the teaser rates and tax advantages.

Energy demand rose in Chindia.

Speculation in oil was allowed by deregulation under Phil Gramm and Wendy.

USA govt went deficit, including use of SS Trust Fund.

Fed printed money feverishly.

The Bush-Cheney mission has been to raise oil prices.

USA let in all of China's output without tariffs and without regard to China's low labor costs, thievery, and toxics.

All: economic suicide.

No effective opposition from Democrats, who are still dominated by Carterites and social-liberals (no economic favor to poor)

Both parties subservient to TLC.

Collapse of Big Labor.

You guys are looking for something, oblivious to huge realities. Tell me: when you find it, what then?

Pay attention!

Phil: It is when a losing player successively bets his watch, his truck, his farm, and his wife that this overflow is noted.

LeRoy: And, possibly, even before the truck?

Not if she's standing behind him while he plays!

At least, not in my neck of the woods.