How Well Can the "Experts" Predict Earthquakes?

It has been a curious experience for me to be immersed in research on earthquake prediction while the country is going through a financial crisis. I was intrigued by the following quote from USA TODAY (January 2, 2009), as well as similar quotes that I see nearly every day in the blogosphere:

"Market gurus predict stock rebound but won’t rule out extreme move up – or down."

Now, that's a pretty safe prediction! It's like taking a multiple choice test with the options increase, decrease, or remains the same, and choosing “all of the above.” I think that pretty much captures the state of the art regarding earthquake prediction. Seismologists can predict earthquakes about as well as those market “gurus” can predict the stock market. That is, not well at all.

The perennial promise of successful earthquake prediction captures the imagination of a public hungry for certainty in an uncertain world. Yet, the sober reality is that there is no reliable, scientifically proven method of predicting where and when an earthquake of a given magnitude will occur.

It is understandable that people are forever hopeful that there would be “experts” somewhere who can provide us with some methods for predicting earthquakes and thus avoid human tragedy. Unfortunately, as in the case of financial markets, it's not clear that "experts" are able to predict earthquakes any better than intelligent “lay” people who are given access to seismological data.

While procrastinating on my research/writing about earthquake prediction, I was browsing in my local library and found a fascinating book written by Nassim Taleb, entitled The Black Swan: The Impact of the Highly Improbable. Taleb analyzes uncertainty and problems with prediction from the perspective of his career in various aspects of financial markets.

The notion of a “black swan” is described in Wikipedia as:

“a large-impact, hard-to-predict, and rare event beyond the realm of normal expectations ... The term black swan comes from the assumption that 'All swans are white'. In that context, a black swan was a symbol for something that could not exist. The 18th Century discovery of black swans in Western Australia metamorphosed the term to connote that the perceived impossibility actually came to pass.”

Here is Taleb’s working definition:

“What we call here a Black Swan (and capitalize it) is an event with the following three attributes: First, it is an outlier, as it lies outside the realm of regular expectations, because nothing in the past can convincingly point to its possibility. Second, it carries an extreme impact. Third, in spite of its outlier status, human nature makes us concoct explanations for its occurrence after the fact, making it appear to be explainable and predictable.”

Taleb argues that, because of the possibility of black swans, many (most?) types of phenomena are essentially impossible to predict. He further argues that many so-called experts in a wide variety of fields have developed very sophisticated mathematical models of all sorts of phenomena—but because black swans are hard to incorporate into the models, the models typically don’t account for these high-impact events that actually do occur. Thus, he concludes that experts don't predict any better than “non-experts” are able to predict. In fact, if you compare what actually occurs to what the experts predicted to occur, they typically do at least a little bit worse than non-experts!

According to Taleb:

“... [C]ertain professionals, while believing they are experts, are in fact not. Based on their empirical record, they do not know more about their subject matter than the general population, but they are much better at narrating – or, worse, at smoking with complicated mathematical models.”

Some would say that Taleb takes too extreme a position on these matters. Read the book and decide for yourself; but when it comes to earthquake prediction, I do think that there is more than a grain of truth to his argument.

In 2004, a feature article posted on NASA’s website heralded an earthquake prediction project they funded as an “amazing success” - Earthquake Forecast Program Has Amazing Success Rate.

Being inherently skeptical about these matters, a colleague and I decided to test just how amazing this particular success was (Kafka and Ebel, 2007). We tested the NASA-funded prediction model, known as Pattern Informatics (PI, Rundle et al., 2002*), by comparing it to a much simpler method that I developed, called Cellular Seismology (CS, Kafka, 2002, 2007). At its core, CS assumes nothing more than that future earthquakes will occur near past earthquakes. Thus, testing PI against the CS model is a good way to compare how well an “expert” model performs when compared to what I would argue is essentially a "non-expert" model. I see CS as a non-expert model because I imagine that many non-experts would likely come up with some variation on CS as a “common sense” component of what they think should be included in an earthquake prediction scheme.

Our results: PI doesn't perform any better than CS, and we have yet to find anything in the record of past seismicity that is any more predictive of where future earthquakes are likely to occur than the trivial, intuitive notion that future earthquakes tend to occur near past earthquakes.

We continue to compare other methods to CS, and it will take many more years of observing earthquakes before we will have enough evidence to discern the extent to which other methods might actually outperform CS. But to date, we have yet to find any method that does any better than CS (see for example Kafka and Ebel, 2009). Although we have certainly not conducted an exhaustive analysis of the questions addressed here, the results of our studies do suggest the possibility that Cellular Seismology, simple as it is, may actually be a measure of “all we can know” about the future occurrence of earthquakes. Hard as it may be for the public (and for seismologists themselves) to accept this conclusion, the results of my research suggest that such a possibility needs to be considered.

As Yogi Berra said, “It's tough to make predictions, especially about the future.”


* Rundle, J.B., K.F. Tiampo, W. Klein, and J.S.S. Martins (2002). Self-organization in leaky threshold systems: The influence of near-mean field dynamics and its implications for earthquakes, neurobiology, and forecasting, Proc. Natl. Acad. Sci. U.S.A. 99, 2514-2521, Suppl. 1.