The Improbability Principle

In The Improbability Principle, the renowned statistician David J. Hand argues that extraordinarily rare events are anything but. In fact, they're commonplace. Not only that, we should all expect to experience a miracle roughly once every month. But Hand is no believer in superstitions, prophecies, or the paranormal. His definition of "miracle" is thoroughly rational. No mystical or supernatural explanation is necessary to understand why someone is lucky enough to win the lottery twice, or is destined to be hit by lightning three times and still survive. All we need, Hand argues, is a firm grounding in a powerful set of laws: the laws of inevitability, of truly large numbers, of selection, of the probability lever, and of near enough. Together, these constitute Hand's groundbreaking Improbability Principle. And together, they explain why we should not be so surprised to bump into a friend in a foreign country, or to come across the same unfamiliar word four times in one day. Hand wrestles with seemingly less explicable questions as well: what the Bible and Shakespeare have in common, why financial crashes are par for the course, and why lightning does strike the same place (and the same person) twice. Along the way, he teaches us how to use the Improbability Principle in our own lives—including how to cash in at a casino and how to recognize when a medicine is truly effective. An irresistible adventure into the laws behind "chance" moments and a trusty guide for understanding the world and universe we live in, The Improbability Principle will transform how you think about serendipity and luck, whether it's in the world of business and finance or you're merely sitting in your backyard, tossing a ball into the air and wondering where it will land.

A well-known statistician presents his theory that extraordinary and rare events are actually commonplace and cites stories of two-time lottery winners and other bizarre coincidences to support his theory that unlikely events statistically must happen. 50,000 first printing.

Why is it that incredibly unlikely phenomena actually happen quite regularly and why should we, in fact, expect such things to happen? Here, in this highly original book - aimed squarely at anyone with an interest in coincidences, probability or gambling - eminent statistician David Hand answers this question by weaving together various strands of probability into a unified explanation, which he calls the improbability principle. This is a book that will appeal not only to those who love stories about startling coincidences and extraordinarily rare events, but also to those who are interested in how a single bold idea links areas as diverse as gambling, the weather, airline disasters and creative writing as well as the origin of life and even the universe. The Improbability Principle will change your perspective on how the world works – and tell you what the Bible code and Shakespeare have in common, how to win the lottery, why Apple's song shuffling was made less random to seem more random. Oh and why lightning does in fact strike twice...

"Data describe and represent the world. However, no matter how big they may be, data sets don't - indeed cannot - capture everything. Data are measurements - and, as such, they represent only what has been measured. They don't necessarily capture all the information that is relevant to the questions we may want to ask. If we do not take into account what may be missing/unknown in the data we have, we may find ourselves unwittingly asking questions that our data cannot actually address, come to mistaken conclusions, and make disastrous decisions. In this book, David Hand looks at the ubiquitous phenomenon of "missing data." He calls this "dark data" (making a comparison to "dark matter" - i.e., matter in the universe that we know is there, but which is invisible to direct measurement). He reveals how we can detect when data is missing, the types of settings in which missing data are likely to be found, and what to do about it. It can arise for many reasons, which themselves may not be obvious - for example, asymmetric information in wars; time delays in financial trading; dropouts in clinical trials; deliberate selection to enhance apparent performance in hospitals, policing, and schools; etc. What becomes clear is that measuring and collecting more and more data (big data) will not necessarily lead us to better understanding or to better decisions. We need to be vigilant to what is missing or unknown in our data, so that we can try to control for it. How do we do that? We can be alert to the causes of dark data, design better data-collection strategies that sidestep some of these causes - and, we can ask better questions of our data, which will lead us to deeper insights and better decisions"--

In this book, the authors bring together for the first time an extensive collection of papers by many distinguished scholars who present a variety of arguments that the existence of God is improbable.

This exploration of the meaning and power of data traces the history of data collection and interpretation and examines our understanding of the world, covering mathematics, science, statistics and metaphysics. It looks at the systems of data collection and transfer that we rely on and explores the danger of false views of data.

From Aristotle's Physics to quantum teleportation, learn about the scientific pursuit of instantaneous connections in this insightful examination of our world. For millennia, scientists have puzzled over a simple question: Does the universe have a speed limit? If not, some effects could happen at the same instant as the actions that caused them -- and some effects, ludicrously, might even happen before their causes. By one hundred years ago, it seemed clear that the speed of light was the fastest possible speed. Causality was safe. And then quantum mechanics happened, introducing spooky connections that seemed to circumvent the law of cause and effect. Inspired by the new physics, psychologist Carl Jung and physicist Wolfgang Pauli explored a concept called synchronicity, a weird phenomenon they thought could link events without causes. Synchronicity tells that sprawling tale of insight and creativity, and asks where these ideas -- some plain crazy, and others crazy powerful -- are taking the human story next.

Anthropic Bias explores how to reason when you suspect that your evidence is biased by "observation selection effects"--that is, evidence that has been filtered by the precondition that there be some suitably positioned observer to "have" the evidence. This conundrum--sometimes alluded to as "the anthropic principle," "self-locating belief," or "indexical information"--turns out to be a surprisingly perplexing and intellectually stimulating challenge, one abounding with important implications for many areas in science and philosophy. There are the philosophical thought experiments and paradoxes: the Doomsday Argument; Sleeping Beauty; the Presumptuous Philosopher; Adam & Eve; the Absent-Minded Driver; the Shooting Room. And there are the applications in contemporary science: cosmology ("How many universes are there?", "Why does the universe appear fine-tuned for life?"); evolutionary theory ("How improbable was the evolution of intelligent life on our planet?"); the problem of time's arrow ("Can it be given a thermodynamic explanation?"); quantum physics ("How can the many-worlds theory be tested?"); game-theory problems with imperfect recall ("How to model them?"); even traffic analysis ("Why is the 'next lane' faster?"). Anthropic Bias argues that the same principles are at work across all these domains. And it offers a synthesis: a mathematically explicit theory of observation selection effects that attempts to meet scientific needs while steering clear of philosophical paradox.