The Fog Blog

What, exactly, is an "existential risk"?

In a paper titled "Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards", published in the Journal of Evolution and Technology, Vol. 9, March 2002, Oxford philosopher Nick Bostrom writes the following:

Because of accelerating technological progress, humankind may be rapidly approaching a critical phase in its career. In addition to well-known threats such as nuclear holocaust, the prospects of radically transforming technologies like nanotech systems and machine intelligence present us with unprecedented opportunities and risks. Our future, and whether we will have a future at all, may well be determined by how we deal with these challenges. In the case of radically transforming technologies, a better understanding of the transition dynamics from a human to a 'posthuman' society is needed. Of particular importance is to know where the pitfalls are: the ways in which things could go terminally wrong. While we have had long exposure to various personal, local, and endurable global hazards, this paper analyzes a recently emerging category: that of existential risks. These are threats that could cause our extinction or destroy the potential of Earth-originating intelligent life. Some of these threats are relatively well known while others, including some of the gravest, have gone almost unrecognized. Existential risks have a cluster of features that make ordinary risk management ineffective. A final section of this paper discusses several ethical and policy implications. A clearer understanding of the threat picture will enable us to formulate better strategies.

Bostrom's entire paper is available here. It's disquieting reading. Bostrom begins his paper as follows:

It's dangerous to be alive and risks are everywhere. Luckily, not all risks are equally serious. For present purposes we can use three dimensions to describe the magnitude of a risk: scope, intensity, and probability. By 'scope' I mean the size of the group of people that are at risk. By 'intensity' I mean how badly each individual in the group would be affected. And by 'probability' I mean the best current subjective estimate of the probability of the adverse outcome.

We can distinguish six qualitatively distinct types of risks based on their scope and intensity... The third dimension, probability, can be superimposed on the two dimensions plotted in the figure. Other things equal, a risk is more serious if it has a substantial probability and if our actions can make that probability significantly greater or smaller.

'Personal', 'local', or 'global' refer to the size of the population that is directly affected; a global risk is one that affects the whole of humankind (and our successors). 'Endurable' vs. 'terminal' indicates how intensely the target population would be affected. An endurable risk may cause great destruction, but one can either recover from the damage or find ways of coping with the fallout. In contrast, a terminal risk is one where the targets are either annihilated or irreversibly crippled in ways that radically reduce their potential to live the sort of life they aspire to. In the case of personal risks, for instance, a terminal outcome could for example be death, permanent severe brain injury, or a lifetime prison sentence. An example of a local terminal risk would be genocide leading to the annihilation of a people (this happened to several Indian nations). Permanent enslavement is another example.

In this paper we shall discuss risks of the sixth category... This is the category of global, terminal risks. I shall call these existential risks.

Existential risks are distinct from global endurable risks. Examples of the latter kind include: threats to the biodiversity of Earth’s ecosphere, moderate global warming, global economic recessions (even major ones), and possibly stifling cultural or religious eras such as the 'dark ages', even if they encompass the whole global community, provided they are transitory... To say that a particular global risk is endurable is evidently not to say that it is acceptable or not very serious. A world war fought with conventional weapons or a Nazi-style Reich lasting for a decade would be extremely horrible events even though they would fall under the rubric of endurable global risks since humanity could eventually recover. (On the other hand, they could be a local terminal risk for many individuals and for persecuted ethnic groups.)

I shall use the following definition of existential risks:

Existential risk – One where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential.

An existential risk is one where humankind as a whole is imperiled. Existential disasters have major adverse consequences for the course of human civilization for all time to come.

Immediately following this, Bostrom asserts that,

Risks in this sixth category are a recent phenomenon. This is part of the reason why it is useful to distinguish them from other risks... The first manmade existential risk was the inaugural detonation of an atomic bomb.

I can't recommend strongly enough that everyone read this paper—simply in order to know just where we stand, and how precarious is our situation.