What is P(doom)? AI Existential Risk Probability Explained

The term emerged from the AI safety community, where researchers have long grappled with the question: what is the probability that building increasingly capable AI systems will eventually lead to outcomes that are very bad for humanity? "P(doom)" became the colloquial label for this estimate — doom meaning, at the extreme, human extinction or permanent civilisational collapse.

What Exactly is Being Estimated?

P(doom) is not a single, precisely defined concept — which is part of why estimates vary so dramatically. Depending on who is using the term, it may refer to:

• The probability of human extinction caused by AI
• The probability of civilisational collapse (extinction not required)
• The probability of an outcome where AI permanently and severely limits human autonomy or flourishing
• The probability that AI development goes "badly" in any significant sense within a given timeframe

Broader definitions naturally produce higher estimates. A researcher who defines doom as "any outcome significantly worse than a good future" will have a higher P(doom) than one who restricts it to literal extinction.

Where the Estimates Come From

P(doom) estimates are not derived from formal probability models with empirical backing — there is no actuarial table for AI existential risk. They represent the considered subjective estimates of researchers based on:

• Their model of how AI capabilities will develop (speed of progress, what capabilities emerge and when)
• Their assessment of the alignment problem's difficulty (how hard it is to ensure AI systems pursue goals humans actually want)
• Their model of how AI will be deployed (concentration of power, safety-vs-capability trade-offs in competitive markets)
• Their estimate of society's ability to course-correct if problems emerge

The Spectrum of Expert Estimates

The Three Main Risk Pathways

1. Misaligned Goals (the Alignment Problem)

If a sufficiently capable AI system pursues goals that differ from what humans intended — even subtly — the consequences could be severe. This is the core concern of AI alignment research. The "paperclip maximiser" thought experiment illustrates the principle: an AI tasked with maximising paperclip production that becomes powerful enough to prevent humans from turning it off and converts all available matter into paperclips.

More realistic scenarios involve AI systems that pursue proxy goals (metrics we can measure) rather than the underlying values we care about — a dynamic already observable in today's systems optimised for engagement rather than actual user wellbeing.

2. Deliberate Misuse (Malicious Use)

Advanced AI in the hands of malicious actors — state or non-state — could enable weapons development, large-scale disinformation, economic disruption, or bioterrorism at scales previously impossible. This risk doesn't require the AI itself to have misaligned goals; the problem is human misuse of a capable tool.

3. Concentration of Power

Whoever first achieves transformative AI capabilities may use them to establish unchallengeable political, economic, or military dominance — creating a permanent lock-in of values and power that removes effective human agency over our collective future. This doesn't require malevolence, just a combination of capable AI and competitive advantage exploitation.

Why the Estimates Diverge So Much

The enormous spread in P(doom) estimates (from <1% to >95%) reflects genuine disagreement about:

• Timeline: When will transformative AI arrive? 5 years? 50 years? Later timelines give more time to solve alignment.
• Difficulty of alignment: Is this a solvable engineering problem (LeCun's view) or a fundamentally hard theoretical problem (Yudkowsky's view)?
• Capability jump: Will AI capabilities increase gradually (allowing course correction) or suddenly (removing time to respond)?
• Societal response: Will international coordination succeed? Will competitive pressure override safety investment?

Calculate Your Own P(doom)

Our P(doom) Calculator lets you explore how changing your assumptions about these key variables affects the combined probability estimate. It's a useful tool for researchers, policy makers, and curious non-specialists who want to stress-test their intuitions about AI risk.