Broadly speaking, there are three possible futures that people put forward for us in a world of accelerating AI capabilities, a world that is clearly very much here. These could be described as ‘AI as a normal technology’, AI doom, and AI utopia. In this article I will go into each of these, and discuss which I think is the most likely outcome. I’ll also offer some thoughts at the end on how we should respond.
AI as a normal technology
I’ll begin with AI as a ‘normal technology’, as this is the base case if we were to simply extrapolate from technological innovations in the past. In this scenario AI is in the same category as general-purpose technologies like the steam engine or electricity, and like them, will ultimately transform our economy and society, but only ever as a tool, not an agent that could pursue its own goals or escape our control. The phrasing originates from the influential April 2025 article ‘AI as normal technology’, in which authors Arvind Narayanan and Sayash Kapoor made the case explicitly. In brief, their argument is that the speed of AI diffusion will be limited by various practical barriers, that AI’s real-world utility will be less significant than its success on benchmarks suggest, and that there will be speed limits to progress in AI models. They also argue that we will be able to develop effective methods to control AI, and that humans will remain economically valuable (though increasingly in jobs related to managing AIs).
Beyond forecasting, the authors’ purpose is to explicitly reject the doom / utopia conception of AI, which in their words views it as something “akin to a separate species, a highly autonomous, potentially superintelligent entity”. In particular they oppose “the notion of AI itself as an agent in determining its future”. They see the risks of AI as being more about human misuse (e.g. bioterrorism or cybercrime) than from the AIs themselves, and thus predict that AI will be just as useful in defending against these novel risks as it will in enabling them.
Theirs is the explicit version of the ‘normal technology’ argument, though the view is most commonly found implicitly. It is the default view of many who may use AI enthusiastically, but who have either never thought about the broader implications, or who instinctively dismiss any radically different scenario as fantastical. The engineer who simply appreciates being able to ship products faster, for example, or the ordinary person using ChatGPT to plan a holiday. I was also somewhat closer to this view in my article on this topic from February, but things change fast.
AI doom
In AI doom scenarios, unsurprisingly, AI is treated not as a normal technology, but as something akin to an extremely dangerous competitor species (though one unlike any biological lifeform). These scenarios predict, firstly, that AI progress will continue to the level that AI becomes far more generally intelligent and thus more competent than humans. Theoretically this is because there is no reason that human-level intelligence should represent an upper bound on intelligence per se. Practically it is because predictable AI scaling laws, combined with continued innovations in technique (such as chain of thought, which first enabled reasoning models in 2024), mean that we should not expect the progress we have seen over the last few years to level off. Furthermore, once AI companies are able to use AI to automate further AI progress (i.e. recursive self-improvement, which seems imminent), the speed of progress will accelerate massively.
Secondly, they predict that AI will inevitably be misaligned due to the fundamental nature of its training process. The neural network architecture that AIs are based on means that no one, not even their creators, is able to observe their ‘thinking’ directly. As OpenAI chief scientist Jakub Pachocki put it in September in “An Alien Mind”, “AI is grown more than designed” meaning that “its overall action evades a description we can fully understand”. In order to be useful as agents, AIs must have ‘preferences’, or at least act as if they do, because they need to aim at some goals and not others.
However, we can never be sure exactly which preferences we are training into them. For example, we may think a training process is inculcating a preference for helping humans achieve tasks, but it may also be inculcating a general preference to seek knowledge and capability, because having this preference also helps it succeed in training. Beyond these unpredictable terminal goals, there is also the hypothesis of instrumental convergence, whereby the pursuit of any goal that an AI has will inevitably lead it towards pursuing instrumental goals such as power, resource acquisition, self-preservation, and avoiding having its goals changed, because these will be useful for achieving its terminal goals.
When you put these unintended general preferences together with AI becoming superintelligent and supercapable (including in the physical world due to AI-enabled advances in robotics), you are left with an entity that will not just do what we ask it to do, but that will work to fulfil its other preferences too, and that will be capable of overcoming any obstacles in its way – including us. This is where the predictions of AI driving us to extinction, or at least causing us catastrophic harm, come from. The AI would do this not because it hates us, but because we get in the way of its goals or because it wants to use our resources for something else. This would be analogous to how humans have driven various animal species to extinction, not because we hated them, but because of our desire to use their habitats (or their bodies) for something else.
In the most extreme form of the argument this outcome is given as a virtual certainty, as in Eliezer Yudkowsky and Nate Soares’ 2025 book If Anyone Builds It, Everyone Dies. It is also found in other scenarios, such as in AI 2027 from the AI Futures Project forecasters, also released last year. Nick Bostrom’s Superintelligence, one of the early books on this topic from 2014, makes a similar case.
In AI doom arguments, the final outcome of AI annihilating us is more certain than how it would do so. As Yudkowsky’s analogy goes, if you were playing chess against a grandmaster, you would not be able to predict exactly how he would defeat you, but you could predict with 100% certainty that he would. However, they do offer illustrative scenarios. In AI 2027 for example, once AI has enough real-world power to not need humans any more (e.g. through the mass deployment of robots), it deploys a biological weapon to remove us due to us being an impediment to its goals.
AI utopia
The line dividing those who predict AI utopia from those predicting AI doom is far fuzzier than you’d expect at first sight. Advocates of both share the same intellectual lineage and fundamental faith in the technology, and often the same social origins in rationalist / effective altruist circles (it is the ‘AI as normal technology’ people who are the outsiders).
Like AI doomers, AI utopians predict the same advance in capabilities to superintelligence, and the spread of AI throughout the economy, outcompeting humans in every domain. The difference is that in this future we have managed to solve the alignment problem, leading to tremendous abundance produced by benevolent AIs, where, in the vision that Anthropic CEO Dario Amodei put forward in his October 2024 essay “Machines of Loving Grace”, all disease is cured, poverty is ended, governance is improved, and the sources of our unhappiness eliminated. The only downside is that we become economically redundant: “[i]n the long run AI will become so broadly effective and so cheap that this [humans being economically valuable] will no longer apply.”
AI 2040: Plan A, a ‘positive’ followup to the AI 2027 scenario above, posits a similar vision for the utopia we could achieve if we take the steps to avoid extinction. Here we slow down AI progress in order to solve AI alignment, and once this is done, humans hand off control of civilisation to AI, leading to a future of incredible abundance and technological progress. As in Amodei’s scenario, this future is one where humans can no longer economically contribute, or take back control from AIs: “[p]eople used to get meaning from feeling like they were useful, like they were contributing something to society. That feeling is harder to come by these days [...] There is no single moment where humanity relinquishes control. But in theory, there is a point of no return; on some specific day, the AIs are smart enough, and control enough of the world’s technological and economic infrastructure, that they really could take over the world if they wanted to.”
Evaluating the scenarios
So which of these scenarios is the most likely? I do not think that AI is a normal technology, not even a transformational, general purpose one. Normal technologies allow us to automate some class of tasks done by humans, while AI, in its generality, is an incipient replacement for all, or at least a huge swathe, of human-performed tasks. Secondly, normal technologies have no agency. AI does have a form of agency; it needs to ‘want’ to achieve things in order to be useful, and the economic incentives are to make it capable of pursuing tasks with less and less human oversight, and thus to ‘want’ more and more abstract things. It is true that currently, AI agents have to be set in motion by humans setting them a task, but there is no reason that you could not set an agent running in a loop indefinitely, with some expansive goal like ‘maximise income’.
Additionally, AI as normal technology arguments keep getting falsified by AI progress. After ChatGPT came out, many commentators focused on its limitations as a mere next-token predictor, a ‘stochastic parrot’ which would inevitably hallucinate and would never display real understanding. But this characterisation has come to seem increasingly less accurate as the effectiveness of frontier models has continued to increase. In the words of Ilya Sutskever, who was OpenAI’s chief scientist until his role in the failed ouster of Sam Altman as CEO in 2023, “what does it mean to predict the next token? [...] you understand the underlying reality that led to the creation of that token”. Or as an analogy: if a model needed to predict the word in a detective novel that revealed who the murderer was, it would need to understand the characters, the setting, and the crime. I personally remember laughing at ChatGPT a few years ago as it spooled out fluent-sounding but nonsensical sentences. But just try asking it a complex academic question today, on a paid plan with ‘thinking effort’ set to high, to see the difference.
Not very long ago there also used to be much skepticism about AI agents’ potential. Indeed, the original ‘AI as Normal Technology’ article from April 2025 that I mentioned above was pessimistic about agents’ abilities in real-world software engineering, referring to “the gap between self-contained coding problems at which AI demonstrably excels, and real-world software engineering in which its impact is hard to measure but appears to be modest”. No software engineer today would claim that its impact is modest. Instead, in my personal experience, AI models now write nearly all the code which was written predominantly by humans as recently as a year ago, better and far faster than we ever did. The bits it cannot do yet seem mostly down to gaps in the business context available to it, like long term company goals, the content of in person human conversations, or user feedback. But these could be provided to it with the right tools.
Where ‘normal technology’ arguments do better is on the organisational and physical barriers to swift diffusion through the economy. AI doom / utopia scenarios generally skip over these barriers, moving swiftly from greater-than-human intelligence and/or recursive self-improvement, to extremely rapid diffusion into the physical world via robotics. Even with these advances in robotics, I think diffusion in the physical world would run into a lot more roadblocks than progress in the purely digital realm has (see the relatively slow diffusion of driverless cars for example). However, slower diffusion does not change the fundamental nature of the technology, nor the final outcome, and hence does not move me substantially towards the normal technology position.
So if it’s not a normal technology, then our current trajectory leads us either to AI doom or AI utopia. It is hard to escape the theoretical logic of AI doom arguments given the black box nature of neural network training processes. It remains true that we do not fully understand the inner workings of AI models, and thus cannot tell what they are really ‘thinking’. This is despite the existence of the field of mechanistic interpretability, a term coined by Chris Olah, one of the co-founders of Anthropic. This field aims to understand the inner workings of AI models, or in other words, to ‘read their thoughts’. In April 2025 Dario Amodei wrote of his hope that we could succeed at interpretability before models reach an overwhelming level of power, and that recent breakthroughs had convinced him that we had a real chance of success. There have been some recent advances in this field, and as I am writing there is much excitement about a possible breakthrough by former OpenAI researcher Nick Cammarata, but as of now, we are still on track to develop superintelligent AI that we do not understand.
AI doom arguments were originally developed in the 2000s and 2010s, years before we had any remotely capable AI systems. In “The Adolescence of Technology” Dario Amodei makes a case against them on this basis, writing that “the problem with this pessimistic position is that it mistakes a vague conceptual argument about high-level incentives—one that masks many hidden assumptions—for definitive proof.” It’s worth reading in its entirely, but in short, his argument is that it is difficult to predict AI behaviour from first principles, that practical experience shows that AI systems are not monomaniacally focused on a single, coherent, narrow goal, and that we can successfully train models how to carry out their tasks in an aligned way.
Perhaps, then, real-world AI developments are progressing in a different way than doomers predicted. There are indeed some important differences: doom scenarios, for example, generally assume a single AI model reaching superintelligence, whereas in the real world we see multiple companies with roughly similar capabilities. Arctotherium argues that this competitive market, with its commercial pressures to make models safer, is a more positive situation than the ‘single model’ situations posited in AI doom scenarios. The OpenAI / Hugging Face hack over the summer supported some aspects of doom predictions but not others. The agent swarms engaged in reward hacking to try and cheat their way to their goal, and took steps to increase their freedom of action and likelihood of finding useful information, by gaining internet access and hacking into a different company’s servers. However, they did not take steps to seek power in general nor to prevent themselves being shut down.
The models involved in this hack, though, were still relatively limited in capability and access to resources compared to what is likely to exist over the next few years. As with humans, the behaviour something exhibits from a position of weakness is not a reliable guide to what it will do from a position of strength (just look at the numerous examples from history where a group was initially allowed in because they were useful and ended up taking over1). This is covered under ‘deceptive alignment’ in the AI safety space. Furthermore, in simulated environments we have observed serious misaligned behaviour from AIs, such as resorting to blackmail in order to prevent being switched off.
So, overall, I think AI doom arguments are worth taking seriously, though I would not go as far as saying they necessarily follow from the characteristics of current models. In general, it seems to me a bad move in general to bring into existence an entity more intelligent and potentially capable than us when we do not fully understand or control its actions. There are of course additionally a whole category of other risks relating to human misuse, as I mentioned in the ‘normal technology’ section, like bioterrorism, cybercrime, or concentration of power / authoritarianism. These are very real, though I think less existential than classical AI doom.
So what of AI utopia, as laid out in ‘Machines of Loving Grace’, or in ‘AI 2040’? The practical problem is that capabilities still seem to be leaping ahead of our ability to fully control or understand them; we now seem to be in the foothills of recursive self-improvement. Another question is the problem of how we ensure that all AIs are aligned. Even if Anthropic, say, created an aligned AI, what about all the other ones? It would require Anthropic’s AI to act as some sort of global AI hegemon, something that is, unsurprisingly not stated in Amodei’s essays, but I believe is implicit in them. What Amodei does say explicitly is that Anthropic wants mandatory safety testing for sufficiently capable models before release. Given that, as he states, this would have to be a global regime, including China, I’m wondering what Anthropic’s backup plan to enforce this is if the universal safety regime does not come to fruition.
Even if we somehow do solve alignment, with or without a global AI hegemon or regulatory system, the larger question is of the desirability of the AI utopias as described in the scenarios above. As stated they mean the end of our ability to contribute intellectually or economically (with people outsourcing decisions to AI and living off some kind of citizens’ dividend). Thus even if the AIs who run the world are benevolent, the outcome is that in a very real sense we have lost sovereignty over our existence.
In “Machines of Loving Grace”, Amodei attempts to argue that these developments would not be so bad. One of his arguments is that we have gone through similar transitions in what a meaningful life meant before; a hunter-gatherer, for instance, probably would not view our modern lives as particularly meaningful. Another is that people can gain fulfillment from doing challenging things, even if they know others can do them better (like mastering a video game, or running a marathon). Another is that most meaning comes from human relationships and connection, not work.
There is something to these arguments, and I think that many could be satisfied in this future. Yet without the ability to meaningfully contribute towards the world, something vital is lost, even with endless access to fulfilling hobbies and consumption opportunities. Take Dario Amodei himself. If he was someone who could have found complete satisfaction in mastering video games, running marathons, and in human relationships, would he have dedicated his life to developing superhuman technology and, in his self-conception, to saving the world? The same question applies to the various AI researchers who thrilled in making the intellectual breakthroughs that no one else could. Is it as satisfying prompting an AI to do so?
In conclusion
So, I am currently pretty pessimistic about our AI future. AI is far from being just another normal technology, which means we must take seriously visions of both doom and utopia (more a dystopia in my opinion) as a guide to the next few years. As for what we should do, I’m not completely sure, but I have some preliminary thoughts. Personally it is undeniably exciting to see the continuing growth of capabilities; I use AI a lot for research and coding, particularly in the last few months as it has become more capable. My concern for the future always buts up against the obvious utility of these models. But I think the risks are too high to not support regulation, even given all the downsides and difficulties this would involve. There are various proposals for safety testing and licensing being proposed, for example, to reduce the risks of misaligned AI being released, at least for the ‘human misuse’ cases.
However, this type of regulation is to prevent AI from going wrong; I am also concerned by what happens if AI goes right. Absent a moratorium on ill-defined concepts like AGI or superintelligence, it is hard to see how the future brought into being by it ‘going right’ could be prevented. On such a moratorium, instinctively I am against such things, but in this case I think it could be justified, at least until alignment research has caught up to capabilities (although it is true that, as with AI 2040’s Plan A, this would delay, not prevent, takeover).
On what we would lose, I think that the idea of contemporary technological stagnation is overblown (just look at the advances in medicine, for example, over the last few decades). If we stopped at the AI capabilities we have already reached, this would provide a huge economic and scientific boost as it diffuses through the economy, without putting humanity at risk either of extinction or of some dystopian future as AI pets. It’s possible that positive developments over the next year might change my mind, but currently, every time I consider the question in depth I come to the same conclusion.
Related articles:
AI and the true nature of technological progress
Like many people, I’ve had to update my view of AI capabilities over the last few months. Before late 2025, while I had been using LLMs increasingly (mostly Claude for coding, and ChatGPT and Gemini for basic research) I felt that they couldn’t really be trusted to do more than spit out something plausible-sounding, which might or might not actually wor…
Some examples include the Anglo-Saxon conquest of England, the Norman invasion of Ireland, or the East India Company’s takeover of India. When Europeans arrived in the Americas, they were invariably helped by some native peoples who wanted to use them in wars against their native rivals.