https://nostalgebraist.tumblr.com/post/781305946452475904

nostalgebraist:

Anthropic’s stated “AI timelines” seem wildly aggressive to me.

As far as I can tell, they are now saying that by 2028 – and possibly even by 2027, or late 2026 – something they call “powerful AI” will exist.

And by “powerful AI,” they mean… this (source, emphasis mine):

In terms of pure intelligence, it is smarter than a Nobel Prize winner across most relevant fields – biology, programming, math, engineering, writing, etc. This means it can prove unsolved mathematical theorems, write extremely good novels, write difficult codebases from scratch, etc.

In addition to just being a “smart thing you talk to”, it has all the “interfaces” available to a human working virtually, including text, audio, video, mouse and keyboard control, and internet access. It can engage in any actions, communications, or remote operations enabled by this interface, including taking actions on the internet, taking or giving directions to humans, ordering materials, directing experiments, watching videos, making videos, and so on. It does all of these tasks with, again, a skill exceeding that of the most capable humans in the world.

It does not just passively answer questions; instead, it can be given tasks that take hours, days, or weeks to complete, and then goes off and does those tasks autonomously, in the way a smart employee would, asking for clarification as necessary.

It does not have a physical embodiment (other than living on a computer screen), but it can control existing physical tools, robots, or laboratory equipment through a computer; in theory it could even design robots or equipment for itself to use.

The resources used to train the model can be repurposed to run millions of instances of it (this matches projected cluster sizes by ~2027), and the model can absorb information and generate actions at roughly 10x-100x human speed. It may however be limited by the response time of the physical world or of software it interacts with.

Each of these million copies can act independently on unrelated tasks, or if needed can all work together in the same way humans would collaborate, perhaps with different subpopulations fine-tuned to be especially good at particular tasks.

In the post I’m quoting, Amodei is coy about the timeline for this stuff, saying only that

I think it could come as early as 2026, though there are also ways it could take much longer. But for the purposes of this essay, I’d like to put these issues aside […]

However, other official communications from Anthropic have been more specific. Most notable is their recent OSTP submission, which states (emphasis in original):

Based on current research trajectories, we anticipate that powerful AI systems could emerge as soon as late 2026 or 2027 […]

Powerful AI technology will be built during this Administration. [i.e. the current Trump administration -nost]

See also here, where Jack Clark says (my emphasis):

People underrate how significant and fast-moving AI progress is. We have this notion that in late 2026, or early 2027, powerful AI systems will be built that will have intellectual capabilities that match or exceed Nobel Prize winners. They’ll have the ability to navigate all of the interfaces… [Clark goes on, mentioning some of the other tenets of “powerful AI” as in other Anthropic communications -nost]

—-

To be clear, extremely short timelines like these are not unique to Anthropic.

Miles Brundage (ex-OpenAI) says something similar, albeit less specific, in this post. And Daniel Kokotajlo (also ex-OpenAI) has held views like this for a long time now.

Even Sam Altman himself has said similar things (though in much, much vaguer terms, both on the content of the deliverable and the timeline).

Still, Anthropic’s statements are unique in being

• official positions of the company
• extremely specific and ambitious about the details
• extremely aggressive about the timing, even by the standards of “short timelines” AI prognosticators in the same social cluster

Re: ambition, note that the definition of “powerful AI” seems almost the opposite of what you’d come up with if you were trying to make a confident forecast of something.

Often people will talk about “AI capable of transforming the world economy” or something more like that, leaving room for the AI in question to do that in one of several ways, or to do so while still failing at some important things.

But instead, Anthropic’s definition is a big conjunctive list of “it’ll be able to do this and that and this other thing and…”, and each individual capability is defined in the most aggressive possible way, too! Not just “good enough at science to be extremely useful for scientists,” but “smarter than a Nobel Prize winner,” across “most relevant fields” (whatever that means). And not just good at science but also able to “write extremely good novels” (note that we have a long way to go on that front, and I get the feeling that people at AI labs don’t appreciate the extent of the gap [cf]). Not only can it use a computer interface, it can use every computer interface; not only can it use them competently, but it can do so better than the best humans in the world. And all of that is in the first two paragraphs – there’s four more paragraphs I haven’t even touched in this little summary!

Re: timing, they have even shorter timelines than Kokotajlo these days, which is remarkable since he’s historically been considered “the guy with the really short timelines.” (See here where Kokotajlo states a median prediction of 2028 for “AGI,” by which he means something less impressive than “powerful AI”; he expects something close to the “powerful AI” vision [“ASI”] ~1 year or so after “AGI” arrives.)

—-

I, uh, really do not think this is going to happen in “late 2026 or 2027.”

Or even by the end of this presidential administration, for that matter.

I can imagine it happening within my lifetime – which is wild and scary and marvelous. But in 1.5 years?!

The confusing thing is, I am very familiar with the kinds of arguments that “short timelines” people make, and I still find the Anthropic’s timelines hard to fathom.

Above, I mentioned that Anthropic has shorter timelines than Daniel Kokotajlo, who “merely” expects the same sort of thing in 2029 or so. This probably seems like hairsplitting – from the perspective of your average person not in these circles, both of these predictions look basically identical, “absurdly good godlike sci-fi AI coming absurdly soon.” What difference does an extra year or two make, right?

But it’s salient to me, because I’ve been reading Kokotajlo for years now, and I feel like I basically get understand his case. And people, including me, tend to push back on him in the “no, that’s too soon” direction. I’ve read many many blog posts and discussions over the years about this sort of thing, I feel like I should have a handle on what the short-timelines case is.

But even if you accept all the arguments evinced over the years by Daniel “Short Timelines” Kokotajlo, even if you grant all the premises he assumes and some people don’t – that still doesn’t get you all the way to the Anthropic timeline!

To give a very brief, very inadequate summary, the standard “short timelines argument” right now is like:

1. Over the next few years we will see a “growth spurt” in the amount of computing power (“compute”) used for the largest LLM training runs. This factor of production has been largely stagnant since GPT-4 in 2023, for various reasons, but new clusters are getting built and the metaphorical car will get moving again soon. (See here)
2. By convention, each “GPT number” uses ~100x as much training compute as the last one. GPT-3 used ~100x as much as GPT-2, and GPT-4 used ~100x as much as GPT-3 (i.e. ~10,000x as much as GPT-2).
3. We are just now starting to see “~10x GPT-4 compute” models (like Grok 3 and GPT-4.5). In the next few years we will get to “~100x GPT-4 compute” models, and by 2030 will will reach ~10,000x GPT-4 compute.
4. If you think intuitively about “how much GPT-4 improved upon GPT-3 (100x less) or GPT-2 (10,000x less),” you can maybe convince yourself that these near-future models will be super-smart in ways that are difficult to precisely state/imagine from our vantage point. (GPT-4 was way smarter than GPT-2; it’s hard to know what “projecting that forward” would mean, concretely, but it sure does sound like something pretty special)
5. Meanwhile, all kinds of (arguably) complementary research is going on, like allowing models to “think” for longer amounts of time, giving them GUI interfaces, etc.
6. All that being said, there’s still a big intuitive gap between “ChatGPT, but it’s much smarter under the hood” and anything like “powerful AI.” But…
7. …the LLMs are getting good enough that they can write pretty good code, and they’re getting better over time. And depending on how you interpret the evidence, you may be able to convince yourself that they’re also swiftly getting better at other tasks involved in AI development, like “research engineering.” So maybe you don’t need to get all the way yourself, you just need to build an AI that’s a good enough AI developer that it improves your AIs faster than you can, and then those AIs are even better developers, etc. etc. (People in this social cluster are really keen on the importance of exponential growth, which is generally a good trait to have but IMO it shades into “we need to kick off exponential growth and it’ll somehow do the rest because it’s all-powerful” in this case.)

And like, I have various disagreements with this picture.

For one thing, the “10x” models we’re getting now don’t seem especially impressive – there has been a lot of debate over this of course, but reportedly these models were disappointing to their own developers, who expected scaling to work wonders (using the kind of intuitive reasoning mentioned above) and got less than they hoped for.

And (in light of that) I think it’s double-counting to talk about the wonders of scaling and then talk about reasoning, computer GUI use, etc. as complementary accelerating factors – those things are just table stakes at this point, the models are already maxing out the tasks you had defined previously, you’ve gotta give them something new to do or else they’ll just sit there wasting GPUs when a smaller model would have sufficed.

And I think we’re already at a point where nuances of UX and “character writing” and so forth are more of a limiting factor than intelligence. It’s not a lack of “intelligence” that gives us superficially dazzling but vapid “eyeball kick” prose, or voice assistants that are deeply uncomfortable to actually talk to, or (I claim) “AI agents” that get stuck in loops and confuse themselves, or any of that.

We are still stuck in the “Helpful, Harmless, Honest Assistant” chatbot paradigm – no one has seriously broke with it since that Anthropic introduced it in a paper in 2021 – and now that paradigm is showing its limits. (“Reasoning” was strapped onto this paradigm in a simple and fairly awkward way, the new “reasoning” models are still chatbots like this, no one is actually doing anything else.) And instead of “okay, let’s invent something better,” the plan seems to be “let’s just scale up these assistant chatbots and try to get them to self-improve, and they’ll figure it out.” I won’t try to explain why in this post (IYI I kind of tried to here) but I really doubt these helpful/harmless guys can bootstrap their way into winning all the Nobel Prizes.

—-

All that stuff I just said – that’s where I differ from the usual “short timelines” people, from Kokotajlo and co.

But OK, let’s say that for the sake of argument, I’m wrong and they’re right. It still seems like a pretty tough squeeze to get to “powerful AI” on time, doesn’t it?

In the OSTP submission, Anthropic presents their latest release as evidence of their authority to speak on the topic:

In February 2025, we released Claude 3.7 Sonnet, which is by many performance benchmarks the most powerful and capable commercially-available AI system in the world.

I’ve used Claude 3.7 Sonnet quite a bit. It is indeed really good, by the standards of these sorts of things!

But it is, of course, very very far from “powerful AI.” So like, what is the fine-grained timeline even supposed to look like? When do the many, many milestones get crossed? If they’re going to have “powerful AI” in early 2027, where exactly are they in mid-2026? At end-of-year 2025?

If I assume that absolutely everything goes splendidly well with no unexpected obstacles – and remember, we are talking about automating all human intellectual labor and all tasks done by humans on computers, but sure, whatever – then maybe we get the really impressive next-gen models later this year or early next year… and maybe they’re suddenly good at all the stuff that has been tough for LLMs thus far (the “10x” models already released show little sign of this but sure, whatever)… and then we finally get into the self-improvement loop in earnest, and then… what?

They figure out to squeeze even more performance out of the GPUs? They think of really smart experiments to run on the cluster? Where are they going to get all the missing information about how to do every single job on earth, the tacit knowledge, the stuff that’s not in any web scrape anywhere but locked up in human minds and inaccessible private data stores? Is an experiment designed by a helpful-chatbot AI going to finally crack the problem of giving chatbots the taste to “write extremely good novels,” when that taste is precisely what “helpful-chatbot AIs” lack?

I guess the boring answer is that this is all just hype – tech CEO acts like tech CEO, news at 11. (But I don’t feel like that can be the full story here, somehow.)

And the scary answer is that there’s some secret Anthropic private info that makes this all more plausible. (But I doubt that too – cf. Brundage’s claim that there are no more secrets like that now, the short-timelines cards are all on the table.)

It just does not make sense to me. And (as you can probably tell) I find it very frustrating that these guys are out there talking about how human thought will basically be obsolete in a few years, and pontificating about how to find new sources of meaning in life and stuff, without actually laying out an argument that their vision – which would be the common concern of all of us, if it were indeed on the horizon – is actually likely to occur on the timescale they propose.

It would be less frustrating if I were being asked to simply take it on faith, or explicitly on the basis of corporate secret knowledge. But no, the claim is not that, it’s something more like “now, now, I know this must sound far-fetched to the layman, but if you really understand ‘scaling laws’ and 'exponential growth,’ and you appreciate the way that pretraining will be scaled up soon, then it’s simply obvious that –”

No! Fuck that! I’ve read the papers you’re talking about, I know all the arguments you’re handwaving-in-the-direction-of! It still doesn’t add up!

Dwarkesh Patel 00:34:26
Yeah. First of all, I actually just really appreciate that there is an AI organization out there where- because there’s other people who take the compute perspective seriously, or try to think empirically about scaling laws and data and whatever. And taking that perspective seriously leads people to just be like, “okay, 2027 AGI”, which might be correct, but it is just interesting to get, “no, we’ve also looked at the exact same arguments, the same papers, the same numbers. And we’ve come to a totally different conclusion”.

So I asked Dario this exact question two years ago, when I interviewed him, and it went viral.

Ege Erdil 00:35:11
Didn’t he say AGI in two years?

Dwarkesh Patel 00:35:13
That, but Dario’s always had short timelines.

Ege Erdil 00:35:15
Okay, but we are two years later.

Dwarkesh Patel 00:35:18
Did he say two years? I think he actually did say two years.

Ege Erdil 00:35:20
Did he say three years?

Tamay Besiroglu 00:35:21
So we have one more year.

Dwarkesh Patel 00:35:22
One more year.

Tamay Besiroglu 00:35:23
Better work hard.

Dwarkesh Patel 00:35:27
But he’s, I mean, I think he’s like, he in particular has not been that well calibrated. In 2018, he had like…

Tamay Besiroglu 00:35:33
I remember talking to a very senior person who’s now at Anthropic, in 2017. And then he told various people that they shouldn’t do a PhD because by the time they completed it everyone will be automated.

lol

https://nostalgebraist.tumblr.com/post/781305946452475904

https://nostalgebraist.tumblr.com/post/780750780895526912

rickparsa:

Happy 413!!!

https://nostalgebraist.tumblr.com/post/780750780895526912

https://nostalgebraist.tumblr.com/post/780555620787126272

ALT

ALT

ALT

ALT

ALT

I typed out these messages in a discord server a moment ago, and then thought “hmm, maybe I should make the same points in a tumblr post, since I’ve been talking about software-only-singularity predictions on tumblr lately”

But, as an extremely lazy (and somewhat busy) person, I couldn’t be bothered to re-express the same ideas in a tumblr-post-like format, so I’m giving you these screenshots instead

(If you’re not familiar, “MCP” is “Model Context Protocol,” a recently introduced standard for connections between LLMs and applications that want to interact with LLMs.

Its official website is here – although be warned, that link leads to the bad docs I complained about in the first message. The much more palatable python SDK docs can be found here.)

EDIT: what I said in the first message about “getting Claude to set things up for you locally” was not really correct, I was conflating this (which fits that description) with this and this (which are real quickstarts with code, although not very good ones, and frustratingly there’s no end-to-end example of writing a server and then testing it with a hand-written client or the inspector, as opposed to using with “Claude for Desktop” as the client)

https://nostalgebraist.tumblr.com/post/780555620787126272