Outside Auditor Finds AI Has Roughly Doubled Anthropic's Pace

Read the episode (Narration script)

Good morning and welcome to Cyber World News. One big story at the top today: an outside group got a look inside a leading AI lab and came back with a number worth your full attention. After that, a hacking tool that mostly runs itself, a more modest scorecard from the real economy, and the end of the road for the Cybertruck. Let's get started.

AI researchers have long watched for one particular loop: AI good enough to help build better AI, which then builds the next one faster. On Thursday, the evaluation nonprofit METR published the most detailed outside measurement yet of whether that loop has started at Anthropic. Its answer is probably yes.

Anthropic invited METR in. Over about eleven weeks this fall, its team saw internal benchmarks, compute and spending figures and staff surveys, and interviewed around fifteen researchers. The finding: since October of last year, Anthropic's internal models have been improving on well defined tasks at roughly twice the rate of the previous year and a half. The central estimate is 1.9 times, and METR wouldn't be surprised if the truth landed anywhere from 1.3 to 3. It ruled out the obvious explanation, more training compute. Its best guess is that about 80 percent of the speedup comes from Anthropic using its own AI to do the research.

On the ground, that mostly looks like AI as a very fast assistant. Researchers told METR the overhead of running experiments has been "driven towards zero." Ideas that used to take a day or two to build are now nearly free, so people hand ten of them to an agent and see what works. Fully autonomous AI contributions to the most important training work are still small, though researchers reported "signs of life."

Two things give this weight. METR found no convincing bottleneck that would stop progress from speeding up further over the next twelve months, while admitting it changed its mind on that several times. And earlier this week, Anthropic told investors it may imminently hit its own policy threshold for dramatically accelerated, AI driven progress. On Friday, the forecaster Ajeya Cotra, who wasn't involved in the report, argued that on a plain reading that line has already been crossed, and that doubling a lab's overall progress implies something like a tenfold boost for each researcher. Her picture is a time bubble, with ten days passing inside for every day outside.

The caveats are real. Anthropic redacted most of the evidence METR found most informative, so the public largely has to take METR's word for it. Partway through, Anthropic also dropped a planned check of how it allocates its computing power. Staff polls put their personal speedup around twelve times, which METR calls very loose evidence. Even so, this is an independent group reading the internal numbers and concluding that AI is now meaningfully speeding up the making of AI. How fast that figure moves over the next year is one of the most important numbers in technology.

Outside the labs, the scorecard reads differently. The Bank of England said this week that firms estimate AI has lifted UK productivity by about half a percent in total over three years, with virtually no effect on overall employment so far. Card data from Ramp shows American companies' AI spending barely grew in August and September after a 42 percent jump in July, then picked up about 9 percent in October, helped along by price cuts. Ramp's economist summed it up: "You can use more AI, but use it for less."

Now, what happens when hacking gets automated and cheap. Artex is a Chinese penetration testing tool, released free in July, that coordinates AI agents to probe a system from first look to data theft. Check Point found ten Artex servers whose entire databases were exposed, which meant reading the attackers' own instructions. In one campaign, an operator aimed agents at ten shady iPhone app signing shops for about two weeks and came away with more than 300 Apple code signing certificates and their passwords. Three passed every validity check, which means they could sign any iPhone app. The AI bill was an estimated 700 to 1,500 dollars. Most of the operator's messages simply asked how it was going. When one operator tried Anthropic's Opus model, its safeguards refused, so they switched to Chinese models. Separately, Censys counted more than 800 Artex setups sitting on the open web with no password at all. The break ins mostly came through old basics: hard coded passwords, missing logins, weak credentials.

Tesla is ending the Cybertruck. Elon Musk posted Friday that production in Texas will wind down around the middle of next year to make room for the Cybercab, the two seat robotaxi with no steering wheel. Musk once predicted 250,000 sales a year. Kelley Blue Book counted about 20,000 in the US last year, down nearly half. Its lead engineer says the truck "lives on through Cybercab," which mostly means its factory floor does.

One more before we go. Amazon now lets some US customers text Alexa to shop, after blocking Meta's Muse agent from its site in September. Agents are welcome to shop at Amazon, provided they're Amazon's.

That about wraps it up for today, friends. Thanks for listening; see you tomorrow!