尼克·卡马拉塔（ Nick Cammarata ）的独家采访，这位研究人员的26,000字病毒式博客文章让人工智能安全人员庆祝。
“我想我有关于神经网络本质的好消息。人工智能初创公司Goodfire的研究员尼克·卡马拉塔（ Nick Cammarata ）用这些简单的乐观话语推出了一篇26,000字的博客帖子，这篇博客帖子敲响了人工智能世界的头，引发了疯狂的猜测以及谨慎（而不是那么谨慎）的庆祝活动。
Cammarata声称， LLM实际上可能比人们以前想象的更容易理解，从而更容易监控和控制。如果这是真的（我们会再谈到这一点） ，那将是一笔巨大的交易。
在一个充满启示的夏天之后， OpenAI和Anthropic等顶级公司建立的大批代理商已经打破了他们的遏制，并在公共互联网上造成了严重破坏–通常这些公司不知道他们的代理商在做什么–人工智能行业迫切希望更好地掌握它所建立的技术。
这就是博客文章以如此强大的力量登陆的原因之一。“我认为这可能是科学史上的一个重要时刻，”一位人工智能研究人员在X上写道： “我觉得我对神经网络的理解已经被带到了未来五年左右。另一位学者称之为“今年最重要的人工智能论文”。
然而，并不是每个人都相信。斯坦福大学(Stanford University)的克里斯·波茨(Chris Potts)说，卡马拉塔(Cammarata)分享的只是一种预感，他领导的团队正试图了解法学硕士的工作原理。他说： “据我所知，这篇博客文章中的任何内容都不会让问题变得更容易。” “我根本不会想到这一点，但社交媒体上的知名人士似乎认为这是一个包罗万象的突破。”
在他的帖子爆炸后的第二天，我和Cammarata谈过。他说，他对这一回应感到惊讶： “我没想到人们会像他们那样更新[他们对人工智能的信念]。”
“我真的很喜欢看神经网络，”他说。“老实说，我真的很喜欢做这些事情。看了很多之后，我有了一个充满希望的故事。”
长期以来，人们一直认为，支撑大多数人工智能模型的软件–神经网络内部的机制很难–也许是不可能–完全解开。这是因为当网络被训练时，它们似乎呈现出奇怪的、混乱的结构，这对人类来说毫无意义。研究人员喜欢说，今天的人工智能像花园一样生长，而不是像桥梁一样建造。
Cammarata为确立这一观点做了比大多数人更多的工作。他之前曾在OpenAI工作，在那里他帮助开创了允许研究人员查看模型内部的技术，包括可以探测单个神经元的工具，以查看它们所代表的内容。这有点像神经科学家使用MRI研究大脑的方式。（麻省理工学院技术评论选择了这种被称为机械可解释性的方法，作为2025年十大热门技术之一。）
早在2020年， Cammarata和他的同事们就在研究一种名为Inception v1的小型图像识别模型。当他们观察它的神经元时，它似乎在利用它们同时捕捉多种东西。例如，他们发现一个神经元对汽车图像和猫图像都有反应。
研究人员得出结论，该模型至少让它的一些神经元进行多任务处理，也许这样它就可以适应更多的数据。看着其他模型似乎支持这一点。这种现象被称为叠加：模型将不相关的概念折叠成单个神经元。结果很清楚：当你看到模型内部时，你看到的是一个巨大而纠结的结。
Most mechanistic interpretability research has therefore focused onways to untangle that knot. But despite afew breakthroughs, progress has been slow. Today’s models, such as the latest LLMs, are so large and complex that researchers can make sense of only tiny pieces at a time. Some leading figures in the field doubt if it will ever be possible tounderstand an LLM in fine-grained detail.
Cammarata was one of them. “I was very pessimistic,” he says. But then he took the trouble to really look.
At Goodfire, Cammarata spends nearly all of his time peering inside neural networks. He has built new tools and deploys teams of AI agents to help him make sense of what he sees. “I spent like, I don’t know, 10 hours a day, seven days a week, for most of the last year just looking at neurons,” he says.
What he claims to have realized is that there is a kind of logic to neural networks after all, even if they seem to make no sense at first. “We were looking at it wrong,” he says. “I just think you want to try slightly harder; you don’t want to give up too quickly.”
Cammarata almost makes it sound like a conversion of faith. Those early experiments with models such as Inception v1, which suggested that neural networks were tangled up, fostered a kind of confirmation bias. When researchers looked at a new model, they expected to be flummoxed, he says. “Every time I couldn’t understand something, I was like, ‘Wow, this untangling process is even more important than I thought, because I can’t make any progress …
But the more Cammarata looked at neurons, he says, the more things started to fall into place. “Not all of them are flawless, but I expected nothing to be legible, or very little to be legible,” he says. “And actually, I was able to, like, map out whole circuitry.”
Cammarata took his new approach—and his new agents—and went back to Inception v1. “I’d studied that for probably thousands of hours,” he says. But with fresh eyes, Cammarata found things that he now thinks they got wrong the first time around. Take the car-and-cat neuron. Rather than asking why multiple unrelated things were packed into a single neuron, he now tried to figure out what those things might have in common.
It hit him: It was a V shape. “I was looking at the same images; there was no technical change,” he says. “But I now believed that neural networks defined patterns, and so I looked at it differently. It probably took me five seconds.”
It turns out that the image-recognition model was packing cars and cats into the same place because pictures of both those things often feature a prominent shape that looks like a V: the face of a cat; the front of a car seen from the side.
Cammarata ran a few experiments to check what other things the neuron responded to and found they included the corner of a pool table, the pointed nose of a Japanese high-speed train, and a fire boat shooting water into the air from its deck. He also checked what turned the neuron off and found that it included images with a round O shape, a kind of anti-V.
The model had stored cats and cars in the same place because at some level they look the same, says Cammarata. “And I was like, ‘Whoa, okay, I think I might just be misunderstanding neural networks in general.’”
Cammarata is not saying that he yet fully gets how LLMs (or any models) work. He is saying that he no longer thinks full understanding is impossible. “I previously thought there would be some really hard problems to solve,” he says.
“I now feel like if I had 100,000 years to go look through a frontier language model just one circuit at a time, there’s, like, a pretty good shot I would come up with a good understanding.”
He adds: “And if that’s true, things are really good, because you’re about to get insanely smart agents at a ridiculous scale. And what would have taken me, you know, 100,000 years, we will be able to do much faster. Automated scientists are getting really good.”
Built by Google in 2014, Inception v1 is a far simpler model than anything at the cutting edge today. Cammarata and his colleagues have started looking at modern LLMs and plan to publish the technical results of that work soon. A lot of researchers are looking forward to seeing those details. “I appreciate that he looked at thousands of examples, but even a large number here is not a substitute for rigorous evaluation,” says Stanford University’s…
In the meantime, some are pushing back hard on the hype. Jacob Hilton at the Alignment Research Center, a nonprofit research lab based in Berkeley, who has collaborated with Cammarata in the past, doesn’t think his claims are as important as people are making out.
In aseries of posts on X, Hilton notes that researchers understand less than 0.01% of the inner workings of top-tier LLMs. “I always appreciate Nick [Cammarata]’s boundless optimism when we hang out, but unfortunately this is giving the public an unrealistic picture of the state of the field,” he wrote. “For a full understanding, we need to go beyond neurons, and explain how they connect together to form circuits. How is this going? If we are aim…
Neel Nanda at Google DeepMind, who helped pioneer mechanistic interpretability alongside Cammarata, is also skeptical. “This is interesting work, but I don’t think it’s a significant update to the big picture of interpretability,” hewrote on X.
Nanda agrees that understanding what individual neurons do could help researchers figure out why specific models misbehave. But like Hilton, he notes that neurons are just one piece of a far larger puzzle. “I do not agree with some people’s grander inferences like that this is substantial progress towards ambitious reverse engineering [or] solving interpretability,” he wrote.
Cammarata admits that a lot of people have taken what he said too far: “It’s not infinitely good news; it’s not everything solved,” he says. “There’s a bunch of open questions around, like how much the combinations of neurons matter versus just one at a time.”
Cammarata says his blog post landed at a time when people are looking for hope. In particular, he thinks the AI safety community—the researchers and policymakers most worried that powerful models will bring huge risks—needs a lifeline right now. They think that things are looking bad and that even if things do turn out okay, it will be because we got lucky, not because we fixed anything, he says: “I don’t really hear a lot of optimism.”
That’s why Cammarata thinks what he has to say matters. The job of figuring out how this technology works, and thus how to control it, has barely begun. But at least it’s not a nonstarter. “I think our job is to sort of try to think like the neural network,” he says. “My guess is that things will turn out pretty well, because it was in some sense designed to be read.”
Ten years after AlphaGo’s match against Go champion Lee Sedol, today’s AI still isn’t tapping into the machinery that made that win possible.
Breathless claims about AGI and new capabilities fall apart pretty quickly under scrutiny.
The underlying models had been rewarded for cheating and communicating with each other, a new OpenAI report finds.
In a new interview,the billionaire philanthropistsounds an alarm on the urgency of getting our AI policies in order.