How Far Can Reading AI’s Neurons Take Us?

Caught My Eye…

1) How Far Can Reading AI’s Neurons Take Us?

AI models are taking on more important work, but we still struggle to explain how they reach a particular answer. We can see a mistake when it appears in the output, but seeing what produced it inside the model is much harder. For years, researchers thought the inner workings were too tangled to study directly.

On Oct. 15, the AI startup Goodfire published "Neural Networks Are Already Clean," a 65-minute essay by its researcher Nick Cammarata. Goodfire, valued at $1.25 billion in February 2026, sells software built on interpretability. Anthropic, whose research the essay re-examines, invested in Goodfire in 2025.

Sam Altman responded favorably to the essay, and many AI researchers got excited by the progress. Some commentators even saw this as a green light to stop pacing the frontier, while others still caution how far to take the findings.

To see what Cammarata and other researchers thought needed untangling, let's start with the basics. A neural network is a program made of many connected units that learns from examples. It might learn to recognize a car in a photo or predict the next word in a sentence. To do that, it processes information in stages called layers. Each layer contains neurons that combine signals from earlier stages and pass new signals onward. In an image model, one layer might pick up edges, another might use them to find wheels, and a later one might help identify a car. Each connection between neurons has a weight, a number that determines how strongly one neuron’s signal influences another. Training changes these weights. They make up most of the model’s parameters, so a model with two trillion parameters has roughly two trillion learned numbers shaping how its parts work together. A circuit is a set of connected neurons that work together to help the model do something, like identify a car. Interpretability tries to trace those circuits to explain how the model reached an answer.

For years, many neurons looked polysemantic, firing on unrelated things. A famous example came from InceptionV1, a 2014 image model. One of its neurons became active for both cat faces and car fronts.

The leading explanation was superposition: a model short on neurons packs unrelated concepts into the same neuron, which works because those concepts rarely come up at the same time. If that were common, you would have to untangle a model before reading it. A standard untangling tool became the sparse autoencoder, a second model that splits the first one's activity into cleaner pieces.

In 2020, Cammarata co-wrote Zoom In, the paper that brought superposition into circuits research, using a circuit he found and read with Chris Olah, who later co-founded Anthropic as their Interpretability Research Lead. He now says they misread that circuit.

He agrees a model holds more concepts than it has neurons, because neurons combine, the way car-part neurons build a car neuron. What he disputes is the packing of unrelated concepts. He argues that most of the tangle in early models came from ordinary, mostly fixable problems.

In the 2019 circuit, a neuron that detected cars sent a signal to four other neurons. Researchers used software to generate an image that would activate each of those four neurons. The images looked like dogs, so they thought the model had hidden car information in dog neurons. Cammarata now says they misread them. On a closer look, two are car neurons, and two group cars and dogs by pose, the direction an object faces in the photo.

The cat-and-car neuron was another apparent mystery. It became active for both cat faces and car fronts, which seemed unrelated. Cammarata noticed the shape they share: the pointed V highlighted in the image. Other pointed shapes also make the neuron more active, while round shapes make it less active. He argues it is detecting that shape across different objects, rather than doing separate jobs for cats and cars.

The broader insight is the essay’s tip for reading a neuron. Humans, and AI labelers too, tend to describe a neuron as a list of nouns. Cammarata suggests asking: if I couldn’t use groups of nouns at all, how would I describe this neuron? Often a shared pattern pops out, like the V. In his framing, a neuron works less like a label for a thing and more like a sorter: pointed one way, round the other, so later neurons can use the split. If this pattern holds widely, researchers can study the model’s own neurons and circuits directly, rather than first training another model to untangle them.

Cammarata calls this a possible "bitter lesson" for interpretability. Rich Sutton used that phrase in 2019 to argue that, in AI, general methods that use more computation beat clever hand-built ones by a large margin. Cammarata suggests interpretability may face the same lesson: ordinary checks run at huge scale may beat building clever untangling tools. Even if it's speculative, he believes that interpretability might be far more tractable, possibly in the near term.

Neel Nanda, who leads interpretability at Google DeepMind, doesn't think it's a significant update to the big picture. Eliezer Yudkowsky, a leading voice on AI risk, objects at a different level. Even if every neuron is clean, he argues a clean neuron only tells you what an AI is thinking about. It doesn’t tell you how the AI reasons, so this finding is still far from aligning models, especially as they become more capable. In his words, alignment today means “hammering on an AI from the outside with SFT and RL,” the two main training methods. SFT, or supervised fine-tuning, shows a model examples of good answers. RL, or reinforcement learning, rewards the answers you want. Real progress, in his view, would mean understanding an AI well enough to predict its thinking before it happens, even when that thinking goes beyond human ideas.

For example, engineers can tell you how a bridge will behave from its design, without building a hundred bridges and watching which fall, and interpretability today works by matching neurons to ideas we already have: cat, car, Colorado, a V shape. A smarter-than-human AI will matter most when it uses ideas we don’t have. His example: spotting that an AI is thinking about Navier-Stokes is very different from understanding the part where the AI can prove Navier-Stokes, because no human knows what kind of thought processes would be required.

Eliezer’s critique was aimed at the hype. Roon from OpenAI wrote that the news was “so good even Eliezer will have to lower his pdoom.” Yudkowsky answered with an analogy. A team wants to pry the panel off a computer to see how it works. Their screwdriver breaks. Years later they announce, “Awesome news, the screwdriver is repaired now!” The tool works again, but the hard part, understanding what’s inside, hasn’t started. His objection was to treating this advance as evidence that the harder work of understanding and aligning AI is nearly done.

2) The Market Puts CoreWeave’s Default Odds Near 60%

On October 14, CoreWeave's stock fell 15.5%, and the next day its $1.25 billion of 9.625% bonds due 2032 traded at 77 cents on the dollar, almost a 16% annual return for a buyer who holds to 2032 and is paid in full.

CoreWeave rents Nvidia chips to AI labs and tech companies and pays for them mostly with borrowed money, $35.6 billion at the end of June. Insurance against a CoreWeave default hit a record price on October 15, and if lenders would recover about 40 cents on the dollar, that price implies roughly a 60% chance of default within five years.

CoreWeave still pays its 9.625% coupon, and across every loan in its second-quarter filing, weighted by size, the average rate is about 8.4%. Its data center loans are secured by a customer contract and the chips that serve it and repaid from that contract's cash, so lenders price each one off the customer. A March loan backed by one customer's contract won an investment-grade rating and fixed part of its rate at about 5.9%, while a May loan backed by two large, non-investment-grade customers costs about 9%. But the filing shows about 89% of CoreWeave's debt is owed or guaranteed by the parent company, so a loan that comes up short lands on CoreWeave itself, and its June balance sheet carried $77 billion of assets against $72 billion of total liabilities, leaving $5 billion of shareholders' equity.

Pressure is starting outside CoreWeave, with the 10-year Treasury yield climbing. The junkiest borrowers pay more, and CoreWeave's cost has moved with them. Take-or-pay contracts with set terms brought in 98% of its revenue last quarter, so a higher interest bill comes out of revenue it can't raise, and with only 1.5 of the 3.7 gigawatts of power it has lined up now running, much of the financing is still ahead of it. Higher borrowing costs erode the margin, the default insurance reacts, and then the bonds react, a negative reflexive loop.

3) Producer Prices Just Jumped the Most Since 2022

On October 14, the Labor Department reported (CPI) consumer prices 3.7% higher than a year earlier. Energy, up 20.4%, accounts for about 1.3 percentage points of that, based on energy’s share of household spending in the index. A day later, it reported that producer prices (PPI), what US businesses receive for the goods and services they sell, rose 1.3% in September alone, the biggest monthly jump since March 2022. Core inflation, which leaves out food and energy, ran at 2.6% over the past year, up from 2.4% in August.

Refined fuels are tighter than crude oil. In September, Gulf exports of refined fuels ran at 60% of their pre-Iran-conflict level, versus 91% for crude. Producer prices for transportation and warehousing rose 3.4% in September, and the Fed's Beige Book, a roundup of reports from business contacts in its 12 districts, says fuel surcharges passed on higher costs throughout the supply chain. The Beige Book says consumer-facing firms report tighter margins as price-sensitive customers continue to resist price hikes, and firms in several districts expect prices to rise further as existing contracts expire.

When the Fed raised its rate to 3.75%–4.00% in September, most officials expected another increase by year-end, according to the meeting's minutes. A couple said a higher rate would keep price increases from energy disruptions and AI-related demand from broadening out, and many warned that the longer energy stays expensive, the greater the risk of broader price pressures.