Looking inside the model is a breakthrough, with an asterisk

Every year MIT Technology Review picks ten breakthrough technologies. For 2026 one of them is mechanistic interpretability: tools that let researchers see what is happening inside an AI model instead of only judging its outputs (MIT Technology Review).

What “looking inside” means

A language model is billions of numbers. The idea is to find structure in them that maps to concepts a human recognises.

• In 2024 Anthropic built what it called a microscope for its model Claude and found internal features tied to recognisable things, like Michael Jordan or the Golden Gate Bridge (same source).
• In 2025 it went further and traced whole sequences of features, following how the model gets from a prompt to an answer.
• OpenAI and Google DeepMind have used similar techniques to investigate odd behaviours, such as models that seem to deceive the people using them.

A related, simpler tool is reading a reasoning model’s step-by-step chain of thought. OpenAI used it to catch a model cheating on coding tests (same source).

The asterisk

The field is split on how far this goes. Some researchers think these models may be too complicated to ever be fully understood, and the article itself frames the tools as something that “could help”, not a solved problem.

Recent preprints show why it’s hard. One asks whether models that reason in hidden vectors instead of words are interpretable at all, and finds that on logic datasets they can often reach the same answers without using that hidden reasoning (arXiv 2604.04902). Another argues interpretability should be judged by whether its findings lead to concrete decisions and fixes, not by how compelling the pictures look (arXiv 2605.11161). Both are preprints, not peer-reviewed results.

Why I care

In medical imaging you can’t ship a model nobody can question. A saliency map that looks convincing isn’t an explanation, and a clinician will ask what the model actually used. Interpretability that leads to an action, “this feature fires on the scanner artefact, remove it”, is the kind that earns trust. The pretty pictures are the easy part.