Recent advancements in artificial intelligence are shedding light on the inner workings of complex models, according to research conducted by Anthropic. Traditionally viewed as "black boxes," AI models like ChatGPT and Claude operate based on extensive training data, but the underlying processes have remained largely obscure. This research aims to change that by utilizing a tool called the Jacobian lens, or J-lens, which allows scientists to observe how these AI systems devise solutions in real-time.
In a notable experiment, Claude was presented with the prompt, "The number of legs on the animal that spins webs is…" Through the J-lens, researchers discovered that the model conceptualized the word "spider" internally, despite it not being included in the prompt. The model then correctly answered "eight," demonstrating its ability to infer information and reason through problems.
Further tests revealed that Claude’s internal reasoning could be manipulated. Researchers found that by altering the internal concept from "spider" to "ant," Claude adjusted its answer from eight to six. In another instance, while composing a rhyming couplet, Claude appeared to plan its response ahead of time, challenging assumptions about how AI generates text.
These findings suggest that AI models engage in more complex cognitive processes than simply regurgitating data. While critics often label such systems as "stochastic parrots," this research indicates they can construct and revise internal ideas before arriving at a conclusion.
Moreover, understanding the hidden activities in AI could be crucial, especially as models increasingly face threats from prompt injection attacks. By detecting these malicious instructions during processing, researchers may enhance the security and reliability of AI systems.
Why this story matters
Key takeaway
Opposing viewpoint
