Anthropic Study Reveals AI Coding Tools May Hinder Developer Skill Acquisition

A new study from Anthropic, titled “How AI Impacts Skill Formation,” has provided nuanced insights into the role of AI coding assistants in developer learning. The research involved 52 mostly junior Python engineers tasked with building projects, with some utilizing an AI assistant and others working without. While the AI-assisted group completed tasks marginally faster (approximately two minutes), this difference was not statistically significant. More notably, participants using AI scored significantly lower on a post-task quiz, averaging 50% compared to 67% for the manual coding group—a gap equivalent to nearly two letter grades. The most pronounced discrepancies were observed in debugging questions, raising concerns about AI’s potential to impede the development of critical problem-solving skills. Interestingly, the study noted that a considerable portion of AI users spent time manually retyping AI-generated code, rather than copying, an interaction pattern that, when accounted for, suggested a 25% speed improvement for other AI users.

The study also categorized AI interaction patterns, linking “AI delegation” (heavy reliance on AI for code generation and debugging) to the lowest quiz scores, indicating cognitive offloading. Conversely, patterns like “generation then comprehension” (generating code and then using AI for follow-up explanations) and “conceptual inquiry” (focusing solely on conceptual questions) correlated with higher comprehension, albeit these groups were notably small. Industry reactions have highlighted limitations in the study’s design, including the short task duration and the use of unfamiliar environments by junior developers, arguing these factors may not fully reflect real-world productivity gains often seen in experienced developers. Indeed, Anthropic’s own conclusion differentiates these findings from prior work showing up to 80% productivity increases in familiar tasks, suggesting AI’s impact may vary between accelerating existing skills and hindering new skill acquisition. The discussion extends to AI’s potential as a motivational tool, providing early successes to prevent burnout among new learners, with some educators, like the creator of HTMX.org, advocating for AI agents to function as teaching assistants rather than direct code generators. The research underscores the ongoing need for a balanced approach to AI integration in software development and education.