Artificial Intelligence1 min read

We tried “Claude, fix this bug”—here’s what actually worked

By · Published by Everything Blog

In short

Duolingo created AI STR Fix using the Claude Agent SDK to autonomously investigate software defects submitted by admin and beta testers. The system deploys Claude Sonnet for basic tasks and reserves Claude Opus for more complex operational challenges. To keep costs low, heuristic pre-filters clean incoming data by dropping irrelevant A/B experiments and prioritizing errors closest to the incident report. The resulting context is posted directly into Jira so engineers can bypass manual research and implement fixes faster.

Key points

  • AI STR Fix automates time-consuming technical investigations instead of attempting to gen…: AI STR Fix automates time-consuming technical investigations instead of attempting to generate direct, unassisted code patches.
  • Built-in heuristic pre-filters reduce token expenditures by stripping away repetitive act…: Built-in heuristic pre-filters reduce token expenditures by stripping away repetitive activity logs and irrelevant A/B experiment data.
  • Enriched debugging insights are posted directly to Jira tickets, sharply reducing the man…: Enriched debugging insights are posted directly to Jira tickets, sharply reducing the manual triage required from software engineers.

Sources checked by our analysis pipeline: 1

Reporting bugs is fast, but fixing them was slow. At Duolingo, admin and beta users can shake their phone to open a bug report form. We call these Shake-to-Report, or STR, bugs. Reports submitted by admin users automatically become Jira tickets for engineers to investigate. These STR Jira tickets include lots of key debugging information like a screenshot, logs, device metadata, and more. With more tickets coming in than we could resolve, our bug backlog was ever growing.

Insight #1: We need help… so let’s throw AI at it

I’m on the DevX AI team, whose mission is to accelerate developer productivity using AI, so of course we decided to throw AI at it. We started with a simple prompt: “Claude, fix this bug.”

It turns out there’s a lot more to fixing a bug than just making the code change. Engineers first need to piece together what happened. That might mean digging through logs, reconstructing the bug reporter’s activity, checking which A/B experiments were enabled, searching through the codebase for the relevant code, grabbing a device and trying to…

Artificial Intelligence