Artificial Intelligence1 min read

Memdoor: A team of 5 Qwen3.8-27B models beats Fable 5 on LiveCodeBench Hard

By · Published by Everything Blog

In short

The rebuilt setup achieved a 9.8 out of 10 mean score across the first ten LiveCodeBench Hard problems, successfully solving 49 out of 50 total runs. To coordinate without shared memory, the agents launch with fresh contexts, interact entirely via disk files, and rely on shell scripts for test verification. Operating through OpenRouter, the five-pass evaluation consumed 15.3 million output tokens at a core compute cost of $35.79. Rare failures during the runs occurred when manager agents generated faulty counterexamples or accidentally overwrote completed code.

Key points

  • Orchestrating multiple smaller models into specialized functional roles can surpass the p…: Orchestrating multiple smaller models into specialized functional roles can surpass the performance of single leading systems.
  • Relying on disk-based communication and automated test scripts allows agents to coordinat…: Relying on disk-based communication and automated test scripts allows agents to coordinate without sharing internal context.
  • Potential training data contamination remains an open concern because the benchmark probl…: Potential training data contamination remains an open concern because the benchmark problems originated between March and April 2025.

Sources checked by our analysis pipeline: 1

A multi-agent harness for programming problems, written as a Memdoor workflow. Five roles (planner, two ideators, solver, manager) and a repair worker each start from a fresh context and talk only through files on disk. A shell script that runs the samples decides success, not a model. It is the method of GVS5H (Lee, 2026: five Qwen3.8-27B models match Fable 5 on LiveCodeBench Hard).

Rebuilt with the authors' exact prompts on Qwen3.8-27B, their loop scored a mean of 9.8 of 10 over five passes on the first ten LiveCodeBench Hard problems, against 9.6 for their own harness and 8.0 for Fable 5 alone (below, every run published).

The coder wrote these files itself from a description of the method (2026-10-25). Nobody hand-wrote them.

GVS5H's own harness, rebuilt as a Memdoor workflow on the same model: Qwen3.8-27B, a 128K reply cap, the authors' prompts and loop. The first ten problems of their LiveCodeBench Hard list, five passes, every program graded on every public and hidden test with their corrected grader.

pass | Memdoor workflow | authors' 27B harness | Fable 5…

Artificial Intelligence