A Small Agent Benchmark
August 28, 2026
I tried Qwen3 using an open-source harness. It felt like it was 6–12 months behind the frontier models, which is a pretty big difference.
I thought this might be an interesting benchmark: retrieve the BibTeX entry for a paper. In the first video, the open-source harness is on the left and Codex is on the right.
The frontier model finishes the job in about one minute. The open-source version needs follow-up prompts, encounters errors, and ultimately fails.
I was able to finish the job manually in 30 seconds 😄