> blog
Longer-form writing on LLM evaluation, benchmarks, and applied agent work.
Before You Score the Model, Score the Benchmark
A Skeptical View Into Current Agentic Software Engineering Benchmarks
Argues that dataset-side noise often dominates model-side signal in agentic SWE evaluation, and that the field doesn't audit its benchmarks nearly enough. Includes the FeatBench-Verified case study, where gold-patch resolution lifted from 37.31% (±1.88) to 99.81% (±0.10) after diagnosing harness and environment defects.