以下部分引用 AI 总结，现阶段 AI 仍然有幻觉。请以内容的原文为准。

数据新鲜度群聊 07-14 ✓卖方 07-14 ✓Wrap 断档15天 ✕News 07-15 ✓X 断档12天 ✕页面生成 07-15

SWE-bench

9 atoms · 跨 3 天 · 首见 2026-05-06 · 最近 2026-05-30

三色: 🟦 fact 6 · 🟥 take 3 · stance ▲3/▼1/◆2

来源: X 9

时态: fresh:8 · aging:1

标签: 好观点:4 · 好数字:3 · 好信源:3

2026-05-30 · 替代方法：自建bug
For SWE-bench, another approach could've been to make our own bugs+fixes in repos we would have written- that would have been so artificial, of such lower quality, and also probably much easier to solve
tags: 好观点 · → daily
value: direction=artificial_and_lower_quality
2026-05-06 [老化中] · 自动化缺陷修复和小功能需求
Benchmarks aren't just about showing where we are now, benchmarks are a treasure map that shows us how to get to the future the benchmark specifies. SWE-bench was about automating bugfixing and small feature reqs, this is about automating entire repo development.
tags: 好观点 · → daily

2026-05-06 · cost with proper harness and Sonnet
with a proper harness + prompt it will be 10-15K for sonnet
tags: 好数字 · → daily
value: qty=10-15K
2026-05-06 · cost expected to decrease quickly
This is an expensive benchmark, but costs will go down quickly.
tags: 好观点 · → daily

2026-05-06 · evaluation cost reduction trend
Cost for evaluating on SWE-bench went down so much (50% for some $10 now!!)
tags: 好数字·好观点 · → daily
value: qty=50% for some $10 now
2026-05-30 · 数据来源类型
SWE-bench as the first category (finding it in the real world)
tags: 好信源 · → daily
value: direction=real_world_code_mined
2026-05-30 · 数据来源模式
SWE-bench showed the eval+train potential of GitHub issues
tags: 好信源 · → daily
value: description=revealed potential of GitHub issues
2026-05-28 · 当前解决状态
SWE-bench still unsolved!
tags: 好信源 · → daily
value: direction=unsolved
2026-05-06 · max run cost with Sonnet
Our most expensive run was 5k with Sonnet
tags: 好数字 · → daily
value: qty=5k