Delta Attention Residuals

4 atoms · 跨 1 天 · 首见 2026-05-20 · 最近 2026-05-20

三色: 🟦 fact 0 · 🟥 take 4 · stance ▲4/▼0/◆0

来源: X 4

时态: fresh:4

标签: 好数字:4

事实 (fact) (4)

• 2026-05-20 · sharper cross-layer routing
  Delta Attention Residuals lift max attention weight from ~0.2 → ~0.6 (0.62 vs 0.35 avg) and curing routing collapse in deep layers.
  tags: 好数字 · → daily
  value: qty=max attention weight from ~0.2 → ~0.6 · avg=0.62 vs 0.35
• 2026-05-20 · validation PPL reduction at 7.6B scale
  −𝟖.𝟐% 𝐯𝐚𝐥𝐢𝐝𝐚𝐭𝐢𝐨𝐧 𝐏𝐏𝐋 𝐚𝐭 𝟕.𝟔𝐁
  tags: 好数字 · → daily
  value: qty=-8.2% · scale=7.6B
• 2026-05-20 · parameter overhead
  ≤𝟎.𝟎𝟏% 𝐩𝐚𝐫𝐚𝐦𝐞𝐭𝐞𝐫 𝐨𝐯𝐞𝐫𝐡𝐞𝐚𝐝
  tags: 好数字 · → daily
  value: qty=≤0.01%
• 2026-05-20 · throughput compared to Attention Residuals
  runs faster + lighter than Attention Residuals (14.0k vs 12.5k tok/s, 42.7 vs 44.0 GB)
  tags: 好数字 · → daily
  value: qty=14.0k vs 12.5k tok/s

时间轴 (近 20)

• 2026-05-20 · fact · sharper cross-layer routing · → daily
• 2026-05-20 · fact · validation PPL reduction at 7.6B scale · → daily
• 2026-05-20 · fact · parameter overhead · → daily
• 2026-05-20 · fact · throughput compared to Attention Residuals · → daily

← 实体目录 · 系统日志