以下部分引用 AI 总结，现阶段 AI 仍然有幻觉。请以内容的原文为准。
数据新鲜度群聊 07-14 ✓卖方 07-14 ✓Wrap 断档15天 ✕News 07-15 ✓X 断档12天 ✕页面生成 07-15
Muon
18 atoms · 跨 6 天 · 首见 2026-05-15 · 最近 2026-07-01
三色: 🟦 fact 5 · 🟥 take 13 · stance ▲3/▼2/◆10
来源: X 18
时态: fresh:17 · stale:1
标签: 好思考:10 · 好观点:7 · 好数字:3 · 好信源:3 · 好问题:1
叙事 (narrative) (11)
- 2026-06-10 · 对 epsilon 选择极为敏感
beyond LR and weight decay, they are extremely sensitive to the various epsilon choices
tags: 好观点·好思考 · → daily
- 2026-06-10 · 与 SOAP-like 排名可因 epsilon 微小改变而翻转
A tiny change to eps was enough to flip the ranking between SOAP-like and Muon
tags: 好观点·好思考 · → daily
- 2026-06-10 · 与 Shampoo/SOAP-like 排名可因不同非确定性训练运行而翻转
Different nondeterministic training runs could also flip the ranking between Shampoo/SOAP-like and Muon
tags: 好观点·好思考 · → daily
- 2026-06-10 · 从许多理论角度看像是 Shampoo/SOAP 的削弱版
from many theoretical angles, Muon looks like a nerfed version of Shampoo/SOAP
tags: 好观点·好思考 · → daily
- 2026-05-16 · 学习率推荐值
lr should be 5x / 10x of adamw (if you're not rescaling a la Kimi, and even in that case, you should still tune lr)
tags: 好数字·好思考 · → daily
value: qty=5x / 10x of AdamW
- 2026-05-16 · 超参数调优必要性
you cannot expect optimizer changes to work without tuning lr, that'd be purely relying on luck
tags: 好思考 · → daily
value: direction=requires tuning lr
- 2026-05-15 · update is always of a constant size
Muon update is always of a constant size, it has nothing to dampen it. It has the same problem as signgd.
tags: 好思考 · → daily
- 2026-05-15 · cannot attain exact convergence without a learning rate scheduler
it seems like it can't attain exact convergence even after infinite time without a learning rate scheduler, only epsilon convergence at best
tags: 好思考 · → daily
- 2026-05-15 · erratic behavior around gradient~0 regions is a feature, not a bug
tags: 好思考 · → daily
- 2026-06-09 · introduced several innovations
Muon also introduced several innovations and it may be an over-simplification to say that it's just Shampoo with b2=0.
tags: 好观点 · → daily
- 2026-05-15 · 在扩散模型中的训练速度
Muon is slower and lags behind in the long run
tags: 好观点 · → daily
value: direction=slower than AdamW
预测 (forecast) (1)
- 2026-06-10 · 在其他任务上可能弱于 Shampoo/SOAP-like 方法
On other tasks, Shampoo/SOAP-like methods may show a much cleaner advantage over Muon
tags: 好观点·好思考 · → daily
立场 (position) (1)
展开 1 条老旧 / 已过期
- 2026-05-20
[老旧] · optimizer adoption status
Is Muon the first optimizer in a while that actually made people abandon AdamW?
tags: 好问题 · → daily
事实 (fact) (5)
- 2026-07-01 · FLOP 利用率提升
Muon - by not treating updates to different weights as independent, and treating them as a component of a matrix - improved FLOP utilisation by 100%.
tags: 好数字 · → daily
value: qty=100%
- 2026-07-01 · 达到同样 loss 所需 FLOPs 减少
You reach the same loss in less than 50% FLOPs.
tags: 好数字 · → daily
value: qty=<50%
- 2026-06-09 · use of Nesterov momentum
For one the use of Nesterov momentum
tags: 好信源 · → daily
- 2026-06-09 · use of Newton-Schulz which makes it more GPU friendly
Clever use of Newton-Schulz which makes it more GPU friendly than eigh or QR or coupled newton
tags: 好信源 · → daily
- 2026-06-09 · scaling the update rms norm to match 0.2
Scaling the update rms norm to match 0.2 (Adam update rms in the limit) which made grafting with Adam unnecessary
tags: 好信源 · → daily
value: qty=0.2
时间轴 (近 20)
- 2026-07-01 ·
fact · FLOP 利用率提升 · → daily
- 2026-07-01 ·
fact · 达到同样 loss 所需 FLOPs 减少 · → daily
- 2026-06-10 ·
narrative · 对 epsilon 选择极为敏感 · → daily
- 2026-06-10 ·
narrative · 与 SOAP-like 排名可因 epsilon 微小改变而翻转 · → daily
- 2026-06-10 ·
narrative · 与 Shampoo/SOAP-like 排名可因不同非确定性训练运行而翻转 · → daily
- 2026-06-10 ·
narrative · 从许多理论角度看像是 Shampoo/SOAP 的削弱版 · → daily
- 2026-06-10 ·
forecast · 在其他任务上可能弱于 Shampoo/SOAP-like 方法 · → daily
- 2026-06-09 ·
narrative · introduced several innovations · → daily
- 2026-06-09 ·
fact · use of Nesterov momentum · → daily
- 2026-06-09 ·
fact · use of Newton-Schulz which makes it more GPU friendly · → daily
- 2026-06-09 ·
fact · scaling the update rms norm to match 0.2 · → daily
- 2026-05-20 ·
position · optimizer adoption status · → daily
- 2026-05-16 ·
narrative · 超参数调优必要性 · → daily
- 2026-05-16 ·
narrative · 学习率推荐值 · → daily
- 2026-05-15 ·
narrative · 在扩散模型中的训练速度 · → daily
- 2026-05-15 ·
narrative · update is always of a constant size · → daily
- 2026-05-15 ·
narrative · cannot attain exact convergence without a learning rate sche · → daily
- 2026-05-15 ·
narrative · erratic behavior around gradient~0 regions is a feature, not · → daily
← 实体目录 · 系统日志