以下部分引用 AI 总结，现阶段 AI 仍然有幻觉。请以内容的原文为准。

数据新鲜度群聊 07-14 ✓卖方 07-14 ✓Wrap 断档15天 ✕News 07-15 ✓X 断档12天 ✕页面生成 07-15

MoE architecture

5 atoms · 跨 2 天 · 首见 2026-04-20 · 最近 2026-05-20

三色: 🟦 fact 4 · 🟥 take 1 · stance ▲0/▼1/◆4

来源: X 5

时态: fresh:5

标签: 好信源:3 · 好思考:2

2026-05-20 · 标准 MoE 输出公式
_(routed * route_scale) + shared, route_scale is a hyperparameter_
tags: 好信源 · → daily
2026-05-20 · route_scale 典型值范围
_generally route_scale is >1 so you'd have routed experts contribute more_
tags: 好信源 · → daily
2026-05-20 · MoE 架构定义
MoEs aren't a bunch of individual models, they only replace MLP layers
tags: 好信源 · → daily
2026-05-20 · mean pooling 与加权求和等价性
if each output was 2x as large it's the same result. x + y = 2((x+y)/2), holds for all x and y, so it's equally expressive
tags: 好思考 · → daily
2026-04-20 · context scaling limitation
moe doesn't scale with context
tags: 好思考 · → daily
value: direction=does not scale