Skip to content
Back/AI Ecosystem

LLM Inference Mechanics

View in Graph
Updated 2026-10-28
4 min read
936 words

LLM Inference Mechanics

What it is

An overview of how large language models execute inference efficiently, covering the full stack from hardware to software optimization. Based on a systematic technical summary of AI infrastructure for beginners.

Why it matters

推理不是“训练结束后的部署细节”,而是决定 Agent 产品是否能在可接受的延迟、成本和可靠性下运行的系统工程。优化目标也不是单一峰值 FLOPS,而是每个现实任务的等待时间、吞吐、能源、缓存命中、失败恢复和单位价值成本。

Key concepts

1. Model Parallelism Strategies

Strategy Mechanism When to use
Tensor Parallelism (TP) Split individual layers across multiple GPUs Single node, large models that don't fit on one GPU
Pipeline Parallelism (PP) Split model into stages, each on a different GPU Very deep models, cross-node deployment
Sequence Parallelism (SP) Split sequence dimension across GPUs Long-context inference
Expert Parallelism (EP) Route tokens to different expert GPUs MoE models
Context Parallelism (CP) Distribute attention computation across sequence length Ultra-long context

2. Memory Optimization

  • Weight quantization: FP16 → INT8/INT4 reduces memory by 2-4× with acceptable accuracy loss
  • KV-cache management: The key bottleneck in autoregressive generation. Optimizations include:
    • PagedAttention (vLLM): Treat KV cache as non-contiguous memory blocks, eliminating fragmentation
    • Dynamic memory allocation rather than pre-allocating max context size
  • Offloading: Move less-active layers to CPU/SSD when GPU memory is constrained

3. Serving System Architecture

  • Continuous batching (inflight batching): Unlike static batching, new requests join an ongoing batch as soon as a slot frees up—dramatically improving GPU utilization
  • Speculative decoding: A smaller draft model generates candidate tokens; the large model verifies them in parallel. 2-3× speedup when acceptance rate is high
  • Prefix caching: Cache the KV vectors of common prompt prefixes (system prompts, few-shot examples) to avoid recomputation
  • Disaggregated serving: Separate prefill (compute-intensive, parallelizable) and decode (memory-bandwidth-bound, sequential) phases onto different GPU pools

4. Request Scheduling

  • Shortest-job-first with preemption: Prioritize requests that will finish quickly; preempt long-running requests if needed
  • Chunked prefill: Break long prefills into chunks to avoid starving short requests
  • Load balancing: Route requests based on current queue depth and KV-cache pressure

5. vLLM Architecture

vLLM represents a generation shift in inference serving:

  • PagedAttention: Borrowed from OS virtual memory—treat KV cache as pages that can be allocated/freed dynamically
  • Copy-on-write: When a request forks (e.g., beam search), share KV cache pages until one branch modifies them
  • Continuous batching: Dynamic batching with iteration-level scheduling

Evidence across sources

Source Durable evidence Implication
EP668 推理与网络 缓存感知路由、prefill/decode 分离、投机解码、量化和 KV cache 需要联合优化 推理性能来自请求路径,不是单一内核
EP646 Cerebras 多轮 Agent 交互让每用户每秒 token 和尾延迟成为体验指标;SRAM/HBM/软件生态共同决定结果 “硬件护城河”必须按完整栈评估
EP609 Etched 专用推理芯片仍受内存带宽、互联、低电压、封装和可制造性约束 峰值性能不能替代量产与供应链证据
EP607 软硬件协同 实时推理基准需要同时比较芯片、软件栈和数据中心运营 Benchmark 的测量边界决定结论
EP623 基础设施 GPU、网络、电力、传输和转换会轮流成为瓶颈 单位成本下降会把需求推向新的约束层
EP633 体系结构 功耗、存储层级、领域专用架构、编译器和库共同决定实际性能 CUDA 等软件生态是架构的一部分
EP614 模型效率 MoE、量化、上下文压缩、多 token 预测和蒸馏共同降低推理成本 模型结构与 serving 系统不可分开优化
EP595 AI 云 电力、土地、运维、软件和安全生产部署能力都可能比 GPU 数量更稀缺 Capacity 是多资源组合,而非单一库存
EP620 Agent 云 突发负载、GPU snapshot、沙箱、持久文件和区域调度改变 Agent workload 的 serving 需求 通用云原语未必直接适配 Agent
EP675 AI 超级周期经济学 从能源到应用的每一层都要回答资本开支如何转成可付费价值 推理经济必须连接到应用收入,而非只看 token 数

Durable synthesis

  • 瓶颈会迁移:算力增加后,约束会移动到内存、网络、电力、排队、软件栈、封装、运维或应用需求。
  • 优化单位是任务:有用指标是完成一个真实任务所需的延迟、成本、可靠性和能耗,而非孤立 token/s 或峰值 FLOPS。
  • Agent 放大尾延迟:一次工作包含多轮模型调用和工具调用,单步慢一点会在整条链路中累积;缓存、路由和恢复因此比单次基准更重要。
  • 软硬件共同构成护城河:专用芯片可能在固定 workload 上有优势,但可编程性、编译器、库、供应链和客户迁移成本决定优势能否持续。
  • 便宜不等于需求封顶:单位推理成本下降可能通过更长上下文、更多 Agent 步骤和新模态扩大总需求,形成新的容量压力。

Counterpoints & Gaps

  • 多期节目包含芯片厂商、云厂商和投资人的利益相关叙事;性能、融资和产能数字没有在本轮逐项核验。
  • 访谈中的峰值指标通常缺少工作负载、批大小、精度、功耗和尾延迟等比较条件,不能直接横向排名。
  • 专用硬件对未来模型结构的适应性仍不确定;当前 Transformer workload 的优势未必能跨架构延续。
  • 本批逐字稿是 AI 翻译与声纹克隆后的二手材料,应与一手技术文档、benchmark 方法和生产 trace 交叉验证。

Open questions

  • At what context length does disaggregated prefill/decode become necessary?
  • How do quantization-aware training and post-training quantization compare for production accuracy?
  • Will hardware specialization (TPU, Groq) change these software optimizations, or do the principles transfer?

2026-10:推理工程纪律化——从 7B 扛 1M token 到 GPU 记账

多个 10 月中旬信源指向同一变化:模型公司开始把推理优化当作核心能力公开披露。

  • AI Briefing 2026-10-07:MiniMax M3 用 lightning attention(线性注意力)让 7B 模型原生支持 100 万 token 上下文——架构选择直接决定长上下文成本曲线,长上下文从「旗舰模型特权」变成「小模型架构即可提供」。
  • AI Briefing 2026-10-12:Anthropic 内部 SRE 团队为 Claude Opus 4.7 做推理层复盘,定位到 GPU 层面的硬件瓶颈。头部实验室已把推理 infra 的问题深入到硬件记账粒度。
  • AI Briefing 2026-10-15:Kimi K3 推理优化文章披露 GB300 NVL72 机架部署的 GPU 分配机制。国产模型的推理工程披露深度已可对标一线实验室。

含义:竞争焦点从「谁的模型强」转向「谁的 GPU 记账更细、架构更省」。与本页 inference mechanics 主题互补:机制理解开始直接转化为部署成本优势。

2026-09-19:扩散式语言模型把推理优化推进到模型架构层

来源:AI Briefing 2026-09-19 Afternoon

No Priors 对 Inception CEO Stefano Ermon 的访谈把扩散式语言模型描述为并行生成路线:模型不是严格逐 token 自回归,而是并行修正候选序列。Inception 将 Mercury、后训练与自建 serving stack 放在一起,说明延迟、吞吐和 intelligence per dollar/watt 不能只由模型架构推导;内核、调度、批处理、后训练和实际 workload 共同决定结果。

这给现有推理框架增加一个架构层变量:比较系统时要记录生成范式、质量约束、尾延迟、能耗、批处理条件和完整 serving stack,而不能把并行生成直接等同于端到端更快。

Counterpoints & Gaps

  • 访谈来自公司创始人与播客,Mercury 的速度和效率主张未在本轮用独立 benchmark 复现。
  • 并行生成在长文本一致性、工具调用、结构化输出和复杂推理上的质量边界仍需逐项比较。

Sources (addendum)

Pending Jean review

  • ai-ecosystem/llm-inference-mechanics — ai_draft; compare one fixed workload across generation paradigm, tail latency, quality, energy, batch conditions, and total serving cost.

2026-10-22:DeepSeek DeepInfer 作为生产参考实现

来源:AINews 2026-10-22 — DeepSeek DeepInfer 开源推理栈

DeepSeek 开源的 DeepInfer(基于 vLLM 定制,Apache 2.0)把本页多个机制的规模化生产形态一次性公开:

  • PD 分离的部署形态:prefill 用普通 8 卡机,decode 用 64–128 卡超节点加宽专家并行(wide EP)分发 MoE experts,保持大有效 batch。
  • 按算力预算调度,而非按 token 数:prefill 调度估算 attention 计算成本分配工作,chunk 大小随时间预算变化;decode 路由同时均衡运行请求数与累积 KV 长度(请求数近似调度压力,KV 长度捕捉显存压力)。
  • 缓存亲和不是必需的:持久前缀池化在共享存储(约 100 PiB 3FS SSD 集群),请求可发往任何空闲 prefill worker;详见 harness-engineering/prompt-caching。
  • SWA Bounded Replay:只用最新 128 个 tokens 重建短寿命滑窗状态,DeepSeek 报告几乎将总 prefill 计算减半;这是模型特异的近似,可移植性未定。
  • 细粒度 KV 块拷贝走 GDR/RDMA:CUDA copy engine 路径性能差,目标是不占 SM 的同时打满 PCIe 带宽;持久块异步加载与层执行重叠。
  • 热门专家复制:周期性再分配副本以减少路由偏斜造成的设备间等待,代价是额外 HBM,复制因子收益递减。
  • CPU 调度与加速器执行重叠:异步调度消除 decode 步间空档。
  • DSpark 自适应投机验证:预测草稿接受概率并动态调整验证长度,相对"始终验证 5 个草稿"基线 +6% 生产吞吐。
  • 披露的 Blackwell 配置中未见张量并行(TP),有评论追溯到 H800 时代互联约束;这是对配置的解读,不是"TP 普遍更差"的证据。

Counterpoints & Gaps

  • 引擎开源但 KV-cache 管理与负载均衡未释放,完整服务无法直接复现。
  • vLLM 报告匹配配置下 decode 吞吐差距 6% 以内,但该对比不覆盖缓存运营、路由与端到端延迟。
  • 规模门槛存在:中小规模部署(少量 prefill worker)更宜流水线并行;宽 EP 需要足够流量与 worker 数才划算(实践者判断,非通用临界点)。

Sources (addendum)

Sources

Synthesized from 31 sources
  • AI Infra入门干货总结:大模型是如何高效推理的Supporting source listed by this page.Whole pagemediumbody
  • EP668 推理与网络Supporting source listed by this page.Whole pagemediumbody
  • EP675 AI 超级周期经济学Supporting source listed by this page.Whole pagemediumbody
  • EP646 CerebrasSupporting source listed by this page.Whole pagemediumbody
  • EP633 体系结构Supporting source listed by this page.Whole pagemediumbody
  • EP623 基础设施Supporting source listed by this page.Whole pagemediumbody
  • EP620 Agent 云Supporting source listed by this page.Whole pagemediumbody
  • EP614 模型效率Supporting source listed by this page.Whole pagemediumbody
  • EP609 EtchedSupporting source listed by this page.Whole pagemediumbody
  • EP607 软硬件协同Supporting source listed by this page.Whole pagemediumbody
  • EP595 AI 云Supporting source listed by this page.Whole pagemediumbody
  • 2026-09-19-17-42Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-10-22 AINews DeepSeek DeepInfer Open Source SOTA InferenceSupporting source listed by this page.Whole pagemediumabsorb log
  • 2026-09-19-17-42Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-06-15 至 2026-08-18 AI 节目专题索引Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-06-19 EP592.陈立武:66 岁接手 Intel,AI 算力战争下的半导体供应链重建Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-06-26 EP599.Nebius创始人:AI基础设施不是泡沫,但世界过度集中才是最大威胁Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-06-26 EP600.DeepMind预训练负责人:如何用“研究品味”敲开顶尖实验室大门Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-06-29 EP604.OpenAI Noam Brown:模型评估方式,正在错过 AI 真正的能力上限Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-06-29 EP605.Dwarkesh:AI 训练的下一个范式,为什么你的模型永远毕不了业Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-07-01 EP608.3Blue1Brown创始人:AI攻克数学前沿,人类创造力与科学发现将何去何从Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-07-07 EP617.Groq创始人:从濒临破产到200亿美元NVIDIA交易,AI时代如何用提问取代回答Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-07-08 EP619.为何我们并未处于 AI 资本支出泡沫中 - 能源主权与电力未来Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-07-24 EP643. 内核、强化学习与智能体奖励黑客专题Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-07-29 EP652.黄仁勋:打造英伟达的思维模式Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-08-01 EP656.All-in-芯片股暴跌,200 亿美元基金遭追加保证金,前沿实验室:请放慢 AI 发展步伐Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-08-02 EP659.xAI 联合创始人:Igor Babuschkin 解读模型开发的未来Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-08-04 EP662.:谷歌首席科学家Jeff Dean- 人工智能构建中的 1%法则Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-08-04 EP663.Gavin Baker- AI 抛售与数据不符 - 顶级 AI 投资者解析Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-08-12 EP671.Ryan Greenblatt:我们能控制 AI 递归自我改进吗?Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-08-15 EP677.Post-Traning 前沿现状与问题Supporting source listed by this page.Whole pagemediumabsorb log

Evolution

1 event
  1. absorbed

    Derived from source material

    This page is currently synthesized from 31 sources.

    From AI Infra入门干货总结:大模型是如何高效推理的, EP668 推理与网络, EP675 AI 超级周期经济学, EP646 Cerebras, EP633 体系结构To LLM Inference Mechanics
    Sources: raw/to-learn/AI Infra入门干货总结:大模型是如何高效推理的.md · raw/to-learn/跨国串门儿计划/episodes/2026-08-11 EP668.AI领域的下一个百倍增长:推理、网络与自我优化模型.md · raw/to-learn/跨国串门儿计划/episodes/2026-08-14 EP675.斯坦福2026 年春季 AI 经济学课程:从 GPU 到生命科学,一次听懂 AI 超级周期的九层经济学.md · raw/to-learn/跨国串门儿计划/episodes/2026-07-25 EP646.Cerebras CEO:CUDA绝非护城河.md · raw/to-learn/跨国串门儿计划/episodes/2026-07-17 EP633.图灵奖得主:GPU 与 TPU、计算机体系结构、RISC 与 CISV.md · raw/to-learn/跨国串门儿计划/episodes/2026-07-11 EP623.Dylan Patel 谈驱动 AI 革命的基础设施.md · raw/to-learn/跨国串门儿计划/episodes/2026-07-09 EP620.Modal CTO:你的 AI 应用不该跑在K8s 上,Agent 时代的云应该长什么样.md · raw/to-learn/跨国串门儿计划/episodes/2026-07-04 EP614.NVIDIA 核心科学家 Bryan Catanzaro:当算力堆到极限,开源模型如何靠“效率”继续变强.md · raw/to-learn/跨国串门儿计划/episodes/2026-07-01 EP609.挑战英伟达:Etched AI推理芯片如何让token成本暴跌.md · raw/to-learn/跨国串门儿计划/episodes/2026-06-30 EP607.SemiAnalysis创始人:AI的真正100倍杠杆,藏在软硬件协同设计里.md · raw/to-learn/跨国串门儿计划/episodes/2026-06-24 EP595. Lambda CTO:GPU 神话破灭后,AI 时代最稀缺的到底是什么.md · /Users/jean/Documents/Obsidian Vault/raw/briefing/AI Briefing/2026-09-19-17-42.md · /Users/jean/Documents/Obsidian Vault/raw/newsletters/AINews/2026-10-22 AINews DeepSeek DeepInfer Open Source SOTA Inference.md · raw/briefing/AI Briefing/2026-09-19-17-42.md · raw/to-learn/跨国串门儿计划/2026-06-15 至 2026-08-18 AI 节目专题索引.md · raw/to-learn/跨国串门儿计划/episodes/2026-06-19 EP592.陈立武:66 岁接手 Intel,AI 算力战争下的半导体供应链重建.md · raw/to-learn/跨国串门儿计划/episodes/2026-06-26 EP599.Nebius创始人:AI基础设施不是泡沫,但世界过度集中才是最大威胁.md · raw/to-learn/跨国串门儿计划/episodes/2026-06-26 EP600.DeepMind预训练负责人:如何用“研究品味”敲开顶尖实验室大门.md · raw/to-learn/跨国串门儿计划/episodes/2026-06-29 EP604.OpenAI Noam Brown:模型评估方式,正在错过 AI 真正的能力上限.md · raw/to-learn/跨国串门儿计划/episodes/2026-06-29 EP605.Dwarkesh:AI 训练的下一个范式,为什么你的模型永远毕不了业.md · raw/to-learn/跨国串门儿计划/episodes/2026-07-01 EP608.3Blue1Brown创始人:AI攻克数学前沿,人类创造力与科学发现将何去何从.md · raw/to-learn/跨国串门儿计划/episodes/2026-07-07 EP617.Groq创始人:从濒临破产到200亿美元NVIDIA交易,AI时代如何用提问取代回答.md · raw/to-learn/跨国串门儿计划/episodes/2026-07-08 EP619.为何我们并未处于 AI 资本支出泡沫中 - 能源主权与电力未来.md · raw/to-learn/跨国串门儿计划/episodes/2026-07-24 EP643. 内核、强化学习与智能体奖励黑客专题.md · raw/to-learn/跨国串门儿计划/episodes/2026-07-29 EP652.黄仁勋:打造英伟达的思维模式.md · raw/to-learn/跨国串门儿计划/episodes/2026-08-01 EP656.All-in-芯片股暴跌,200 亿美元基金遭追加保证金,前沿实验室:请放慢 AI 发展步伐.md · raw/to-learn/跨国串门儿计划/episodes/2026-08-02 EP659.xAI 联合创始人:Igor Babuschkin 解读模型开发的未来.md · raw/to-learn/跨国串门儿计划/episodes/2026-08-04 EP662.:谷歌首席科学家Jeff Dean- 人工智能构建中的 1%法则.md · raw/to-learn/跨国串门儿计划/episodes/2026-08-04 EP663.Gavin Baker- AI 抛售与数据不符 - 顶级 AI 投资者解析.md · raw/to-learn/跨国串门儿计划/episodes/2026-08-12 EP671.Ryan Greenblatt:我们能控制 AI 递归自我改进吗?.md · raw/to-learn/跨国串门儿计划/episodes/2026-08-15 EP677.Post-Traning 前沿现状与问题.md

Linked from