Skill Engineering as Algorithm Design
What it is
Skill engineering is the practice of treating an AI agent not as a conversational partner but as a deterministic algorithm embedded in a controlled execution environment. The core insight: LLMs are probabilistic by nature, but the systems around them can be engineered for determinism, recoverability, and auditability.
This approach emerged from a practical frustration: when all rules are stuffed into a single SKILL.md, compliance degrades as the file grows. At around 200 lines, agents begin skipping steps, confusing field names across APIs, and executing without confirmation. The problem is not model intelligence but attention architecture — LLMs do not uniformly attend to all tokens in a long context.
Why it matters
The "agent as algorithm" framing resolves two production pain points:
- Token waste from trial-and-error: An agent exploring a "create permission set" flow might take 5 conversational rounds. A pre-built CLI command reduces this to 1 round.
- Path uncertainty: The same prompt can produce different outputs across models or context orderings. In production, this non-determinism is a liability.
The solution is architectural separation: the agent retains its strengths (natural language understanding, judgment, synthesis) while all deterministic operations (flow order, data formats, API calls, state management) are extracted into an external, non-LLM program that the agent cannot bypass.
Key points
- CLI as deterministic backbone: Shell commands, scripts, and typed interfaces handle everything that does not require judgment. The agent's job is reduced to parsing intent and emitting structured parameters.
- Context minimization: Instead of 200-line rule files, the agent receives only the rules relevant to the current step. This mimics how humans use checklists, not manuals.
- Error taxonomy: Production failures fall into predictable categories — YAML indentation errors, field name mismatches, skipped confirmation steps. These are harness problems, not model problems.
- Recoverability: When the deterministic layer fails, the system knows exactly where and why. When the LLM layer fails, the fallback is to ask for clarification rather than guess.
Evidence across sources
| Source | Key Claim | Relevance |
|---|---|---|
| <a href="/wiki/raw/to-learn/当我把-ai-变成一个"算法":skill-工程化设计的心路历程.md" class="wikilink">腾讯程序员 — Skill 工程化设计 | "不改变河的本性,但给它修好渠" — deterministic layer + probabilistic decision engine | Primary framework source |
| AI 简报 2026-06-07 | Self-improving skill loop: examples → trigger description → evals → memory → meta-skill cleanup | Concrete loop implementation that adds feedback and memory to deterministic skill design |
Open questions
- Where is the precise boundary between "should be deterministic" and "should be left to the model"? This boundary shifts as models improve.
- Does excessive determinism reduce the agent's ability to handle edge cases that were not anticipated by the CLI designer?
- How does this pattern scale to domains where no clean API or CLI exists?
Prompts for witness
- In Jean's current skill setup, which operations are deterministic enough to be extracted into scripts or CLIs?
- What would a "skill audit" look like — reviewing existing skills to identify deterministic vs. probabilistic components?
- When has an agent failed because the harness was too rigid, not because the model was too weak?
Related
- What is an Agent Harness?
- Multi-Agent Coordination Patterns
- Building Skills Best Practices
- AI Engineering Culture Stack
2026-09-17:把低频细节移出永久上下文
本次晚间 briefing 的技能观点认为,模型变强后,永久技能应更少、更薄;低频细节移到按需 references,权限和发布不变量放进代码与测试。这个规则把技能从静态说明书变成随模型能力更新的算法边界。
Counterpoints & Gaps
- 统计来自单个作者的技能库,不能直接代表所有项目。
- 删除旧规则前必须证明它没有承载安全、权限、数据边界或发布约束。
Sources (addendum)
- AI Briefing 2026-09-17 Evening
- AI Briefing 2026-11-12 Evening — Skill Audit 的触发与效果审计流程。
Pending Jean review
- harness-engineering/skill-engineering-as-algorithm —
ai_draft; audit one skill for expired rules, permanent context cost, and invariants that belong in executable checks.
2026-09-18:更强模型不意味着更厚的技能
本次 evening briefing 的技能讨论再次把“少而薄”作为可检验的设计方向:把低频细节放到按需 references,把安全、权限和发布不变量放进代码与测试,并让实现与审查分工。更薄的技能减少永久上下文成本,但并不允许删除仍承担安全边界的规则。
Counterpoints & Gaps
- “技能应变少”来自个人经验和社区讨论,不是跨项目统计结论。
- 删除或后置任何规则前,必须证明它没有承载权限、数据、认证或发布不变量。
Sources (addendum)
Pending Jean review
- harness-engineering/skill-engineering-as-algorithm —
ai_draft; audit one project skill for permanent-context cost, expired rules, and invariants that must remain executable.
2026-10-27:Skill 精简需要同任务对照
Nico Preme 描述的 skill-diet 流程把上下文精简变成可比较的评估:删去模型本身已能完成的指引,把低频细节移到按需文件,修正过时内容,再用同一组任务比较旧 Skill、新版和不加载 Skill。只有新版表现不差时才保留。作者称自己的 Skill 减少了 70–80% token 且结果不变;这是个人案例,不是跨项目测量。
这补充了“低频细节移出永久上下文”的设计原则:精简提案应由固定任务结果验证。评估前仍需标出不能删减的权限、认证、数据和发布不变量。
Counterpoints & Gaps
- token 减少和结果不变是作者自述,推文未提供测试集、评分标准或原始结果。
- 新旧版本对照要覆盖成功与失败路径;仅比较 token 或少数代表任务不足以证明安全约束未退化。 Skill Audit 补上了维护环的运行证据检查:先复盘最近三个本应触发技能的任务,区分没有触发与遥测缺失;再选 1–3 个任务,用固定输入比较技能启用与关闭时的质量、耗时、成本和返工。低频安全或事故响应技能不应仅凭 30 天零调用记录删除。这个方法把触发、记录和效果分开审计,但小样本与作者自述仍不足以证明普遍有效。AI Briefing 2026-11-12 Evening
Sources (addendum)
Pending Jean review
- harness-engineering/skill-engineering-as-algorithm — 选一个 Skill,用相同任务比较旧版、新版和不加载 Skill 的结果;先列出不得移除的安全与权限不变量。
2026-11-03 — 团队共享技能库的两个案例
AI Briefing 2026-11-03 Morning 汇总了两个团队案例:Figma 的 PM OS 据节目页简介包含 31 个技能、约 60–70 位产品经理使用;Grok Bot 的数据助手把指标定义、源表、查询逻辑、诊断检查和分析流程写入团队维护的共享技能,并由经理审核拟发布的共享记忆。
Inference: 技能可作为团队维护的操作接口,把定义、步骤和检查方法交给同事复用。现有来源是帖子与节目简介,没有给出采用日志、正确率或维护成本;人数与效果仍需核验。
Counterpoints & Gaps
- 两个案例都来自当事人或节目页转述,未提供独立评估。
- 团队规模本身不能说明技能质量;应另外记录复用率、错误率、维护成本和审核责任。
Sources (addendum)
- AI Briefing 2026-11-03 Morning — Figma PM OS 与 Grok Bot Data Bot 的团队技能案例。
- @clairevo — PM OS · @bot — Data Bot
Pending Jean review
- harness-engineering/skill-engineering-as-algorithm —
ai_draft; 找一个团队技能库,记录复用率、正确率、维护成本与审核责任,再决定是否作为稳定模式保留。