Skip to content
Back/Harness Engineering

Personal AI Benchmark

View in Graph
Updated 2026-11-12
1 min read
57 words

Personal AI Benchmark

方法论(口袋里的失败、pass/fail、辨识地平线)清楚,但页面后半被企业级 evals 新闻(微软 16,000 evals)冲淡;最关键的落地步骤——Jean 自己的 10 任务清单——自始至终没有执行。

主题导航

  • 从口袋里的失败构建个人基准:从失败任务和日常纠正构建 pass/fail 基准,并用合理变体防止过拟合;区分任务不够难与评审者无法核实答案两种边界。
  • 基准过期与环境泄漏:基准是活文档需按月复审难度,且任务环境本身会泄答案——RL 环境预置原题、不可达 Git object 里藏着参考修复这类审计发现说明分数可信度取决于环境清理。审计清单尚无固定版本。
  • 判定质量:verifier bug 与实质幻觉:评测必须把『任务完成』『无实质幻觉』『工具未越权』分开测(加入幻觉条件后通过率从 26.7% 掉到 8.9%),且自动 grader 自身要先审计(AutomationBench 206 个 verifier bug)。阈值来自单一评测,不可直接搬用。

来源与原文

拆分前完整原文与来源

Sources

Synthesized from 20 sources

Evolution

1 event
  1. absorbed

    Derived from source material

    This page is currently synthesized from 20 sources.

    Sources: /Users/jean/Documents/Obsidian Vault/raw/briefing/AI Briefing/2026-09-25-08-03.md · /Users/jean/Documents/Obsidian Vault/raw/briefing/AI Briefing/2026-09-26-08-04.md · /Users/jean/Documents/Obsidian Vault/raw/briefing/AI Briefing/2026-09-27-08-03.md · /Users/jean/Documents/Obsidian Vault/raw/briefing/AI Briefing/2026-10-01-08-03.md · /Users/jean/Documents/Obsidian Vault/raw/briefing/AI Builders Digest/2026-11-10.md · /Users/jean/Documents/Obsidian Vault/raw/newsletters/Every/2026-09-21 How to Create Your Own Personal AI Benchmark.md · /Users/jean/Documents/Obsidian Vault/raw/newsletters/Every/2026-09-25 Why Evals Are So Hot Right Now.md · /Users/jean/Documents/Obsidian Vault/raw/newsletters/Every/2026-09-26 Copilot Gets a Seat in the Org Chart.md · raw/briefing/AI Briefing/2026-09-25-08-03.md · raw/briefing/AI Briefing/2026-09-26-08-04.md · raw/briefing/AI Briefing/2026-09-27-08-03.md · raw/briefing/AI Briefing/2026-10-01-08-03.md · raw/briefing/AI Briefing/2026-10-27-17-40.md · raw/briefing/AI Builders Digest/2026-11-10.md · raw/newsletters/AINews/2026-10-10 not much happened today.md · raw/newsletters/Every/2026-09-21 How to Create Your Own Personal AI Benchmark.md · raw/newsletters/Every/2026-09-25 Why Evals Are So Hot Right Now.md · raw/newsletters/Every/2026-09-26 Copilot Gets a Seat in the Org Chart.md · raw/newsletters/Every/2026-10-01 Getting Started With Open Models.md · raw/newsletters/Every/2026-11-12 What's Left for Humans After Automation.md

Linked from