Agent in Loop Methodology
What it is
Agent-in-loop engineering gives an agent the project context, tools, and feedback needed to complete a bounded task through a repeatable cycle: clarify the expected behavior, build, inspect the result, and feed recurring failures back into the harness.
Why it matters
A capable model cannot compensate for missing context or unverifiable expectations. A well-shaped loop can reduce handoffs while keeping the result accountable to evidence and human review at consequential boundaries.
Key points
- Give the agent a usable harness. Provide relevant repository context and focused tools that make logs, documentation, and other evidence navigable. Elvis Sun’s reported one-shot turbo-cache fix illustrates the potential of a complete debugging environment; it is an example, not a reliability guarantee.
- Specify before building. In one account, agents first discussed a system and produced a specification, then built it. The author estimated a week of work took one or two hours; this is a personal estimate, not a general benchmark.
- Check independently after generation. The same account describes post-build checks that surfaced bugs, omissions, and dead code. Acceptance criteria and checks make those findings actionable instead of relying on the agent’s own completion claim.
- Turn repeated findings into process. A small-team adoption proposal recommends treating feedback as candidate work, applying human approval where needed, and moving recurring issues into skills, workflows, lint, or tests.
- Bound autonomy by the task. Start with a measurable workflow, named owner, known data and permissions, and an explicit acceptance check. Decide which steps can run unattended and where a person must review the result.
Evidence across sources
| Source | Key claim | Relevance |
|---|---|---|
| AI Briefing 2026-04-11 | Elvis Sun describes giving an agent a suitable harness and reports a one-shot turbo-cache fix. | Supports the role of context and tools in an engineering loop. |
| AI Briefing 2026-10-22 Morning | An account describes spec-first build and post-build checks; a second proposes human-reviewed feedback and encoding recurring failures into workflow assets. | Adds explicit specification, verification, and feedback stages; the reported time saving remains anecdotal. |
Open questions
- Which repeated task in a real repository has a clear enough acceptance check to run as a bounded loop?
- Which findings should become a test or lint rule, and which still require human judgment?
- Does the loop reduce rework when measured across several tasks, rather than one reported example?
Prompts for witness
- Which recurring task in your work loses the most time to missing context or repeated clarification?
- What evidence would convince you that an agent completed that task correctly?
- Where should the loop stop and ask for human review?
Related
Pending Jean review
- harness-engineering/agent-in-loop-methodology — ai_draft; suggested action: keep as a draft and evaluate the specification, independent-check, and feedback loop on a repeated repository task before treating the reported time saving as transferable.
2026-10-24 — FDE 案例把客户 Eval 与 Harness 变成可复制资产
来源:AI Briefing 2026-10-24 Morning;@Xudong07452910 转述 OpenAI FDE 负责人 Colin Jarvis 的工程案例。
该案例补充了“指定行为—构建—独立验证—反馈”的企业落地实例:半导体团队围绕客户流程建立 Eval 与 Harness;推文称定向测试生成节省约 48% 时间,首个团队部署约两个月、复制到第二个团队约一天,并在 Codex 能力增强后移除约 90% 定制代码,改用 Skills 和插件。以上均是推文对案例的转述,应回看原始工程博客后再作为可迁移基准。
Counterpoints & Gaps
- 时间线未包含原始博客全文;百分比、时间和代码删减比例均未独立验证。
- 复用速度取决于 Eval、连接器和客户流程是否相似,不能直接推定其他企业也能一天上线。
Sources (addendum)
Pending Jean review
- harness-engineering/agent-in-loop-methodology —
ai_draft; 找到并核对原始工程博客,再判断案例数字是否适合纳入可复用的企业 Agent 工作流。