Dense Feedback for Agent Debugging
What it is
Dense feedback for agent debugging uses local, discriminative signals to connect observed behavior with testable explanations. The agent gathers facts, identifies useful controls, proposes hypotheses, and checks each hypothesis against the evidence instead of optimizing only a distant pass/fail or aggregate metric.
Why it matters
An aggregate result can confirm that a system regressed without identifying where or why. Raw logs and traces give an agent more evidence to process, but unfiltered data can also obscure the useful signals. Human input remains important for choosing controls, constraints, and the next question to test.
Key points
- Separate observations from explanations. Record behavior as facts before generating hypotheses from implementation details.
- Choose a control that does not show the bug, such as a known-good request, build, or environment. Compare it with the failing case to identify signals that distinguish plausible causes.
- Prefer signals that satisfy only a few candidate explanations. A fact-to-hypothesis table can expose which explanations fit all observations.
- Validate a promising hypothesis against every relevant fact. Do not treat an untested explanation as a root cause.
- For system optimization, expose layered signals for correctness, system behavior, and performance. The earlier inference example reports that such interfaces helped diagnose numerical drift, a transfer bottleneck, and repeated work.
- Agents can process large raw datasets, while people select controls, constraints, and decision points. Aggregation that helps a human dashboard can remove evidence an agent needs.
Evidence across sources
| Source | Key claim | Relevance |
|---|---|---|
| AI Briefing 2026-09-17 Afternoon | The inference team exposed correctness, system-behavior, and performance interfaces to make optimization diagnosable. | Example of local objective signals supporting agent-led investigation. |
| AI Briefing 2026-10-21 Afternoon | Martin Fowler's summary of Cindy Sridharan's debugging method starts with behavioral facts and a control, then tests hypotheses against constraints. | Adds a general investigation sequence and clarifies the human role in selecting controls and rejecting unverified hypotheses. |
Open questions
- Which diagnostic interfaces generalize beyond inference systems?
- How should teams measure whether a local signal improves root-cause attribution instead of encouraging local optimization?
- What is the minimum signal set needed before an agent can run expensive experiments autonomously?
- How can raw data be preserved without making irrelevant noise dominate the agent's investigation?
Prompts for witness
- Which current workflow reports only a final pass or fail when the agent needs a layered explanation?
- What expert debugging judgment could be turned into a cheap local probe?
- Where would dense feedback create more instrumentation than the task value justifies?
Counterpoints & Gaps
- The two sources describe different domains and do not establish that one debugging sequence works for all systems.
- Selecting a control and ranking signals still require domain judgment. Poor controls can steer an agent toward a plausible but incorrect explanation.
- The performance figures in the September source are source-reported and have not been independently reproduced here.
Related
- harness-engineering/agent-trust-verification
- harness-engineering/self-verification-loops
- harness-engineering/judgment-engineering
- harness-engineering/overview
Pending Jean review
- harness-engineering/dense-feedback-for-agent-debugging —
ai_draft; suggested action: keep as a draft or test the control-and-hypothesis loop on a local debugging case.