Constraint Lifecycle
What it is
Constraint lifecycle is the discipline of managing harness rules, skills, and behavioral constraints as living artifacts with explicit expiration conditions. Every constraint — a prompt rule, a skill file, a harness check, a wrapper layer — encodes an assumption about model behavior. When models improve, those assumptions decay. The lifecycle question is: when do you delete, demote, or re-evaluate a constraint?
Why it matters
Accumulated constraints create silent debt. A harness that worked six months ago may now be full of tricks that were necessary for a weaker model but actively harmful for a stronger one. Kuncheng's warning about Oh My Pi applies broadly: bundled harness tricks can look good on benchmarks but become untrustworthy unless every component is re-evaluated on each model release. The bitter lesson frame favors scalable evaluation over accumulating brittle hacks. Re-testing is not only deletion: Aaron Levie frames the same periodic check as a way to find automation that earlier models could not handle.
Evidence across sources
| Source | Key Claim | Relevance |
|---|---|---|
| AI Briefing 2026-08-30 Afternoon — Kuncheng | Bundled harness tricks decay with model releases; minimal harness is easier to re-evaluate and improve | Core lifecycle argument |
| AI Briefing 2026-08-30 Afternoon — Pawel Huryn | Skill memory as living system: test hypotheses, retain confirmed rules, demote broken ones | Rule maintenance loop |
| AI Briefing 2026-09-06 Afternoon — skill-doctor | Delete zero-invocation skills; dry-run report before removal; check safety-critical status before cleanup | Operational deletion protocol |
| AI Briefing 2026-09-06 Afternoon — jordanovweb3 | Move important rules from prompt text into harness boundaries; add loops that enforce constraints and feed learning back | Constraint promotion path |
| AI Builders Digest 2026-11-09 — Aaron Levie | Levie recommends retesting AI workflows every three to six months, because progress in tool use, reasoning, context retention, multimodality and domain knowledge may let agents operate existing APIs and systems | A stated cadence, and the other side of decay: a re-test can add work as well as remove rules |
| AI Builders Digest 2026-11-09 — Mihail Eric, Stanford/Monaco | The digest summary of the episode says examples and instructions that once helped can become unnecessary or overly restrictive as models improve, and advises re-testing tools and workflows every few months with benchmarks or evals | A second cadence claim, with the requirement to measure whether a change helps |
| AI Briefing 2026-11-10 Afternoon — Dave Kiss | Proposes logging rule use against external outcomes, then checking both outcome improvement and whether the rule would have blocked prior wins | Adds outcome evidence and counterfactual harm checks before a rule becomes a hard constraint; his thresholds remain practitioner proposals |
The lifecycle stages
- Creation — a rule is added in response to a observed failure or benchmark result
- Validation — test the constraint against real tasks and external outcomes; log which rules each attempt used, compare results, and check whether enforcement would have blocked earlier wins
- Promotion — a prompt-only rule that proves durable gets upgraded into a harness check
- Monitoring — usage data, invocation counts, and context cost are tracked
- Decay detection — model releases or workflow changes invalidate the original assumption
- Demotion or deletion — the constraint is removed, demoted, or replaced
Deletion heuristics
- Zero invocations over a defined period — candidate for deletion (with dry-run report first)
- Model upgrade — re-test all non-default rules; remove tricks that no longer improve real tasks
- Current run disproves rule — demote immediately, do not wait for scheduled review
- Outcome receipt — keep a rule deterministic only when observed outcomes support it and a counterfactual check shows little risk of blocking prior wins; Dave Kiss’s proposed thresholds are not validated universal cutoffs
- Safety-critical check — verify before removal even if usage is low
- Calendar re-test — Levie names three to six months for workflows and Eric "every few months" for tools and workflows. Both are practitioner advice without measured evidence, so a model release stays the stricter trigger and the calendar is a fallback.
The fat-skills connection
Fat Skills, Fat Code, Thin Harness argues for pushing judgment into skills and execution into code. HiTw93's corollary adds that fat does not mean static: skills need periodic audit, deleting rules the model now handles stably. A new skill must beat the simpler alternative in risk, rollback, or latency to justify its existence.
Open questions
- What is the optimal audit cadence: per model release, monthly, or continuous? Two practitioner accounts give three to six months and "every few months"; neither reports a measured basis.
- How do you distinguish "safety-critical but rarely invoked" from "genuinely obsolete"?
- Should constraint decay be measured quantitatively (task success delta) or qualitatively (reviewer judgment)?
Related
- Fat Skills, Fat Code, Thin Harness — Layer separation principle
- Harness, Memory, Context Fragments, & the Bitter Lesson — Model progress vs. accumulated infrastructure
- Self-Evolving Harness Composability — Harness that updates itself
- Skills into Loops — Feedback loops for skill improvement
Pending Jean review
- harness-engineering/constraint-lifecycle —
ai_draft; suggested action: run skill-doctor on the local skill folder and produce a dry-run deletion report.