Skip to content
Back/Harness Engineering

Constraint Lifecycle

View in Graph
Updated 2026-11-10
4 min read
894 words

Constraint Lifecycle

What it is

Constraint lifecycle is the discipline of managing harness rules, skills, and behavioral constraints as living artifacts with explicit expiration conditions. Every constraint — a prompt rule, a skill file, a harness check, a wrapper layer — encodes an assumption about model behavior. When models improve, those assumptions decay. The lifecycle question is: when do you delete, demote, or re-evaluate a constraint?

Why it matters

Accumulated constraints create silent debt. A harness that worked six months ago may now be full of tricks that were necessary for a weaker model but actively harmful for a stronger one. Kuncheng's warning about Oh My Pi applies broadly: bundled harness tricks can look good on benchmarks but become untrustworthy unless every component is re-evaluated on each model release. The bitter lesson frame favors scalable evaluation over accumulating brittle hacks. Re-testing is not only deletion: Aaron Levie frames the same periodic check as a way to find automation that earlier models could not handle.

Evidence across sources

Source Key Claim Relevance
AI Briefing 2026-08-30 Afternoon — Kuncheng Bundled harness tricks decay with model releases; minimal harness is easier to re-evaluate and improve Core lifecycle argument
AI Briefing 2026-08-30 Afternoon — Pawel Huryn Skill memory as living system: test hypotheses, retain confirmed rules, demote broken ones Rule maintenance loop
AI Briefing 2026-09-06 Afternoon — skill-doctor Delete zero-invocation skills; dry-run report before removal; check safety-critical status before cleanup Operational deletion protocol
AI Briefing 2026-09-06 Afternoon — jordanovweb3 Move important rules from prompt text into harness boundaries; add loops that enforce constraints and feed learning back Constraint promotion path
AI Builders Digest 2026-11-09 — Aaron Levie Levie recommends retesting AI workflows every three to six months, because progress in tool use, reasoning, context retention, multimodality and domain knowledge may let agents operate existing APIs and systems A stated cadence, and the other side of decay: a re-test can add work as well as remove rules
AI Builders Digest 2026-11-09 — Mihail Eric, Stanford/Monaco The digest summary of the episode says examples and instructions that once helped can become unnecessary or overly restrictive as models improve, and advises re-testing tools and workflows every few months with benchmarks or evals A second cadence claim, with the requirement to measure whether a change helps
AI Briefing 2026-11-10 Afternoon — Dave Kiss Proposes logging rule use against external outcomes, then checking both outcome improvement and whether the rule would have blocked prior wins Adds outcome evidence and counterfactual harm checks before a rule becomes a hard constraint; his thresholds remain practitioner proposals

The lifecycle stages

  1. Creation — a rule is added in response to a observed failure or benchmark result
  2. Validation — test the constraint against real tasks and external outcomes; log which rules each attempt used, compare results, and check whether enforcement would have blocked earlier wins
  3. Promotion — a prompt-only rule that proves durable gets upgraded into a harness check
  4. Monitoring — usage data, invocation counts, and context cost are tracked
  5. Decay detection — model releases or workflow changes invalidate the original assumption
  6. Demotion or deletion — the constraint is removed, demoted, or replaced

Deletion heuristics

  • Zero invocations over a defined period — candidate for deletion (with dry-run report first)
  • Model upgrade — re-test all non-default rules; remove tricks that no longer improve real tasks
  • Current run disproves rule — demote immediately, do not wait for scheduled review
  • Outcome receipt — keep a rule deterministic only when observed outcomes support it and a counterfactual check shows little risk of blocking prior wins; Dave Kiss’s proposed thresholds are not validated universal cutoffs
  • Safety-critical check — verify before removal even if usage is low
  • Calendar re-test — Levie names three to six months for workflows and Eric "every few months" for tools and workflows. Both are practitioner advice without measured evidence, so a model release stays the stricter trigger and the calendar is a fallback.

The fat-skills connection

Fat Skills, Fat Code, Thin Harness argues for pushing judgment into skills and execution into code. HiTw93's corollary adds that fat does not mean static: skills need periodic audit, deleting rules the model now handles stably. A new skill must beat the simpler alternative in risk, rollback, or latency to justify its existence.

Open questions

  • What is the optimal audit cadence: per model release, monthly, or continuous? Two practitioner accounts give three to six months and "every few months"; neither reports a measured basis.
  • How do you distinguish "safety-critical but rarely invoked" from "genuinely obsolete"?
  • Should constraint decay be measured quantitatively (task success delta) or qualitatively (reviewer judgment)?

Pending Jean review

Sources

Synthesized from 5 sources
  • AI Briefing 2026-08-30 AfternoonSupporting source listed by this page.Whole pagemediumbody
  • AI Briefing 2026-08-30 EveningSupporting source listed by this page.Whole pagemediumbody
  • AI Briefing 2026-09-06 AfternoonSupporting source listed by this page.Whole pagemediumbody
  • AI Builders Digest 2026-11-09Supporting source listed by this page.Whole pagemediumbody
  • AI Briefing 2026-11-10 AfternoonSupporting source listed by this page.Whole pagemediumbody

Evolution

1 event
  1. absorbed

    Derived from source material

    This page is currently synthesized from 5 sources.

    From AI Briefing 2026-08-30 Afternoon, AI Briefing 2026-08-30 Evening, AI Briefing 2026-09-06 Afternoon, AI Builders Digest 2026-11-09, AI Briefing 2026-11-10 AfternoonTo Constraint Lifecycle
    Sources: raw/briefing/AI Briefing/2026-08-30-17-41.md · raw/briefing/AI Briefing/2026-08-30-23-07.md · raw/briefing/AI Briefing/2026-09-06-17-40.md · raw/briefing/AI Builders Digest/2026-11-09.md · raw/briefing/AI Briefing/2026-11-10-17-34.md

Linked from