AI Adoption Levels 9–10 — Simulation Beyond Orchestration
What it is
Simulation is a proposed extension beyond AI adoption Level 8 (orchestration). Level 9 compares several Agent teams that pursue different strategies toward the same goal. Level 10 repeats that competition under varied conditions to estimate how much a result depends on chance. This is an author-proposed framework, not a validated maturity scale.
Why it matters
When a wrong strategy is costly and a real-world trial is expensive, simulation may help explore options before acting. Its value depends on whether the modeled environment, personas, and scoring rule predict outcomes that matter in reality. More runs reduce sampling noise only when the model is useful; they cannot repair a biased model.
Key points
- At Level 9, people specify the goal, candidate strategies, scoring, and competition rules. Separate teams generate alternatives; weak teams can critique and revise the current leader.
- At Level 10, repeat the same contest with controlled variation, then report outcome ranges and win rates instead of trusting the single winner.
- A cited 2006 cultural-market study found that visible prior choices increased both inequality and unpredictability across eight independent markets. The article applies this as a reason to test multiple simulated histories.
- Mike Taylor reports that nearly 2,000 Hacker News personas predicted front-page stories with about 60% accuracy, but early votes still created divergent outcomes. The experiment supports iterative screening, not one-shot prediction.
- Every prototypes Dan Lens and KatePass show how multiple candidate outputs can improve coverage or editing suggestions, according to the author's reported internal results. These figures have not been independently reproduced here.
- Human review remains necessary for validating the winning option, the simulation model, and decisions that carry real consequences.
- Use simulation only when its cost is lower than the expected cost of choosing poorly. For low-cost choices, a small real-world experiment may be more informative.
Evidence across sources
| Source | Key claim | Relevance |
|---|---|---|
| Every — The Next Two Levels of AI Adoption | Proposes Level 9 strategy competition and Level 10 repeated simulations; includes internal prototype figures and an HN prediction example | Primary source for the extension; figures are author-reported |
| Salganik, Dodds & Watts, 2006 | Social information increased inequality and unpredictability in an artificial music market with 14,341 participants | Supports the role of path dependence; does not validate AI simulations |
| I Cloned 2,000 Hacker News Users | Reports 60% headline prediction accuracy and identifies early social momentum as a major source of error | A related experiment by the same author; not independent confirmation |
| StrongDM’s Digital Twin Universe | Uses simulated SaaS services and scenario tests to validate software behavior | Related simulation practice with a different target: software correctness rather than strategy prediction |
Open questions
- What real outcomes calibrate a simulated strategy score, and how often should the model be revalidated?
- Which decisions have enough downside risk to justify many parallel runs rather than a direct experiment?
- How should teams report model uncertainty, persona coverage, simulation cost, and sensitivity to scoring choices?
- Can the approach protect the privacy and agency of people whose messages or interviews inform synthetic personas?
Prompts for witness
- Pick a decision where the cost of a wrong choice exceeds the cost of a small simulation. What observable outcome would tell you the model was useful?
- Which assumptions about customers or colleagues should remain human decisions rather than being hidden inside generated personas?
Related
- product-trends/ai-adoption-levels — the original eight-level adoption framework
- harness-engineering/evals-as-ledger — recording outcome, cost, and evaluation evidence
- product-trends/simulation-approval-boundary — separating simulated preferences from real authorization
Pending Jean review
- product-trends/ai-adoption-levels-simulation —
candidate/ai_draft; keep as a draft until one narrow simulation is compared with a real-world outcome and its cost is recorded.