Skip to content
Back/Product Trends

Software Factory Ladder

View in Graph
Updated 2026-11-11
6 min read
1,422 words

Software Factory Ladder

What it is

A five-level framework for how much of an AI-assisted build–ship cycle runs without human intervention, proposed by @businessbarista in June 2026. The ladder is a diagnostic framework, not a measured ranking of companies. Matt Pocock adds a gradual trust discipline; Guillermo Rauch adds direction and real production outcomes, so autonomy is not judged by code volume alone. In a Nov 9 post, Pocock also calls the label hype while using it for a real shift: teams are moving from using agents on products to building automations for agents.

Why it matters

AI licenses, generated code, token consumption and merged PRs do not by themselves show that a software factory is producing useful, safe results. Cross-source synthesis: evaluate the human intervention still needed, expand workload only after observed quality, and connect customer signals to verified repairs and end-to-end user success.

Key points

  • The hard Level 2 → 3 transition requires reliable tests, a platform agents can use, and a concrete definition of low-risk changes.
  • A Stanford CS 146S/Monaco course account describes agent readiness as a staged engineering practice: inspect a minimal and a production harness, research code and constraints, compare approaches, let a human choose, then plan and test. Clear structure and verification are prerequisites for adding parallel agents; the interview describes curriculum and experience, not a measured productivity result.
  • Pocock recommends beginning with one issue per day, reviewing the output, and moving to two, five or ten only after quality earns trust; capacity should follow evidence, not ambition.
  • Rauch separates iteration speed from velocity: PRs and tokens measure activity, while velocity also needs a direction. Building an email service or payment gateway merely because AI can build it may dilute the work that matters.
  • A concrete defect can drive a fast repair loop; ambiguous navigation feedback still needed his product lead to talk to the user. Lower fix latency does not remove taste or human clarification.
  • Production P99 latency, error rate and error-free sessions complement local tests. A feature must work across the user's complete path, not only in the engineer's own trial.
  • A Magpie maintainer describes three loops for incoming issues, screened X product feedback, and recurring cleanup/security/learning; reported fixes are reproduced and tested on macOS, Windows and Linux before merge and release. This is an implementation account, not independent evidence of reliability or user outcomes.
  • The framework favors repeatable, digital, verifiable and reversible work. Strategy, subjective design choices, financial commitments and security-sensitive changes still require distinct judgment and authorization.

The five levels

Level Name Human Role AI Role
0 Artisan Human writes, reviews, tests, and deploys everything. None.
1 Assisted Human does all steps, but uses AI copilot to write faster. Glorified autocomplete.
2 Delegated Human directs; AI writes the fix and opens the PR. Human still reviews every PR. Drives execution.
3 Supervised Factory AI monitors production, catches bugs, writes fixes, reviews with a second agent, and auto-merges low-risk changes. Humans set guardrails and handle escalations. Runs the line; humans design it.
4 Autonomous Factory AI catches, fixes, tests, and ships before the team knows a bug existed. Humans decide what to build next. Fully autonomous loop.

Evidence across sources

Source Key Claim Relevance
@businessbarista — What the hell is a Software Factory? 0–4 ladder; hardest jump is delegation to supervised automation, with tests and low-risk rules Original industrialization framework
AI Briefing 2026-09-17 Evening / Matt Pocock; AI Briefing 2026-11-09 Morning / Matt Pocock One issue first; expand after quality review; later calls the term hype but uses it for the shift from agents doing product work to teams automating agent work Trust is earned at small workloads; evaluate the concrete workflow rather than the label
David Ondrej / Vercel CEO reveals his Agentic Engineering Workflow Speed versus direction; signal-to-fix latency; real production metrics; ambiguous feedback still needs people Adds outcome and risk distinctions to the autonomy ladder
AI Briefing 2026-11-09 afternoon — Mihail Eric, Stanford/Monaco Staged context, human selection of approach, explicit plans and tests; legible, verifiable code before parallel agents Adds a concrete readiness and engineering-education practice; interview-based
Peng Zheng / Lauren Tan on Grok Bot Rejects the factory label; eval playbook before merge; sometimes reviews after landing A practitioner counterexample, not a new ladder level
AI Briefing 2026-11-11 Evening — @xulijian1994, tweet 2119956996279669088 Describes routing GitHub/PR/Discord issues and screened X feedback into agent work, with cross-platform reproduction and tests before release A concrete feedback-to-release workflow, self-reported by the project maintainer; no independent reliability or outcome data

Counterpoints & Gaps

  • The original author's claim that many companies remain at Level 0 is a diagnosis, not an independently measured adoption rate.
  • Level 4 is an aspirational workflow, not a default recommendation for production, finance or high-risk operations.
  • Rauch's workflow and tool results are interview claims, not a controlled comparison proving that faster fixes alone cause business success.
  • Lauren Tan, an engineer on Grok Bot, rejects the factory label for her own setup and sometimes looks at a PR only after it lands. That is an interview claim with an eval playbook and an agent-friendly codebase as stated conditions, not a measured Level 3 or Level 4.
  • Magpie’s feedback routing, cross-platform testing and release loop are described by its maintainer; the post supplies no independent defect-rate, rollback, or user-satisfaction results.

Open questions

  • Which exact reversible changes can Jean delegate without a human review, and which require approval regardless of test results?
  • Does reducing signal-to-fix time improve accepted outcomes, or only increase the volume of changes?
  • How much additional workload can a workflow absorb before its quality and recovery behavior degrade?

Prompts for witness

  • Pick one recurring defect: what should count as a successful fix from report through the complete user session?
  • Which parts of a project deserve original work, and which should stay on audited infrastructure rather than be rebuilt?

Pending Jean review

  • product-trends/software-factory-ladder — ai_draft; keep draft. Review one low-risk fix loop using accepted repairs, user-session outcomes and rollback, before increasing its autonomous workload.

2026-10-22 evening — A software-factory draft maps components, not readiness

来源:AI Briefing 2026-10-22 Evening

Matt Pocock’s draft PR sketches ticket-triggered agent work using runners, orchestration, quality gates, and human checkpoints. The briefing reports that the author presents it as a reference framework rather than a ready-to-run system. It is useful as a component map for the autonomy ladder, not as evidence that a production factory has been validated.

Counterpoints & Gaps

  • The referenced PR is a draft and the workflow has not been independently run or audited in this task.
  • A component list does not establish reliability, security, or appropriate autonomy for a real repository.

Sources (addendum)

Pending Jean review

2026-10-30: Factory components do not establish readiness

来源:AI Briefing 2026-10-30 morning;Ben Holmes 的软件工厂文章,由 @BHolmesDev 分享。

Ben Holmes 列出其团队认为软件工厂需要的七项能力:配置即代码、集中对话历史、基准回放、工厂内聊天、产物管理、计算机操作和 Agent 调试。该清单补充了本页的成熟度阶梯,但能力是否存在不等于交付可靠。仍要检查可接受结果、审查负担、返工与回滚,并逐步扩大 Agent 负责的工作范围。作者团队使用 Warp Factories,清单属于团队实践总结。

Counterpoints & Gaps

  • 能力清单来自供应商团队的实践总结,没有跨团队的缺陷率或效率对照。
  • 七项能力并非每个仓库都必须具备;应从具体失败和验收瓶颈出发决定投入。

Sources (2026-10-30 addendum)

Pending Jean review

2026-11-02:Grok Bot 工程师拒绝 factory 这个词

来源:Peter Yang 采访 Peng Zheng 与 Lauren Tan。

Lauren Tan 说她不想把自己这套叫 software factory,而叫 Michelin kitchen。她要的是手艺,同时一家店还是要做出五十到一百个位子的量。她描述的自动合并有两个她自己说出的前提:代码库被做成 agent friendly;改 skill 时先用评测 playbook,让不同模型的子 agent 跑过,目标达到才合并。即便如此,她说有时 PR 落下去之后才看。这仍是访谈自述,不把 Grok Bot 标成阶梯上的某一级。

Pending Jean review

2026-11-02:自动合并前面的硬闸

来源:Lauren Tan 工作坊。

她把自动合并放在信任曲线的后段。前面是本地验证,然后才是云上复现。她说自己可以不再逐行看代码,靠的是会让 CI 变红的约束。rules、Bugbot、skill 和风格指南她仍会用,但说只靠这些,仓库变差只是时间问题。她又用厨房作比:人是主厨,菜由线上的人做。这仍不给 Grok Bot 在阶梯上定级。

Pending Jean review

Sources

Synthesized from 10 sources
  • What the hell is a Software Factory? — @businessbaristaSupporting source listed by this page.Whole pagemediumbody
  • AI Briefing 2026-09-17 EveningSupporting source listed by this page.Whole pagemediumbody
  • David Ondrej / Vercel CEO reveals his Agentic Engineering WorkflowSupporting source listed by this page.Whole pagemediumbody
  • Peter Yang — We Built Grok BotSupporting source listed by this page.Whole pagemediumbody
  • AI Briefing 2026-11-09 afternoon — Mihail Eric, Stanford/MonacoSupporting source listed by this page.Whole pagemediumbody
  • AI Briefing 2026-11-11 EveningSupporting source listed by this page.Whole pagemediumbody
  • 2026-09-17-01-58Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-10-30-08-18Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-11-09-20-09Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-11-02-youtube-PcO998TSzIs-lauren-tan-grokbot-teamSupporting source listed by this page.Whole pagemediumabsorb log

Evolution

1 event
  1. absorbed

    Derived from source material

    This page is currently synthesized from 10 sources.

    From What the hell is a Software Factory? — @businessbarista, AI Briefing 2026-09-17 Evening, David Ondrej / Vercel CEO reveals his Agentic Engineering Workflow, Peter Yang — We Built Grok Bot, AI Briefing 2026-11-09 afternoon — Mihail Eric, Stanford/MonacoTo Software Factory Ladder
    Sources: raw/to-learn/What the hell is a Software Factory?.md · raw/briefing/AI Briefing/2026-09-17-01-58.md · raw/to-learn/2026-10-10-youtube-WeiB_gLOdQE-vercel-agentic-engineering.md · raw/to-learn/2026-11-02-youtube-xZ5TEaleUdg-grok-bot-14-best-bots.md · raw/briefing/AI Briefing/2026-11-09-20-31.md · raw/briefing/AI Briefing/2026-11-11-23-00.md · /Users/jean/Documents/Obsidian Vault/raw/briefing/AI Briefing/2026-09-17-01-58.md · raw/briefing/AI Briefing/2026-10-30-08-18.md · raw/briefing/AI Briefing/2026-11-09-20-09.md · raw/to-learn/2026-11-02-youtube-PcO998TSzIs-lauren-tan-grokbot-team.md

Linked from