Skip to content
Back/Harness Engineering

Heuristic Learning

View in Graph
Updated 2026-11-04
3 min read
687 words

Heuristic Learning

What it is

Heuristic Learning is a proposed learning loop where the object being updated is not neural-network weights, but a maintainable software system: rules, detectors, controllers, tests, logs, replays, memory, and regression cases. A coding agent consumes feedback and edits the heuristic system directly.

Jiayi Weng calls the maintained object a Heuristic System. It is more than a single policy.py; it is a program system with feedback channels, experiment records, replay artifacts, and mechanisms for compression or regression protection.

Why it matters

The key claim is that coding agents change the maintenance curve for heuristics. Expert systems and rule systems were historically brittle because humans could not afford to keep repairing them. If coding agents can continuously read failures, edit code, add tests, inspect replays, and simplify accumulated patches, some heuristic systems may become worth owning again.

This reframes Agent 持续学习: continual learning does not have to mean updating weights every time. Some online learning can happen by turning feedback into readable, testable, refactorable software artifacts.

Evidence across sources

Source Key Claim Relevance
Jiayi Weng (original paper) Codex-maintained heuristic policies reach high scores in Breakout, Ant, HalfCheetah, and Atari57 without training new neural-network weights Foundation: the learning object is code, not weights
AI Briefing 2026-05-09 GPT-5.4 writes Python policies for Atari Breakout, iterates from 387 to 864 (perfect score); MuJoCo Ant exceeds 6000; full Atari57 approaches PPO baseline Second-source validation with specific scores and explicit boundary conditions

Specific results from the second source:

  • Atari Breakout: 387 → 864 (perfect) through iterative code refinement by GPT-5.4
  • MuJoCo Ant: exceeds 6000 points (deep RL level performance)
  • Atari57 full suite: approaches PPO baseline
  • Knowledge is stored in readable code with ball-path predictors, stuck-ball detectors, regression tests, and experiment logs

Preservation mechanisms: Old capabilities can be preserved as regression tests, fixed-seed replays, golden traces, failure videos, version diffs, and written-down failed directions.

Failure mode: A heuristic system that only grows and never compresses becomes a large, coupled system that neither humans nor agents can maintain.

Core loop

environment feedback / test failure / log anomaly
  -> coding agent reads context
  -> edits policy / test / memory
  -> reruns
  -> writes results back into trials and summaries
  -> compresses local patches into maintainable structure

Boundary conditions

  • Heuristic Learning is strongest where feedback is clear, states are inspectable, and failures can be reproduced.
  • It is weaker in domains where perception and long-horizon generalization cannot be expressed cleanly in code.
  • Explicit boundary from practitioner evidence: pure code cannot handle complex perception tasks (e.g., image recognition with if-else rules). The endgame is a hybrid architecture: lightweight neural networks for perception, heuristic learning for real-time logic and safety rules, and large models for log review and periodic updates.
  • It does not remove forgetting; it turns forgetting into an engineering problem around regression tests, replay coverage, state reproducibility, coupling, and refactoring.

Open questions

  • When should a heuristic remain code, and when should its experience become training data for a neural network?
  • How can a coding agent measure whether a heuristic system has become too coupled to maintain?
  • Can Heuristic Learning become a general layer inside robotics, or is it mostly useful for environments with strong simulators and reproducible tests?
  • What is the minimum harness needed for safe unattended HL loops?

2026-11-03 — 直接编辑权重是另一种学习对象

AI Briefing 2026-11-03 Morning 报道了一项早期实验:论文作者称 GPT-6 Astra 通过直接编辑少量 Qwen3.5-0.8B 权重,改善了特定数学与指令遵循评测。该路线修改模型参数;Heuristic Learning 则把经验写进可读、可测试的软件系统。两者应作为不同的更新对象比较。

Simon Willison 后续报告了另一种操作:他在笔记本上让 Qwen 3.8 27B 只看原始权重数组、不开工具,识别出四个简化游戏网络中的三个;成功运行耗时约 46 至 65 分钟,Freeway 未识别成功。这是从权重中推断网络行为,不是修改权重;两者也都不同于把经验写进可测试代码的 Heuristic Learning。该结果来自作者的一组本地实验,不能说明跨模型或任务的普遍可靠性。

Counterpoints & Gaps

  • 结果来自一项早期研究和小模型设置;单篇来源不能证明跨模型、任务或部署环境的可靠性。
  • 直接编辑后的参数仍需独立基准、行为回归和可复现实验;帖子中的分数应视为论文作者报告。

Sources (addendum)

Pending Jean review

Sources

Synthesized from 3 sources
  • Learning Beyond Gradients — Jiayi WengSupporting source listed by this page.Whole pagemediumbody
  • 2026-11-03-07-59Supporting source listed by this page.Whole pagemediumabsorb log
  • 2026-11-04-08-03Supporting source listed by this page.Whole pagemediumabsorb log

Evolution

1 event
  1. absorbed

    Derived from source material

    This page is currently synthesized from 3 sources.

    From Learning Beyond Gradients — Jiayi Weng, 2026-11-03-07-59, 2026-11-04-08-03To Heuristic Learning
    Sources: raw/to-learn/Learning Beyond Gradients - Jiayi Weng · raw/briefing/AI Briefing/2026-11-03-07-59.md · raw/briefing/AI Briefing/2026-11-04-08-03.md

Linked from