AAperture
Back/harness engineering

Harness, Memory, Context Fragments, & the Bitter Lesson

Updated 2026-04-13
2 min read
447 words

Harness, Memory, Context Fragments, & the Bitter Lesson

Author: Viv Trivedy (@Vtrivedy10) Source: X thread, 2026-04-12 A work-in-progress mental dump on the intersection of harness design, memory accumulation over long timescales, and the search bitter lesson we can't escape.

Harnesses & Context Fragments

A crucial job of the harness is to efficiently and correctly route data within its boundaries into the context window for computation.

The context window is a precious artifact. Harnesses make explicit decisions on how to populate, manage, edit, and organize it. Each loaded object can be thought of as a Context Fragment — an explicit decision by the user and harness designer about what the model needs to do work at any given time.

Many ideas on externalizing objects and loading them into context are pioneered by @a1zhang with RLMs (Reinforcement Learning Machines).

Experiential Memory

We're in the very early days of deploying agents, and agents produce massive amounts of data in every interaction. This is analogous to humans doing things and remembering them.

Agent memory has a massive advantage: it can be accumulated across all agents, which are easily forked and duplicated (unlike humans). As @dwarkesh_sp discusses, this is a massive benefit of artificial systems.

Memory can be treated as an externalized object. The harness is tasked with contextualized retrieval — pulling in the right data from accumulated memories across all agent interactions.

Search & The Bitter Lesson

As we deploy agents over year-long timescales, there will be a hyper-exponential growth in data produced by those agents. We should want to:

  1. Own that data — open ecosystems are critical
  2. Use that data — which means searching over, distilling, and organizing massive amounts of it

Our brains are exceptional at this: using prior experience contextually, and committing the right stuff to memory through intentional practice. Our current infrastructure and algorithms will be put to the test and often break in this new data regime.

Open questions

  • How do we efficiently distill experiences (traces) into higher-level memory primitives that capture the important parts — over ultra-long time horizons?
  • How much of the future is search just-in-time vs search integrated into model weights?
  • How do we make models much better at self-managing their context window?
  • How do we reduce error rates in recursively allowing agents to operate over external objects?

Sources

Linked from