Exploring how models update memory. Updated September 11, 2026.

The compare command shows how different models interpret the same conversation. It puts their summaries, proposed weight changes, connections, and emerging patterns side by side.

The compare command

./dist/attractor.mjs compare conversation.txt
./dist/attractor.mjs compare --session

Runs the same conversation through several models and shows what each would do: summary, vibes, basin deltas, proposed connections, resulting entropy. Nothing is saved to the attractor state.

Set ATTRACTOR_COMPARE_MODELS to a comma-separated list of legs. A leg is an engine:model pair, so you can vary the model, the transport, or both. A bare model name means the local CLI. Without an explicit list, the command uses the configured summary and update models through the CLI, and adds API legs when ANTHROPIC_API_KEY is set. Set the list explicitly when you need a reproducible comparison.

ATTRACTOR_COMPARE_MODELS="cli:claude-haiku-4-5-20251001,cli:claude-opus-5,api:claude-opus-5" \
  ./dist/attractor.mjs compare --session

What this comparison measures

Each leg starts from the same loaded state and transcript, then uses its selected model for both summarization and update generation. It therefore compares the whole pipeline. A different summary can cause a different update even if the update stage behaves identically.

To isolate the update model, a separate experiment should reuse one fixed summary, vibes list, starting state, and prompt across all legs. That fixed-summary experiment is the next step in the comparison plan.

Controls

Transport: same model, cli: vs api:

Running the same model through both engines helps investigate transport differences. Shared prompt-building code reduces avoidable differences, but each leg still generates a separate summary. Individual outputs can also vary between runs. When outputs differ, inspect the requests and repeat the comparison to understand how much variation comes from each stage.

The implementation and development notes identify three sources of avoidable variation:

1. Prompt roles. The API previously sent the attractor prompt as the system prompt, while the CLI concatenated it into the user turn. Both engines now route through one primitive, Engine.call(system, user, model, maxTokens), with the attractor prompt as the system prompt and a fixed short user turn.
2. Sampling. The reviewed API engine sends no temperature, and the CLI wrapper supplies no temperature option. Backend sampling remains a factor to examine through repeated runs.
3. Environment. The CLI is an agent interface with configuration and tools beyond a single completion call. The wrapper passes options to reduce that inherited context. See running.md.

The reviewed CLI wrapper ignores the shared maxTokens argument, while the API engine sends it. Development notes describe an API reply truncated at an 800-token limit while the CLI run succeeded; the API update limit in the reviewed source is 4000. The generation budgets remain different, which matters when interpreting transport comparisons.

Noise: two runs per model

Two runs provide a first look at variation, but are too few to establish a reliable noise estimate. The existing notes record this example from one conversation:

  basin                cli:haiku-4-5 cli:haiku-4-5    cli:opus-5    cli:opus-5
  ----------------------------------------------------------------------------
  Context architecture         +0.05         +0.10         +0.05         +0.05
  Systems design               +0.15         +0.15         +0.09         +0.08
  ----------------------------------------------------------------------------
  connections                      0             0             0             0
  emerging                         0             0             3             4

What the small sample suggests

In this example, Haiku proposed larger positive changes to the main basin (+0.15 in both runs) than Opus (+0.09 and +0.08). Opus returned three or four emerging patterns; Haiku returned none. That gives the next set of runs two concrete things to investigate: delta size and emerging-pattern frequency.

Track emerging patterns and new_basin separately: one records possible directions, and the other proposes a new basin. Tracking repeated updates will also show how quickly weights approach their ceiling.

The next comparison should use fixed summaries and starting states, more conversations, and replicate runs. See the research plan and Attractor — Context Research.

The run log

Successful local CLI ingestion and comparison updates are appended to ~/.attractor/runs.jsonl, from ingest and from each compare leg, marked applied or dry.

./dist/attractor.mjs runs              # grouped by conversation
./dist/attractor.mjs runs HASH         # replace HASH with a transcript hash
./dist/attractor.mjs runs opus         # filter by model

  3553445eb1f8  466 chars, 2 run(s)
  "User: I want the attractor to run without an API key so anyone with..."

    basin                      haiku-4-5          opus-5
    ----------------------------------------------------
    systems-design                 +0.15           +0.09
    context-architecture           +0.08           +0.06
    ----------------------------------------------------
    emerging                           1               4
    via                              cli             cli
    applied                          dry             dry
    when                      2026-09-11      2026-09-11

Runs are grouped by transcript hash, which makes it easy to find outputs for the same conversation. For a controlled rerun, retain the starting state, summary, prompt, configuration, and model alongside the output. Those records complete the inputs needed for replay.

attractor history also records which model produced each step:

2026-09-11 02:35    67% #######...    87% #########.    opus-5
2026-09-11 02:35    65% #######...   100% ##########    haiku-4-5

Local CLI history includes this model provenance to help investigate changes. Hosted snapshots record timestamps and basin weights.

The log is plain JSON Lines (one JSON object per line), so you don't need this tool to analyse it:

python3 -c "
import json, collections
agg = collections.defaultdict(list)
for line in open('$HOME/.attractor/runs.jsonl'):
    r = json.loads(line)
    for u in r['update']['basin_updates']: agg[r['model']].append(u['weight_delta'])
for m, v in agg.items(): print(f'{m:<30} mean delta {sum(v)/len(v):+.3f} (n={len(v)})')
"

claude-haiku-4-5-20251001      mean delta +0.126 (n=5)
claude-opus-5                  mean delta +0.083 (n=4)

Example from the development notes, September 11, 2026. Here n counts basin-update entries; the 40-update keyword tally in the mechanics notes uses a separate sample. Model identifiers are recorded with each run so the examples retain their original context.

Choosing models

Summarization and update generation both interpret the conversation. They can use different models to explore cost and quality, but summary choices must be controlled when measuring the update stage.

ATTRACTOR_SUMMARY_MODEL=claude-haiku-4-5-20251001 \
ATTRACTOR_UPDATE_MODEL=claude-opus-5 \
  ./dist/attractor.mjs ingest --session

The hosted Worker reads the same two names from wrangler.toml. Defaults live in one place, DEFAULT_MODELS in src/engine.ts.

Project overview · All projects