Model Routing — Architecture
README · ARCHITECTURE · TECHNICAL · source
The resolution chain
A call site names an intent. Two lookups turn that into a model id and a parameter set.
Two indirections, absorbing two different kinds of change:
- Profile → tier absorbs a change in what a call is for. If gossip should get
smarter,
chattermoves tomediumin one line and every gossip call follows. - Tier → model id absorbs a change in what models exist. A new Haiku ships,
DEFAULT_MODELS.smallchanges, and nothing else does.
The env override on top means a deployment can pin models without a code change —
useful for holding a deployment steady while testing a model elsewhere.
Why the tiers are named by size
small, medium, large are deliberately unglamorous. The tier says only where a
call sits on the cost/capability curve; the profile carries the intent. So the tier
names make no claim that could age badly, and they are unambiguously ordered.
The alternative — naming tiers haiku, sonnet, opus — reads better right up
until the day haiku points at Sonnet, at which point every call site is
misleading and the indirection has become worse than nothing.
The single call site
Both failure paths exist because of specific debugging pain. A bare status code turns a one-line fix into a guessing game, so the API's own error type and message are surfaced. And when no text block is found, the error lists the block types that were present — which is what turns "reasoning model returned a thinking block" from a mystery into a sentence.
content.find(b => b.type === "text") rather than content[0] is the whole fix
for the reasoning-block bug, and it is one line. The comment above it is longer
than the line, because the line looks arbitrary without it.
Provider routing in the orchestrator
Multi-Provider LLM Orchestrator, originally Journal Club, reuses this adapter for both the reading room and the pilot.
Four providers, two code paths, because three of them speak an OpenAI-compatible
dialect. The interesting work is not the branch — it is the Generation dataclass
every path normalises into, which is covered in the
Systems Logging and Telemetry.
A provider exception does not propagate. It becomes a Generation with
finish_reason="error", so a failed call is still a row in the dataset rather than
a gap in it. In an experiment, a missing row and a failed row are very different
facts.
Queue topology
Selector jobs do no work. They run one query, cap the result, and enqueue one worker job per row. Three properties follow:
- Each job handles one item rather than the full corpus. Its completion still depends on item size, provider latency, and runtime limits.
- A failure retries one conversation, not twenty.
- The cap (
LIMIT 20) bounds the cost of any single cron tick, so a long-idle system catches up over several ticks instead of spending everything at once.
Summarization is also the root of a fan-out: one summarize_conversation enqueues
the branch update, the attractor update and the embedding job. Those three are
independent and can fail independently.
Delivery semantics
The consumer is four lines of real logic:
for each message in batch:
try process(message.body); message.ack()
catch log(type, attempts, error); message.retry()
This is at-least-once delivery, so the jobs are designed to tolerate
redelivery. Current operations use INSERT OR REPLACE on conversation-branch
links, find-or-create on branches, and repeatable R2 deletes to reduce duplicate
effects. Retry safety still depends on each job and its storage operation, so it
needs to be checked when a job changes.
The attempt count is logged on every failure, which helps distinguish a first failure from a repeatedly failing job. The public tradeoff is simple: at-least-once delivery keeps the consumer small, but moves retry safety into each job. TECHNICAL.md describes that boundary.
Access boundaries
The backend separates ordinary asset delivery from authenticated application routes. Model-calling paths use the authenticated, rate-limited boundary; deployment-specific routes and thresholds stay with the private operational notes.
Snapshot scope
This page describes model selection, provider handling, and queue dispatch in the workspace and orchestrator snapshots. It documents the implementation shape, not load-test results or a replay of queued work.