Model Routing and Job Orchestration — source

Projects / model-routing-code-excerpts-readme

Model Routing and Job Orchestration — source

model-routingrouting-and-orchestration

The routing module, profiles, call handling, and queue consumer from the two systems described in this study.

README · ARCHITECTURE · TECHNICAL · source

Find an excerpt

1. The routing module's reason for existing

The docstring is the design document. It records three bugs found in duplicated call sites, each easy to miss while the models in use were older.

/**
 * One place that talks to the Anthropic Messages API.
 *
 * Three bugs were duplicated across every hand-rolled call site in this
 * worker, all of them latent while the models in use were older:
 *
 *  1. `temperature` was sent on every request. Newer models reject it
 *     outright with a 400 ("`temperature` is deprecated for this model"),
 *     so the app would break the moment a model was updated.
 *  2. Replies were read as `content[0].text`. Reasoning models return a
 *     thinking block first, which has no `text` field, so every reply parsed
 *     as an empty string.
 *  3. Failures threw a bare status code, which turns a one-line fix into a
 *     guessing game.
 *
 * Fixing them in one place means the next model change is a config edit
 * rather than a hunt through call sites.
 */

Worth noticing what these three have in common: each is triggered by upgrading a model, the kind of change that is easy to treat as routine. Centralizing the call boundary makes that change easier to inspect and fix once.

2. Tiers, named by size on purpose

/**
 * Models are chosen by size, not by name.
 *
 * Deliberately unglamorous. The tier says only where a call sits on the
 * cost/capability curve — the `CallProfile` below carries the intent — so
 * these names make no claim that could age badly and are unambiguously
 * ordered. They are also, pointedly, not named after the models they map to:
 * naming a tier `haiku` would make it a lie the day it points at Sonnet, and
 * the indirection is the entire reason this exists.
 */
export type ModelTier = "small" | "medium" | "large";

export const DEFAULT_MODELS: Record<ModelTier, string> = {
  small: "claude-haiku-4-5-20251001",
  medium: "claude-sonnet-5",
  large: "claude-opus-5",
};

/** Resolve a tier, letting a wrangler var override it per deployment. */
export function model(env: Env, tier: ModelTier): string {
  const override = {
    small: env.SMALL_MODEL,
    medium: env.MEDIUM_MODEL,
    large: env.LARGE_MODEL,
  }[tier];
  return override || DEFAULT_MODELS[tier];
}

The env override is what makes this safe to change. A model swap can be tried on one deployment, and rolled back, without touching the code that chose the tier.

3. Profiles: intent, because the parameters are coupled

/**
 * What a call is *for*, rather than what parameters it uses.
 *
 * Model tier and temperature are not independent choices — temperature is only
 * accepted by the small tier, so a high temperature on `analysis` would be
 * silently dropped. Naming the purpose keeps the valid combinations together
 * and puts the reason at the call site: a reader sees "chatter" and knows the
 * variety is deliberate, not a leftover.
 *
 *   chatter     idle conversation, gossip. Variety is the feature.
 *   extraction  pull structured data out of text. Cheap and repeatable.
 *   analysis    judgement about what a conversation meant.
 */
export type CallProfile = "chatter" | "extraction" | "analysis";

export const PROFILES: Record<CallProfile, { tier: ModelTier; temperature?: number }> = {
  chatter: { tier: "small", temperature: 0.9 },
  extraction: { tier: "small" },
  analysis: { tier: "large" },
};

export function profile(env: Env, name: CallProfile): { model: string; temperature?: number } {
  const p = PROFILES[name];
  const override = name === "chatter" ? env.CHATTER_TEMPERATURE : undefined;
  const temperature = override !== undefined ? Number(override) : p.temperature;
  return {
    model: model(env, p.tier),
    ...(temperature !== undefined && !Number.isNaN(temperature) ? { temperature } : {}),
  };
}

"Silently dropped" is the operative phrase. Setting a temperature on a model that does not accept it produces no configuration error; the parameter is omitted. Coupling the choices into one named profile makes the intended combination visible at the call site.

The Number.isNaN guard omits a CHATTER_TEMPERATURE value that parses as NaN. It does not validate range or reject every non-finite number, so configuration validation remains limited.

4. A denylist, so unknown means new

/**
 * Whether a model still accepts a `temperature` parameter.
 *
 * Newer models reject it with a 400. This is deliberately a denylist of
 * families rather than an allowlist of ids, so an unknown model is assumed
 * to be new and temperature is dropped rather than breaking the request.
 */
export function supportsTemperature(model: string): boolean {
  return /haiku-4|sonnet-4|opus-4/.test(model);
}

One line, and the direction of the default is the whole design. An allowlist can block a new model until someone updates it. The denylist instead omits an optional sampling control from an unrecognized model; that omission is safer for request compatibility, but it still deserves inspection when models change.

The list will need pruning eventually — it names families, so it grows by one entry per generation that still accepts temperature, and shrinks to nothing when none do.

5. The call, and both ways it can fail

export async function callClaude(env: Env, opts: ClaudeCallOptions): Promise<string> {
  const response = await fetch("https://api.anthropic.com/v1/messages", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "x-api-key": env.ANTHROPIC_API_KEY,
      "anthropic-version": "2023-06-01",
    },
    body: JSON.stringify({
      model: opts.model,
      max_tokens: opts.maxTokens,
      system: opts.system,
      messages: opts.messages,
      // Dropped silently for models that reject it, so a tier change can't
      // turn a working call into a 400.
      ...(opts.temperature !== undefined && supportsTemperature(opts.model)
        ? { temperature: opts.temperature }
        : {}),
    }),
  });

  if (!response.ok) {
    let detail = "";
    try {
      const body = (await response.json()) as { error?: { type?: string; message?: string } };
      detail = body.error?.message ? ` — ${body.error.type}: ${body.error.message}` : "";
    } catch {
      detail = "";
    }
    throw new Error(`Claude API ${response.status}${detail}`);
  }

  const result = (await response.json()) as {
    content: Array<{ type: string; text?: string }>;
  };

  // Not content[0]: reasoning models put a thinking block there.
  const text = result.content.find((b) => b.type === "text")?.text;
  if (text === undefined) {
    throw new Error(
      `No text block in reply (blocks: ${result.content.map((b) => b.type).join(", ") || "none"})`,
    );
  }
  return text;
}

Two error messages built for the person reading them at 2am:

The API's own type and message are unwrapped into the thrown error, so a 400 says what was wrong rather than only that something was.
When no text block is found, the error names the block types that were present. That one detail is the difference between "empty reply, no idea" and "it returned a thinking block."

The error-parsing itself is wrapped in try/catch, because a failing API is exactly the situation where the error body might not be JSON.

6. The queue consumer

Four lines of logic, and the semantics they imply.

export async function handleQueue(batch: MessageBatch<QueueJob>, env: Env): Promise<void> {
  for (const msg of batch.messages) {
    try {
      await processJob(msg.body, env);
      msg.ack();
    } catch (err) {
      console.error(`Job failed (attempt ${msg.attempts}):`, msg.body.type, err);
      msg.retry();
    }
  }
}

At-least-once delivery means a job can be run again. The jobs are designed to tolerate that through operations such as INSERT OR REPLACE on conversation-branch links, find-or-create on branches, and repeatable R2 deletes. Those patterns reduce duplicate effects, but each operation still needs its own redelivery check.

msg.attempts is logged so the failure line distinguishes an early failure from repeated attempts. That makes the retry pattern visible in the current logs.

Sequential rather than parallel, deliberately: several of these jobs make model calls, and a batch fanned out in parallel would multiply the concurrent spend by the batch size.

7. Provider routing, and failure as data

async def generate_detailed(
    config, model, system_prompt, user_message,
    max_tokens: int = 2048,
    max_len: int | None = MAX_RESPONSE_LEN,
) -> Generation:
    """Generate a response, routing to the correct SDK.

    max_len=None disables our own truncation. The experiment passes None: the
    1900-char cut exists for Discord, and applying it to data written to CSV
    would censor response length, which is one of the measures.
    """
    try:
        if model.provider == "anthropic":
            text, reason, extra = await _call_anthropic(
                config, model, system_prompt, user_message, max_tokens)
        else:
            text, reason, extra = await _call_openai_compat(
                config, model, system_prompt, user_message, max_tokens)
    except Exception as e:
        log.exception("Error calling %s: %s", model.name, e)
        return Generation(
            f"*[{model.display_name} is having a moment and couldn't respond: {type(e).__name__}]*",
            "error", 0, False)

Two decisions worth separating.

max_len=None for the experiment. A 1900-character truncation exists because Discord has a message limit. Applying it to data written to CSV would censor response length — which is one of the measures. A presentation constraint leaking into a measurement is the kind of bug that produces a confident, wrong finding.

An exception becomes a Generation, not a raised error. A failed call is still a row, carrying finish_reason="error". In an experiment, a missing row and a failed row are different facts, and only one of them is recoverable at analysis time. This is what made run 1's condition-correlated missingness visible rather than silent.