Systems Logging and Telemetry - Source

Projects / runtime-observability-code-excerpts-readme

Systems Logging and Telemetry - Source

runtime-observabilityobservability-and-subsystems

The small pieces that collect model information and keep response facts from being flattened together.

README · ARCHITECTURE · TECHNICAL · source

Find an excerpt

1. The comment that is the whole point

From the Generation dataclass. Longer than the fields it documents, and earning it.

@dataclass(frozen=True)
class Generation:
    """A response plus the facts you need to know whether it was cut short."""
    text: str
    finish_reason: str          # "stop" | "length" | "error" | provider value
    raw_chars: int              # length before any truncation of ours
    truncated_by_us: bool       # did max_len clip it
    # Extended thinking. None means "this provider does not report it", which
    # is not the same as zero.
    #
    # Presence and content are separate facts. The Anthropic API returns a
    # ThinkingBlock carrying a `signature` but an EMPTY `thinking` string — the
    # reasoning content is not exposed. So the number of blocks is observable
    # and its length is not. Deriving presence from length (`chars > 0`) reads
    # every thinking response as not-thinking, which is the bug this comment
    # exists to prevent recurring.
    thinking_blocks: int | None = None
    thinking_chars: int | None = None   # None = present but not exposed

    # Provider-native reasoning metadata. None means not reported. These values
    # are useful response facts but are not assumed comparable across APIs.
    reasoning_tokens: int | None = None
    # Provider-exposed reasoning text, where available. None means not exposed,
    # never "did not reason".
    reasoning_text: str | None = None

    @property
    def had_thinking(self) -> bool | None:
        """Did the model reason at all. Prefers the token count, which is
        reported even where blocks and text are not."""
        if self.reasoning_tokens is not None:
            return self.reasoning_tokens > 0
        return None if self.thinking_blocks is None else self.thinking_blocks > 0

    @property
    def hit_token_cap(self) -> bool:
        return self.finish_reason == "length"

Three things are being defended here.

None versus 0. Every optional field distinguishes "the provider does not report this" from "the provider reported none." Collapsing them turns an absence of information into a confident negative measurement, and then averages it with real zeros from providers that do report.

had_thinking returns bool | None. A tri-state property in a language where it would be easier to return False. It prefers reasoning_tokens when that field is available, falls back to block count, and returns None when neither exists — rather than guessing.

The bug it prevents. Deriving presence from thinking_chars > 0 reads every Anthropic thinking response as not-thinking, because the content is withheld while the block is present. That is a measure reading floor for one provider and correctly for others, which would introduce an artefact into a cross-provider comparison.

2. Tracking what actually answered

// Model detection and tracking for proxied LLM responses.
// Tracks which model actually responded (vs. what was requested).

/** Track a model response. Call after every proxied API response. */
export async function trackModel(
  kv: KVNamespace,
  modelUsed: string,
  usage: { input_tokens?: number; output_tokens?: number } = {}
): Promise<void> {
  const now = Date.now();
  const family = modelFamily(modelUsed);
  const provider = detectProvider(modelUsed);

  const previous = await kv.get("model:current", { type: "json" }) as ModelCurrent | null;

  await kv.put("model:current", JSON.stringify({
    model: modelUsed, family, provider,
    timestamp: now, isoTime: new Date(now).toISOString(),
  }));

  const counts = (await kv.get("model:counts", { type: "json" }) as FamilyCounts) || {};
  if (!counts[family]) {
    counts[family] = { requests: 0, inputTokens: 0, outputTokens: 0, lastSeen: 0, provider };
  }
  counts[family].requests += 1;
  counts[family].inputTokens += usage.input_tokens || 0;
  counts[family].outputTokens += usage.output_tokens || 0;
  counts[family].lastSeen = now;
  counts[family].provider = provider;

  await kv.put("model:counts", JSON.stringify(counts));

  if (previous && previous.model !== modelUsed) {
    await logModelChange(kv, previous.model, modelUsed, now);
  }
}

The first comment line is the design: what responded, not what was requested. Those diverge when an alias resolves to a dated snapshot, when a fallback fires, or when an account is migrated — and none of those announce themselves.

The change log is conditional, so a thousand identical calls write nothing to it. That is what keeps a drift log small enough to hold in a single KV value.

3. Extraction, with four provider shapes and no throw

/** Extract model identifier from a parsed JSON response body. */
export function extractModelFromBody(body: Record<string, unknown>): string {
  if (!body || typeof body !== "object") return "unknown";
  if (body.model) return body.model as string;
  if (body.modelVersion) return body.modelVersion as string; // Google Gemini
  if (body.meta && (body.meta as Record<string, unknown>).model)
    return (body.meta as Record<string, unknown>).model as string; // Cohere
  return "unknown";
}

/** Extract model from an SSE event's data payload (streaming responses). */
export function extractModelFromSSE(eventData: string): string {
  try {
    const parsed = JSON.parse(eventData);
    // Anthropic streaming: message_start contains the full message object
    if (parsed.type === "message_start" && parsed.message) {
      return parsed.message.model || "unknown";
    }
    // OpenAI / Mistral / xAI / DeepSeek streaming
    if (parsed.model) return parsed.model;
    // Anthropic non-streaming
    if (parsed.type === "message" && parsed.model) return parsed.model;
    return "unknown";
  } catch {
    return "unknown";
  }
}

Every path falls through to "unknown" rather than throwing. That is deliberate and slightly uncomfortable: it means unparseable responses accumulate silently in an unknown bucket.

The alternative is worse. Throwing here would drop the response entirely and bias the counters toward the providers whose shapes happen to be easy to parse — so the usage numbers would be most wrong about exactly the providers you understand least.

4. Families, so a dated snapshot does not fragment history

/**
 * Normalize model string to a short family name.
 * Claude: "claude-opus-4-6" -> "opus", "claude-sonnet-4-20250514" -> "sonnet"
 * Others: returned as-is with trailing date suffixes stripped.
 */
export function modelFamily(model: string): string {
  const m = (model || "").toLowerCase();
  if (m.includes("opus")) return "opus";
  if (m.includes("sonnet")) return "sonnet";
  if (m.includes("haiku")) return "haiku";

  const dateMatch = m.match(/^(.+?)-\d{4}-?\d{2}-?\d{2}$/);
  if (dateMatch) return dateMatch[1];
  return model || "unknown";
}

The Claude branch is a substring check and the general branch is a date-suffix strip. The second rule is looser and will be wrong on a provider that versions differently — it is a guess about naming conventions that are not contracts.

Both exist to answer the question worth asking of a usage counter: how much am I spending on large-tier calls, not how much on the 2024-08-06 build.

5. Operational boundaries

The public excerpts stop at signal capture and response normalization. The storage jobs follow a narrower rule: cleanup should act only on data it can identify confidently, while backups stay separate from telemetry because they preserve application data rather than describe system behavior.

Rate limiting is similarly modest. It provides approximate friction around routes that spend money; it is not presented as an exact quota or an authorization system. The detailed failure-path review remains in the private engineering notes.