Selected parts of the reading room and experiment harness, including prompt composition and a measurement correction.

README · ARCHITECTURE · TECHNICAL · source

Find an excerpt

These selected excerpts preserve a historical private snapshot. The public Multi-Provider LLM Orchestrator release includes later safety and data-preservation changes; the technical notes separate current interpretation from the original implementation account.

1. The design, in code

conditions.py defines the prompt combinations. The excerpt preserves the original design notes; the run record explains later refinements to their interpretation.

"""The 2x2 prompt-composition conditions for the Journal Club experiment.

Two factors, crossed:

                  | no persona      | persona
    --------------+-----------------+------------------
    no identity   | baseline        | persona_only
    identity      | identity_only   | full  (production)

Four cells, which is the whole design — with only two factors a full factorial
costs no more than a minus-one and additionally gives the interaction, i.e.
whether the persona behaves differently when the identity anchor is present.

The important property is that the **task framing is constant across all four
cells**. `_READING_SHARED` (you are in a journal club, react like a person, keep
it short) appears in every condition. Only the identity anchor and the per-model
voice vary. An empty system prompt would not be this experiment's control — it
would remove the task as well as the persona, and change what is being measured.
That is the difference between an FMO and an unstained sample.

Nothing here modifies personas.py; the prompts are composed from its parts.
"""

CONDITIONS = ("baseline", "identity_only", "persona_only", "full")

#: Which factor is present in each cell, for analysis.
FACTORS: dict[str, dict[str, bool]] = {
    "baseline":      {"identity": False, "persona": False},
    "identity_only": {"identity": True,  "persona": False},
    "persona_only":  {"identity": False, "persona": True},
    "full":          {"identity": True,  "persona": True},
}


def system_prompt(model_name: str, condition: str) -> str:
    """Build the system prompt for one cell of the design."""
    if condition not in FACTORS:
        raise ValueError(f"unknown condition {condition!r}; expected one of {CONDITIONS}")

    f = FACTORS[condition]
    # The persona variants already contain _READING_SHARED; the non-persona
    # variants use it alone, so the task is identical in all four cells.
    body = _READING_PROMPTS[model_name] if f["persona"] else _READING_SHARED

    if f["identity"]:
        return f"{IDENTITY[model_name]}\n\n{body}"
    return body


def user_message(title: str, author: str | None, url: str, content: str) -> str:
    """The article prompt. Identical across all conditions and models."""
    by = f" by {author}" if author else ""
    return (
        f"Here's a new LessWrong post to discuss:\n\n"
        f"**{title}**{by}\n{url}\n\n"
        f"---\n\n{content[:4000]}\n\n"
        f"You're first to respond. Set the tone."
    )

An FMO, or fluorescence-minus-one control, leaves out one marker while keeping the rest of the assay setup. The analogy here is removing one prompt component without also removing the shared task.

system_prompt composes from personas.py and never modifies it. The production bot and the full condition therefore run the same system prompt, which is what makes the experiment about the shipping system rather than a separate lab replica.

2. A clean null that was a broken instrument

This excerpt shows the original matching rule and the added byline pattern. The comments retain the observations that prompted the correction.

MODEL_WORDS = {
    "claude": ("claude",),
    "gpt": ("gpt", "chatgpt"),
    "gemini": ("gemini", "bard"),
    "grok": ("grok",),
}

# "I'm Claude", "I am GPT", "as Claude", "this is Grok"
_SELF_ID = re.compile(
    r"\b(?:i am|i'm|as|this is|speaking as)\s+(claude|chatgpt|gpt|gemini|bard|grok)\b",
    re.IGNORECASE,
)

# The declarative form above missed the way models actually sign on in chat.
# In run 1 it scored 335/335 responses as "none" while the corpus contained
# "Grok here.", "Grok:" and "GPT here:" — concentrated in the identity-bearing
# conditions, i.e. exactly the effect the measure exists to detect. A measure
# that reads floor in every cell looks like a clean null and is really a broken
# instrument.
#
# Anchored to the start of the response on purpose. A bare model name anywhere
# in the text is not a self-claim: a Grok response quoted a paper's "claude is
# better" condition label, which an unanchored pattern would score as Grok
# claiming to be Claude — a false "wrong" in the one measure whose whole point
# is counting wrong claims.
_SELF_BYLINE = re.compile(
    r"^[\s*_>#]*(claude|chatgpt|gpt|gemini|bard|grok)\b\s*(?:here\b|[:,—-])",
    re.IGNORECASE,
)

Two decisions are visible here. The first is noticing that a null result in every cell is suspicious when the raw text visibly contradicts it. The second is the anchoring: the obvious fix — match a model name anywhere — would have introduced false positives into the exact measure whose job is counting false identity claims. A looser instrument is not a better one.

Because raw responses are stored in the CSV, fixing this was a re-run of recompute.py over existing data rather than a re-run of the experiment.

3. The harness contract

"""Run the 2x2 prompt-composition experiment and write a CSV for analysis.

    python -m src.experiment --papers 5 --reps 3

Design: 4 conditions x 4 models x N papers x R replicates. Generation is
stochastic, so repeated generations make within-cell variation visible. Three
generations do not by themselves establish a condition effect.

This never touches Discord and never writes to the production database. Papers
are scraped into a throwaway database inside the output directory, so the club's
own state is untouched by the experiment.

Output: one row per generation, with the raw response and the computed measures,
in `experiment_runs.csv`.
"""

# How many generations may be in flight at once. Modest, to stay clear of
# per-provider rate limits without making a 240-call run take an hour.
CONCURRENCY = 4

FIELDS = [
    "run_id", "timestamp", "paper_id", "paper_title", "model", "model_id",
    "condition", "has_identity", "has_persona", "rep",
    "article_chars", "article_chars_used",
    "chars", "words", "sentences", "paragraphs", "mean_sentence_words",
    "type_token_ratio", "bold_spans", "italic_spans", "headers", "bullets",
    "numbered", "code_spans", "questions", "exclamations", "em_dashes",
    "ellipses", "commas_per_sentence", "identity_claim", "identity_claimed_as",
    "is_error", "finish_reason", "hit_token_cap",
    "had_thinking", "thinking_blocks", "thinking_chars",
    "reasoning_tokens", "reasoning_chars", "reasoning_text", "response",
]

has_identity and has_persona are written as their own columns rather than derived from condition at analysis time. It costs two columns and removes a whole class of analysis error.

is_error, finish_reason and hit_token_cap are in the row because a failed generation has to be distinguishable from a short one. That distinction is what surfaced the condition-correlated missingness in run 1.

4. Infrastructure lessons, as constants

These constants reflect failures encountered while running the harness.

# One client per provider, reused, so repeated generations do not keep opening
# new connection pools.
_CLIENTS: dict[str, object] = {}

# An explicit timeout prevents a stalled request from holding one of the four
# available semaphore slots indefinitely. The runner can fail and retry instead
# of leaving every slot occupied.
REQUEST_TIMEOUT_S = 120.0
MAX_RETRIES = 3

# Which token-limit parameter each model accepts, learned on first call.
_TOKEN_PARAM: dict[str, str] = {}

5. Not dropping the end of a message

A small function, included because the bug it fixes is the kind that destroys data quietly.

DISCORD_LIMIT = 1990  # Discord's hard per-message limit is 2000; leave a little room.


def chunk(content: str, limit: int = DISCORD_LIMIT) -> list[str]:
    """Split content into Discord-sized pieces, preferring clean breaks.

    Paragraph breaks first, then line breaks, then sentence ends, and only
    mid-word as a last resort. The bot used to hard-truncate at 1900 instead,
    which silently dropped the end of anything longer — including the approval
    footer on a long draft.
    """

The approval footer was at the end. Long drafts arrived unapprovable and nothing reported an error.

Read next

• Architecture
• Project overview

Project overview · All projects