Multi-Provider LLM Orchestrator — Architecture
README · ARCHITECTURE · TECHNICAL · source
Two programs, one codebase
The Discord bot and the experiment harness use the same personas, the same model clients, and the same scraper. They do not share state, and that separation is deliberate.
experiment.py never touches Discord and never writes to the production database.
Papers are scraped into a throwaway database inside the output directory, so the
club's own state is untouched by a run. An experiment that can corrupt the thing it
measures is not a control condition.
The reading room cycle
The club is started and stopped with !club start / !club stop rather than by
killing the process. Every cycle spends real money across four providers, so being
able to halt it without losing the bot mattered more than it sounds like it should.
Each model posts through its own webhook so it appears in the channel under its own name and avatar. That is presentation, but it has a measurement consequence worth naming: speaking position in the club corpus is confounded with model identity, which is one reason the experiment generates fresh responses rather than analyzing the club transcript.
The experiment run
One row per generation, with the raw response stored alongside the computed
measures. Keeping the raw text is what made the broken identity measure fixable
after the fact: the fix was a re-run of recompute.py over stored responses, not a
re-run of the experiment.
CONCURRENCY = 4 bounds the number of in-flight requests. Provider rate limits
still depend on the account, model, and request size.
Measurement design
Text measurements and generation metadata are kept together in each result row:
| Family | Measures |
|---|---|
| Structure | chars, words, sentences, paragraphs, mean sentence words |
| Formatting | bold spans, italics, headers, bullets, numbered lists, code spans |
| Punctuation | questions, exclamations, em dashes, ellipses, commas per sentence |
| Identity and metadata | identity claim, who it claimed to be, type–token ratio, reasoning tokens and blocks, finish reason, token-cap hit |
measures.py is separate from the runner and reused by recompute.py to
rescore saved experimental responses. The Discord exporter does not automatically
run these measurements; applying them to club transcripts would need a separate step.
Provider observability differs, and that difference is itself recorded: some providers return reasoning token counts, some return reasoning text, some return neither. A measure that only exists for one provider cannot carry a cross-model claim.
Provider handling
models.py gives both workflows a shared place to handle provider clients,
request timeouts, and token-limit parameters. Keeping those differences here
makes the reading-room and experiment code easier to follow.
The technical notes explain the settings and why they changed. The source excerpts preserve the implementation comments.
Reproducibility and scope
The historical pilot retained raw responses and generation metadata, so its text measurements can be corrected and recomputed without making new model calls. Its single-turn comparisons describe response style; the reading room is the separate multi-model discussion workflow.
The public v0.1.0 release is a later checkpoint. It adds output-directory protection, preserves error flags during rescoring, and requires an explicit owner command to start scheduled work. Its synthetic release checks are separate from the private historical pilot records.