Multi-Provider LLM Orchestrator — Technical
README · ARCHITECTURE · TECHNICAL · source
The pilot was as much about learning how to measure the responses as it was about comparing them.
Keep the task in the control
All four conditions retain the shared reading task. The two factors add an identity anchor, persona text, or both. Persona prompts already name the models, so this is an added-anchor comparison, not identity versus no identity.
The persona also adds prompt length. The pilot compares those prompt packages; separating length from persona content would need another experiment. The runner composes existing prompt parts without changing the bot's prompts.
What I measured
| Measurement | How it is collected |
|---|---|
| Response length and structure | Character counts, word-like counts, and sentence/paragraph heuristics |
| Style and formatting | Counts of punctuation, headings, lists, emphasis, and code spans |
| Lexical diversity | Unique word-like tokens divided by total word-like tokens |
| Identity claims | Regular expressions for self-statements and opening bylines |
| Completion information | Error and completion flags, plus provider metadata when available |
These describe text, not whether an answer is correct. Word-like counts are not provider tokenizer counts.
What the saved records show
The initial run has 360 rows: three models, four conditions, ten articles, and three planned generations per cell. It records 24 Claude errors and one Grok error. Replacing the Claude arm with its 120-response rerun leaves 359 non-error responses. The ten saved article records match between runs.
For each model and article, compare the average of the two persona-present conditions with the average of the two persona-absent conditions, then weight each article equally.
| Model | Average additional characters with persona | Articles with a positive difference |
|---|---|---|
| Claude | 190.45 | 10/10 |
| GPT | 119.70 | 10/10 |
| Grok | 40.58 | 8/10 |
Repeated generations show variation within a cell; articles show variation across this stimulus set. They are not ten independent replications of the whole study. These calculations are descriptive and do not include significance testing or reliability estimates.
Thinking blocks and missing metadata
The initial file has no populated thinking or reasoning metadata in its saved fields. The Claude rerun records thinking-block presence on 10 of 120 responses. Thinking character counts are blank on those ten rows and zero on the other 110. Neither file has populated reasoning text or reasoning-token counts.
Those ten block observations are separate from the earlier 24 parser errors. Missing metadata stays unknown, and a returned block is not a general measure of a model's reasoning.
Save the response, improve the measurement
The original identity matcher missed bylines such as "Grok here." Adding a byline pattern made those introductions detectable. Later review showed that quoted self-statements can still trigger the matcher, so annotated examples are the next useful check.
Saving responses alongside measurements lets recompute.py rescore existing text without making new model calls. The public v0.1.0 release also preserves recorded error flags during rescoring and rejects an existing experiment output directory.
Keeping provider calls moving
Both workflows use the same provider adapter, with reused clients, a 120-second request timeout, and compatibility handling for token-limit parameter names. The experiment allows four calls in flight. That bounds concurrency; it does not guarantee that every provider's rate limit will be avoided.
The original implementation and public v0.1.0 release are different checkpoints. The public edition requires an explicit owner command to start scheduled work. Its release checks use synthetic responses; they do not claim live model calls or a live Discord integration test.