Orchestrator vs models alone

The numbers behind the landing's "Measured" section and its flow. bun efficiency (apps/benchmark): 14 shop tasks (7 in English and Russian) on a seeded mock shop, judged by the cart and the shown product, not by the reply's words. Unless a report says local, every model, the orchestrator's too, ran hosted through OpenRouter, so times include the network.

Reports

The orchestrator runs were redone after its delta prompt's examples were moved off every benchmark's domain (they had shared a jacket-in-size-M case and the mock shop's answer layout); the earlier reports are withdrawn.

Reproduce

In apps/benchmark, with .env from .env.example:

  1. For the orchestrator: the stack running, a web channel's key and origin in ORCHESTRA_*, and the widget's limits raised in the orchestrator's env (WIDGET_THREADS_PER_HOUR and the other WIDGET_* limits), then its container recreated.

  2. The orchestrator and its reply model alone:

    bun efficiency --runs 3
    bash
  3. Hosted models alone, straight to OpenRouter (OPENROUTER_API_KEY):

    bun efficiency --runs 3 --models mistralai/mistral-medium-3.1,openai/gpt-6-luna,deepseek/deepseek-v4-flash
    bash
  4. Local models: serve them OpenAI-compatible (llama-server) and d3 on /v1/systemone, add them as a provider in the admin panel and pick them for the agents, then run step 2.

  5. A local model that does not fit next to the others: REPLAY_BASE_URL=… bun replay --dir <run folder> --delta <model> (models behind a llama-server router).

Each run writes results/efficiency/<timestamp>/: report.txt (the files above) and one JSON file per task run with every model call and tool call.