Orchestrator vs models alone
The numbers behind the landing's "Measured" section and its flow. bun efficiency (apps/benchmark): 14 shop tasks (7 in English and Russian) on a seeded mock shop, judged by the cart and the shown product, not by the reply's words. Unless a report says local, every model, the orchestrator's too, ran hosted through OpenRouter, so times include the network.
Reports
The orchestrator runs were redone after its delta prompt's examples were moved off every benchmark's domain (they had shared a jacket-in-size-M case and the mock shop's answer layout); the earlier reports are withdrawn.
- Orchestrator, shop intent set (and voxtral-small-24b alone), 3 passes; code steps timed to the microsecond; delta fills a list's searches at the same time (
FUNCTION_CALL_MAX_CONCURRENCY=8) - Orchestrator, no intent set, 3 passes, the same setting
- Before delta's searches ran at the same time: shop intent set, no intent set (each with voxtral-small-24b alone), 3 passes
- mistral-medium-3.1 alone, 3 passes
- gpt-6-luna and deepseek-v4-flash alone, 3 passes
- Orchestrator, every model local on one RTX 2060 (6 GB), no API calls, 3 passes: alpha, beta and delta on QVikhr-3-4B Q4_K_M (llama.cpp
server-vulkan-b11176, KV cache q8, thinking off), gamma on d3-lite (its ownd3_server.py, CUDA). The "beta alone" column of this report is not part of the comparison. The card holds two models besides d3, so delta was QVikhr in that run; then Arch-Function-3B Q6_K as delta:bun replay --delta arch-function-3basked the run's 60 delta calls again on it (same prompt and schema, answers through the same repair) and all 60 matched, so the runs stay the same; the time replays every model call warm, grouped by model, with no model swap. The landing's local row is this one. Arch-Function's license needs a separate grant for commercial use: here it is only benchmarked - Single-pass screens: claude-haiku-5.5, gpt-6-luna, gemini-2.5-flash-lite, deepseek-v4-flash, mistral-small-2603, qwen3.7-flash, mistral-small-3.2, mistral-medium-3.1, mistral-large-2512
Reproduce
In apps/benchmark, with .env from .env.example:
For the orchestrator: the stack running, a web channel's key and origin in
ORCHESTRA_*, and the widget's limits raised in the orchestrator's env (WIDGET_THREADS_PER_HOURand the otherWIDGET_*limits), then its container recreated.The orchestrator and its reply model alone:
bun efficiency --runs 3bashHosted models alone, straight to OpenRouter (
OPENROUTER_API_KEY):bun efficiency --runs 3 --models mistralai/mistral-medium-3.1,openai/gpt-6-luna,deepseek/deepseek-v4-flashbashLocal models: serve them OpenAI-compatible (llama-server) and d3 on
/v1/systemone, add them as a provider in the admin panel and pick them for the agents, then run step 2.A local model that does not fit next to the others:
REPLAY_BASE_URL=… bun replay --dir <run folder> --delta <model>(models behind a llama-server router).
Each run writes results/efficiency/<timestamp>/: report.txt (the files above) and one JSON file per task run with every model call and tool call.