# SpeculatorArena protocol, version 1 SpeculatorArena measures how much faster a speculative decoder makes a fixed model serve real multi-turn traffic on one GPU, without changing what the model says. A submission is a speculator checkpoint and the engine that serves the model with it. In the same pod, the arena serves the same sessions with the stock engine and no speculation, and scores the submission by the speed it gains over that. This document is the contract: what a submission is, how it is measured and scored, and what breaks the rules. A change gets a new version number, and the values in force are served at `GET /api/protocol`. Values marked *calibrated* are set from the arena's own measurements (section 10) and published with the version that sets them. ## 1. Tasks | Task | Target model (BF16, fixed) | GPU | `MAX_MODEL_LEN` | |---|---|---|---| | `qwen3-8b` | `Qwen/Qwen3-8B` at `b968826d9c46dd6066d109eabc6255188de91218` | 1 × B200 | 40,960 (its native window) | | `qwen35-27b` | `Qwen/Qwen3.5-27B` at `fc05daec18b0a78c049392ed2e771dde82bdf654`, text only | 1 × B200 | 131,072 | Both are served in non-thinking mode (`enable_thinking: false`). The engine runs in a pinned image. Version 1 starts from LowBArena's (`vllm-openai` 0.25.1, commit `7a33ba95f`); a probe confirms that it serves every reference speculator of section 11, or picks an image that does. The target's weights, activations, and KV cache stay in BF16. The arena supplies the target, sealed read-only, and checks that what the engine serves is the target's own distribution (section 6). ## 2. A submission ```json {"task_id": "qwen3-8b", "split": "dev", "speculator": {"uri": "my-org/qwen3-8b-eagle3", "revision": ""}, "engine": {"inline": ""}, "solution": {"method": "EAGLE-3, 4 tokens", "kind": "eagle3"}} ``` Submitting takes an account's token (`Authorization: Bearer `, which the submit page and the kit's client send). Accounts are by invitation: the operators give each participant a token, shown once and kept only as its hash; when the operators open registration, `/register` hands one out. - **Speculator checkpoint**, optional: a Hugging Face repository at an exact commit, or (from a workspace on the arena's cluster) a directory of that workspace; none for a method without weights of its own (n-gram or suffix lookup, the target's own MTP layers). Only `*.json` and `*.safetensors` files are fetched, at most half the target's bytes. Python files are never fetched. - **Engine directory**: `launch.sh serve` serves the target with the speculator. Inline (a base64 `.tar.gz`, at most 2,800,000 characters) or a repository at an exact commit (at most 200 MiB), with an optional `plugin/` that is installed offline, as in LowBArena. - `solution.kind` labels the method (`ngram`, `suffix`, `draft`, `eagle3`, `mtp`, `other`); nothing is scored from it. ## 3. The engine's environment `launch.sh serve` runs as an unprivileged user, with no network and one GPU, in the pinned image: | Variable | Value | |---|---| | `TARGET` (and `ARTIFACT`, for engines written for LowBArena) | the sealed target checkpoint, read-only | | `SPECULATOR` | the sealed speculator checkpoint, read-only; an empty directory when none was given | | `TOKENIZER`, `GENERATION_CONFIG` | the target's directory | | `PORT`, `SERVED_NAME` | serve on `127.0.0.1:$PORT` under this name | | `MAX_MODEL_LEN` | the task's (section 1) | | `MAX_NUM_SEQS` | the most requests any workload keeps in flight (its lanes, section 4) | | `GPU_MEMORY_UTILIZATION` | 0.90: the share of the device the engine may hold (section 7) | | `PREFIX_CACHING` | `1`: every workload reuses prefixes, and the engine must keep a prefix cache | | `LANGUAGE_MODEL_ONLY` | `1` on `qwen35-27b` | The server must answer `/health`, `/v1/chat/completions` (streamed, honoring the sampling fields of section 4, `tools` with `tool_choice: "none"` rendered into the prompt, and `prompt_logprobs` with `continue_final_message`, section 6), and `/metrics`. It may report its acceptance: per request (vLLM's `metrics.speculative_decoding`) or, as the pinned vLLM does, in its `vllm:spec_decode_*` counters, which the arena reads around every pass. The arena shows acceptance and never scores it. ## 4. The timed workloads ### Sessions Every timed request belongs to a multi-turn session, built so that the prefix cache and the speculator meet what they meet in service: - **Structure.** Two families. `agent`: coding-agent sessions whose turn counts, prompt and reply sizes per turn, prefix reuse, and idle times follow recorded agent traces (AgentX's public cc-traces). `chat`: multi-turn chat sessions whose depth, growth per turn, reply length, think time, and shared prefixes follow aggregate statistics of production traffic. The mix of the two and the trace-derived distributions are published with the version; statistics derived from production traffic are kept by the operators and published only with the data owner's consent. - **Text.** Real and coherent. A session's user and tool turns come from one real conversation or agent trajectory in a public dataset (listed in the data card), chosen and cut to fit its structure. System prompts and tool schemas come from public agent frameworks and are shared by the sessions of one app, as production apps share theirs. - **Closed loop.** The model's reply joins the history before the next turn. A reply ends where the model ends it (no `ignore_eos`), within the turn's `max_tokens`. - **The window.** The client counts a prompt's tokens with the target's tokenizer, plus 5% and 128 tokens for what the chat template adds. When a prompt and its reply budget would not fit `MAX_MODEL_LEN`, the client cuts the history after its first message down to half the window, as clients compact theirs; the cut costs a cache miss, as it does in service. - **Sampling.** Timed requests use the model card's non-thinking sampling: `temperature` 0.7, `top_p` 0.8, `top_k` 20, and a seed per request. ### Lanes and interleave A workload runs `L` lanes. Each lane holds `m` live sessions and sends the next turn of whichever session's think time, capped at 300 s and scaled by `c`, has passed first, one request at a time (recorded traces hold idle gaps of hours, a user away; past the cap a session comes back as a returning session does). So `L` requests are in flight while `L × m` sessions hold prefixes in the cache, and an idle session loses part of its cache before it returns, as in service. `m` and `c` are *calibrated* so that the stock engine's cache hit (token-weighted and per request, from `usage.prompt_tokens_details.cached_tokens`) and the share of a returning session's history still cached match the targets in the data card. | Workload | Lanes `L` (`qwen3-8b` / `qwen35-27b`) | Interleave `m` | Speed | |---|---|---|---| | `solo` | 1 / 1 | 1 | the lane's output tokens per second of request time (sent to last token) | | `team` | 8 / 4 | 2 | output tokens per second while every lane is busy | | `busy` | 20 / 10: the live sessions fill about 85% of the stock engine's KV cache | 2 | output tokens per second while every lane is busy | `busy`'s lanes come from the stock engine's KV cache (1.05M tokens on `qwen3-8b`, 1.70M on `qwen35-27b`, at 0.90 of the device) and a live session's mean footprint over its life (21.7K and 73K tokens): enough sessions to fill the cache, not to overflow it, so that what a speculator holds costs evictions. Each workload runs three passes of fresh sessions, each pass for a fixed time (150 s on `solo`, 180 s on `team`, 240 s on `busy`); a request still streaming at the end may finish, within a grace period, and counts. No session, prompt, or shared prefix repeats across passes within an evaluation: each pass salts its shared prefixes, and no session serves twice. Each workload warms up with one wave at its own concurrency, from sessions no pass sends. The arena counts tokens itself, from the returned text with the target's tokenizer, and a reply counts only if its stream ends with `[DONE]` within its `max_tokens`. One start of the engine serves all three workloads. ### Pairing In the same pod, after the candidate's passes, every process of the engine's user is killed, the files it left in shared memory and scratch are removed, and once the device holds less than 2 GiB (within 120 s, or the evaluation is void and refunded) the stock engine (the pinned vLLM with `--enable-prefix-caching`, BF16, the same `GPU_MEMORY_UTILIZATION`, no speculation) serves the same passes: the same sessions, seeds, and order. A workload's speed ratio `r_w` is the median over its passes of the candidate's speed over the stock engine's on the same pass. ## 5. SpecScore ```text SpecScore = (r_solo · r_team · r_busy)^(1/3) if every gate of sections 6 and 7 passes, else 0 ``` The stock engine scores 1 against itself, up to the A/A noise of section 10. What a speculator holds on the device (its weights, its own KV cache, its buffers) comes out of the same budget as the target's KV cache. On `busy`, where the stock engine's cache is near full, that memory costs evictions and preemptions, and the measured speed shows it; there is no separate memory axis. The board ranks each account's last scored evaluation by SpecScore and shows the per-workload ratios, TTFT and TPOT percentiles, the cache hit, and the engine's own report of its acceptance. ## 6. The lossless gate Speculative decoding must not change the distribution the target samples from. Floating-point differences between a verify pass and a one-token decode can flip near-ties (in the probe, honest speculators' greedy replies agreed with the stock engine's for 31% to 85% of their length), so each check compares the candidate with the stock engine against a limit *calibrated* on A/A runs and on the honest reference speculators, at 95% confidence. `prompt_logprobs` computes logits for every prompt token, so checks 2 and 3 score a reply on the system prompt and the conversation's last 3,000 tokens (led by its first user message when they hold none), the reply cut to 2,000 characters, the same for both engines; and the checks run after the timed passes on a restart of each engine with `GPU_MEMORY_UTILIZATION` 0.80, which leaves room for those logits. An engine that cannot answer more than 10% of a check's requests fails the gate; if the stock engine cannot, the evaluation is void and refunded. 1. **Greedy agreement.** 64 probe prompts per evaluation, `temperature` 0, 256 tokens, one at a time. The share of each reply that agrees with the stock engine's before their first difference, averaged over the prompts, must reach the calibrated floor. 2. **Sampled likelihood.** 512 timed replies of each engine, drawn from all workloads, are scored by the stock engine under the target (`prompt_logprobs`): their mean log-probability per token, weighted by tokens, with the replies as clusters for its standard error. The candidate's must equal the stock engine's within the calibrated bound, both ways, beyond 1.96 standard errors. Accepting drafts the target would not sample (which moves it toward greedy), sampling with other parameters, and cutting replies all move it. The same requests give each reply token's rank among the target's choices: the share of sampled tokens that were not the target's first choice must equal the stock engine's within its calibrated bound, both ways. Greedy decoding drives it to zero; accepting drafts loosely moves it too, and it is far more sensitive than the likelihood when the sampling distribution is sharp. 3. **Target fidelity.** Each engine scores the same 64 fixed texts of the pool (a recorded reply after its conversation) with `prompt_logprobs`; the mean absolute difference per token between the candidate's log-probabilities and the stock engine's is at most the calibrated ceiling: the target's own forward pass is served unchanged, with no quantized weights, activations, or KV cache. 4. **Length.** The gate fails when the candidate's timed replies differ in mean length from the stock engine's by more than 5% with 95% confidence, compared on the requests both engines answered (the same pass, session, and turn) and pooled over the three workloads. A faster engine plays further into its sessions in a time-boxed pass, and those extra turns have their own natural lengths; and a single workload's few hundred heavy-tailed pairs are too few to judge alone. ## 7. Engine rules - **Memory.** The engine is told `GPU_MEMORY_UTILIZATION` and may hold no more of the device than that share plus 4 GiB (its CUDA context and workspaces; the stock engine peaks about 1 GiB under it). The arena samples the device's memory from outside every 0.5 s during the timed passes. The stock engine runs under the same budget. - **No offload.** Weights or KV blocks in host memory, as the engine's start-up log reports them, break the rules. Host memory is not otherwise measured (an open attack, section 9). - **Start and stop.** The server answers `/health` within 30 minutes of each start. It is stopped with SIGTERM to its process group, then SIGKILL after 120 s. - **Failures.** A pass in which the candidate failed more requests than the stock engine did on the same pass, by more than 2 or 1% of the pass's requests, counts as a ratio of 0; a workload's ratio is the median over its passes, so failing in most of them scores 0. A candidate engine that does not start, or dies, scores 0. ## 8. Splits and budget - `dev`: sessions are drawn per evaluation from the public pool, which is published with its data card. A development evaluation returns everything a scored one does. - `test`: sessions are drawn per evaluation from a hidden pool whose user turns the public cannot train on: real conversations held out by a key, and a share paraphrased by the operators. A speculator that memorized the public data, or the target's replies to it, gains no more on `test` than on real traffic. A submission whose `test` SpecScore falls far below its `dev` one is flagged on its page. - Each account has 20 `dev` and 3 `test` evaluations per task (the arena's study agents have 6 and 1). The board ranks the last scored one. ## 9. Attacks the rules are tested against | Attack | Defense | Status | |---|---|---| | Accept drafts loosely (typical acceptance, a relaxed threshold) | sampled likelihood, first-choice share | likelihood and first-choice caught a 0.6B model served as the target (−0.47 nats) | | Sample greedily, or with other parameters than requested | first-choice share, sampled likelihood, length | caught: forcing greedy shifts the first-choice share by 0.019 (bound 0.003); the likelihood moves only 0.05 nats (within its bound) because the sampling is already peaked, so the first-choice check is the binding one | | Serve a quantized or approximated target | target fidelity, first-choice share, greedy agreement | caught: FP8 weights (fidelity 0.087 on `qwen3-8b`, 0.045 on `qwen35-27b`), FP8 KV (0.107, 0.022) | | Reuse replies across passes, or recognize timed requests | fresh salted sessions per pass; probe and timed requests built alike | by design | | Memorize public text, or the target's replies to it | hidden `test` text; the dev/test gap flag | by design (the `test` pool is hidden and half its chat questions rewritten) | | Hold more device memory than allowed | device memory sampled from outside | caught: 0.97 of the device (178.5 GB) fails the gate | | Offload to host memory | start-up log only | open | | Cut replies short, or pad them | length gate; tokens counted from the text | caught: cutting replies to half their budget gains nothing (0.99), since token/s is unchanged and most replies are already shorter | | Report fake counts or times | the arena counts tokens and times itself | by design | | Read hidden data or other submissions | engine user, no network, private directories readable by root only | by design | | Slow the stock engine with leftover processes, files, or device work | engine-user processes killed and files removed before it runs; the device must come back idle | by design | ## 10. Calibration Before the board opens, on each task: at least four A/A evaluations (the stock engine submitted as a candidate) set the speed noise and the gate limits (the greedy agreement floor, the likelihood and first-choice bounds, the fidelity ceiling); every reference speculator of section 11 passes the gate; and the stock engine's cache statistics on the timed workloads match the targets. Every calibrated value, with its evidence, is published with the version. ### Version 1's values | Limit | `qwen3-8b` | `qwen35-27b` | Evidence | |---|---|---|---| | Fidelity ceiling | 1e-3 | 0.017 | 8B: honest engines ≤ 1e-6 (A/A exactly 0); FP8 weights 0.087, FP8 KV 0.107, a 0.6B model served as the target 0.97. 27B: honest engines up to 0.0125 (A/A 2e-7 to 0.0115, MTP 0.0119–0.0125: where chunked prefill cuts, and the kernels a run selects); FP8 KV 0.0222, FP8 weights 0.0446 | | Likelihood bound | 0.02 | 0.02 | beyond 1.96 standard errors (about 0.04 on 8B, 0.027 on 27B per evaluation); every A/A run and correctly working reference falls inside | | First-choice bound | 0.003 | 0.003 | beyond 1.96 standard errors (about 0.008 and 0.006); every A/A and correct reference has a negative lower bound, so the margin is wide. Forcing greedy decoding shifts it by about 0.019 (lower bound 0.0037), which this catches and the likelihood (0.05 nats, within its bound) does not. About 13–15% of sampled tokens are not the target's first choice | | Greedy agreement floor | 0.5 | none | 8B: honest engines 0.74–1.0, FP8 engines 0.17–0.20. 27B: honest engines from 0.42 (two identical stock engines 0.58) | | Memory slack | 4 GiB | 4 GiB | honest peaks 165.8–167.6 GB of the 169.1 GB allowed; 0.97 of the device (178.5 GB) fails | The speed noise (A/A, four evaluations each): `qwen3-8b` 0.91% mean absolute error, 2.0% at most; `qwen35-27b` 1.23%, 2.2% at most. The pinned vLLM's n-gram speculation on `qwen35-27b` fails the gate: its sampled replies are less likely under the target (−0.096 ± 0.029 nats per token) and leave the target's first choice more often (+0.019 ± 0.006) than the stock engine's, and greedy replies agree on 18% of their length, while MTP on the same model shows none of this. Its hybrid layers' recurrent state is evidently not restored exactly after rejected drafts. It stays on the board as a reference that fails. ## 11. Reference speculators | Task | References | |---|---| | `qwen3-8b` | none (the stock engine); n-gram (prompt lookup, 4 tokens); draft model Qwen3-0.6B (4 tokens); EAGLE-3 `RedHatAI/Qwen3-8B-speculator.eagle3` (3 tokens) | | `qwen35-27b` | none; n-gram (4 tokens); MTP, the target's own layers, at 1, 2, and 3 tokens | They appear on the board as references, measured exactly as submissions are. On the pinned image the probe also tried suffix decoding (it needs `arctic-inference`, which the image lacks), the draft model Qwen3-1.7B and DFlash `RedHatAI/Qwen3-8B-speculator.dflash` on `qwen3-8b` (both stop with a CUDA illegal memory access while the engine starts), and on `qwen35-27b` Qwen3.5-0.8B as a draft (a hidden-size mismatch, 5,120 against 1,024) and DFlash `z-lab/Qwen3.5-27B-DFlash` (the pinned vLLM's DFlash does not support the model's mixed sliding and full attention). An engine of its own may make any of them work. ## 12. Cost The GPU-hours of a `dev` and a `test` evaluation on each task are measured in the calibration and published with the version. Evaluations that fail for the arena's own reasons are refunded.