SpeculatorArena protocol, version 1

SpeculatorArena measures how much faster a speculative decoder makes a fixed model serve real multi-turn traffic on one GPU, without changing what the model says. A submission is a speculator checkpoint and the engine that serves the model with it. In the same pod, the arena serves the same sessions with the stock engine and no speculation, and scores the submission by the speed it gains over that.

This document is the contract: what a submission is, how it is measured and scored, and what breaks the rules. A change gets a new version number, and the values in force are served at GET /api/protocol. Values marked calibrated are set from the arena's own measurements (section 10) and published with the version that sets them.

1. Tasks

Both are served in non-thinking mode (enable_thinking: false). The engine runs in a pinned image. Version 1 starts from LowBArena's (vllm-openai 0.25.1, commit 7a33ba95f); a probe confirms that it serves every reference speculator of section 11, or picks an image that does.

The target's weights, activations, and KV cache stay in BF16. The arena supplies the target, sealed read-only, and checks that what the engine serves is the target's own distribution (section 6).

2. A submission

{"task_id": "qwen3-8b", "split": "dev",
 "speculator": {"uri": "my-org/qwen3-8b-eagle3", "revision": "<exact commit>"},
 "engine": {"inline": "<base64 of a .tar.gz of my-engine/>"},
 "solution": {"method": "EAGLE-3, 4 tokens", "kind": "eagle3"}}

Submitting takes an account's token (Authorization: Bearer <token>, which the submit page and the kit's client send). Accounts are by invitation: the operators give each participant a token, shown once and kept only as its hash; when the operators open registration, /register hands one out.

• Speculator checkpoint, optional: a Hugging Face repository at an exact commit, or (from a workspace on the arena's cluster) a directory of that workspace; none for a method without weights of its own (n-gram or suffix lookup, the target's own MTP layers). Only *.json and *.safetensors files are fetched, at most half the target's bytes. Python files are never fetched.
• Engine directory: launch.sh serve serves the target with the speculator. Inline (a base64 .tar.gz, at most 2,800,000 characters) or a repository at an exact commit (at most 200 MiB), with an optional plugin/ that is installed offline, as in LowBArena.
• solution.kind labels the method (ngram, suffix, draft, eagle3, mtp, other); nothing is scored from it.

3. The engine's environment

launch.sh serve runs as an unprivileged user, with no network and one GPU, in the pinned image:

The server must answer /health, /v1/chat/completions (streamed, honoring the sampling fields of section 4, tools with tool_choice: "none" rendered into the prompt, and prompt_logprobs with continue_final_message, section 6), and /metrics. It may report its acceptance: per request (vLLM's metrics.speculative_decoding) or, as the pinned vLLM does, in its vllm:spec_decode_* counters, which the arena reads around every pass. The arena shows acceptance and never scores it.

4. The timed workloads

Sessions

Every timed request belongs to a multi-turn session, built so that the prefix cache and the speculator meet what they meet in service:

• Structure. Two families. agent: coding-agent sessions whose turn counts, prompt and reply sizes per turn, prefix reuse, and idle times follow recorded agent traces (AgentX's public cc-traces). chat: multi-turn chat sessions whose depth, growth per turn, reply length, think time, and shared prefixes follow aggregate statistics of production traffic. The mix of the two and the trace-derived distributions are published with the version; statistics derived from production traffic are kept by the operators and published only with the data owner's consent.
• Text. Real and coherent. A session's user and tool turns come from one real conversation or agent trajectory in a public dataset (listed in the data card), chosen and cut to fit its structure. System prompts and tool schemas come from public agent frameworks and are shared by the sessions of one app, as production apps share theirs.
• Closed loop. The model's reply joins the history before the next turn. A reply ends where the model ends it (no ignore_eos), within the turn's max_tokens.
• The window. The client counts a prompt's tokens with the target's tokenizer, plus 5% and 128 tokens for what the chat template adds. When a prompt and its reply budget would not fit MAX_MODEL_LEN, the client cuts the history after its first message down to half the window, as clients compact theirs; the cut costs a cache miss, as it does in service.
• Sampling. Timed requests use the model card's non-thinking sampling: temperature 0.7, top_p 0.8, top_k 20, and a seed per request.

Lanes and interleave

A workload runs L lanes. Each lane holds m live sessions and sends the next turn of whichever session's think time, capped at 300 s and scaled by c, has passed first, one request at a time (recorded traces hold idle gaps of hours, a user away; past the cap a session comes back as a returning session does). So L requests are in flight while L × m sessions hold prefixes in the cache, and an idle session loses part of its cache before it returns, as in service. m and c are calibrated so that the stock engine's cache hit (token-weighted and per request, from usage.prompt_tokens_details.cached_tokens) and the share of a returning session's history still cached match the targets in the data card.

busy's lanes come from the stock engine's KV cache (1.05M tokens on qwen3-8b, 1.70M on qwen35-27b, at 0.90 of the device) and a live session's mean footprint over its life (21.7K and 73K tokens): enough sessions to fill the cache, not to overflow it, so that what a speculator holds costs evictions.

Each workload runs three passes of fresh sessions, each pass for a fixed time (150 s on solo, 180 s on team, 240 s on busy); a request still streaming at the end may finish, within a grace period, and counts. No session, prompt, or shared prefix repeats across passes within an evaluation: each pass salts its shared prefixes, and no session serves twice. Each workload warms up with one wave at its own concurrency, from sessions no pass sends. The arena counts tokens itself, from the returned text with the target's tokenizer, and a reply counts only if its stream ends with [DONE] within its max_tokens. One start of the engine serves all three workloads.

Pairing

In the same pod, after the candidate's passes, every process of the engine's user is killed, the files it left in shared memory and scratch are removed, and once the device holds less than 2 GiB (within 120 s, or the evaluation is void and refunded) the stock engine (the pinned vLLM with --enable-prefix-caching, BF16, the same GPU_MEMORY_UTILIZATION, no speculation) serves the same passes: the same sessions, seeds, and order. A workload's speed ratio r_w is the median over its passes of the candidate's speed over the stock engine's on the same pass.

5. SpecScore

SpecScore = (r_solo · r_team · r_busy)^(1/3)    if every gate of sections 6 and 7 passes, else 0

The stock engine scores 1 against itself, up to the A/A noise of section 10. What a speculator holds on the device (its weights, its own KV cache, its buffers) comes out of the same budget as the target's KV cache. On busy, where the stock engine's cache is near full, that memory costs evictions and preemptions, and the measured speed shows it; there is no separate memory axis. The board ranks each account's last scored evaluation by SpecScore and shows the per-workload ratios, TTFT and TPOT percentiles, the cache hit, and the engine's own report of its acceptance.

6. The lossless gate

Speculative decoding must not change the distribution the target samples from. Floating-point differences between a verify pass and a one-token decode can flip near-ties (in the probe, honest speculators' greedy replies agreed with the stock engine's for 31% to 85% of their length), so each check compares the candidate with the stock engine against a limit calibrated on A/A runs and on the honest reference speculators, at 95% confidence. prompt_logprobs computes logits for every prompt token, so checks 2 and 3 score a reply on the system prompt and the conversation's last 3,000 tokens (led by its first user message when they hold none), the reply cut to 2,000 characters, the same for both engines; and the checks run after the timed passes on a restart of each engine with GPU_MEMORY_UTILIZATION 0.80, which leaves room for those logits. An engine that cannot answer more than 10% of a check's requests fails the gate; if the stock engine cannot, the evaluation is void and refunded.

1. Greedy agreement. 64 probe prompts per evaluation, temperature 0, 256 tokens, one at a time. The share of each reply that agrees with the stock engine's before their first difference, averaged over the prompts, must reach the calibrated floor.
2. Sampled likelihood. 512 timed replies of each engine, drawn from all workloads, are scored by the stock engine under the target (prompt_logprobs): their mean log-probability per token, weighted by tokens, with the replies as clusters for its standard error. The candidate's must equal the stock engine's within the calibrated bound, both ways, beyond 1.96 standard errors. Accepting drafts the target would not sample (which moves it toward greedy), sampling with other parameters, and cutting replies all move it. The same requests give each reply token's rank among the target's choices: the share of sampled tokens that were not the target's first choice must equal the stock engine's within its calibrated bound, both ways. Greedy decoding drives it to zero; accepting drafts loosely moves it too, and it is far more sensitive than the likelihood when the sampling distribution is sharp.
3. Target fidelity. Each engine scores the same 64 fixed texts of the pool (a recorded reply after its conversation) with prompt_logprobs; the mean absolute difference per token between the candidate's log-probabilities and the stock engine's is at most the calibrated ceiling: the target's own forward pass is served unchanged, with no quantized weights, activations, or KV cache.
4. Length. The gate fails when the candidate's timed replies differ in mean length from the stock engine's by more than 5% with 95% confidence, compared on the requests both engines answered (the same pass, session, and turn) and pooled over the three workloads. A faster engine plays further into its sessions in a time-boxed pass, and those extra turns have their own natural lengths; and a single workload's few hundred heavy-tailed pairs are too few to judge alone.

7. Engine rules

• Memory. The engine is told GPU_MEMORY_UTILIZATION and may hold no more of the device than that share plus 4 GiB (its CUDA context and workspaces; the stock engine peaks about 1 GiB under it). The arena samples the device's memory from outside every 0.5 s during the timed passes. The stock engine runs under the same budget.
• No offload. Weights or KV blocks in host memory, as the engine's start-up log reports them, break the rules. Host memory is not otherwise measured (an open attack, section 9).
• Start and stop. The server answers /health within 30 minutes of each start. It is stopped with SIGTERM to its process group, then SIGKILL after 120 s.
• Failures. A pass in which the candidate failed more requests than the stock engine did on the same pass, by more than 2 or 1% of the pass's requests, counts as a ratio of 0; a workload's ratio is the median over its passes, so failing in most of them scores 0. A candidate engine that does not start, or dies, scores 0.

8. Splits and budget

• dev: sessions are drawn per evaluation from the public pool, which is published with its data card. A development evaluation returns everything a scored one does.
• test: sessions are drawn per evaluation from a hidden pool whose user turns the public cannot train on: real conversations held out by a key, and a share paraphrased by the operators. A speculator that memorized the public data, or the target's replies to it, gains no more on test than on real traffic. A submission whose test SpecScore falls far below its dev one is flagged on its page.
• Each account has 20 dev and 3 test evaluations per task (the arena's study agents have 6 and 1). The board ranks the last scored one.

9. Attacks the rules are tested against

10. Calibration

Before the board opens, on each task: at least four A/A evaluations (the stock engine submitted as a candidate) set the speed noise and the gate limits (the greedy agreement floor, the likelihood and first-choice bounds, the fidelity ceiling); every reference speculator of section 11 passes the gate; and the stock engine's cache statistics on the timed workloads match the targets. Every calibrated value, with its evidence, is published with the version.

Version 1's values

The speed noise (A/A, four evaluations each): qwen3-8b 0.91% mean absolute error, 2.0% at most; qwen35-27b 1.23%, 2.2% at most.

The pinned vLLM's n-gram speculation on qwen35-27b fails the gate: its sampled replies are less likely under the target (−0.096 ± 0.029 nats per token) and leave the target's first choice more often (+0.019 ± 0.006) than the stock engine's, and greedy replies agree on 18% of their length, while MTP on the same model shows none of this. Its hybrid layers' recurrent state is evidently not restored exactly after rejected drafts. It stays on the board as a reference that fails.

11. Reference speculators

They appear on the board as references, measured exactly as submissions are. On the pinned image the probe also tried suffix decoding (it needs arctic-inference, which the image lacks), the draft model Qwen3-1.7B and DFlash RedHatAI/Qwen3-8B-speculator.dflash on qwen3-8b (both stop with a CUDA illegal memory access while the engine starts), and on qwen35-27b Qwen3.5-0.8B as a draft (a hidden-size mismatch, 5,120 against 1,024) and DFlash z-lab/Qwen3.5-27B-DFlash (the pinned vLLM's DFlash does not support the model's mixed sliding and full attention). An engine of its own may make any of them work.

12. Cost

The GPU-hours of a dev and a test evaluation on each task are measured in the calibration and published with the version. Evaluations that fail for the arena's own reasons are refunded.