Everyone publishes the big benchmarks. Nobody logs the small ones, which are the ones that decide whether a model is usable: does it read the label, find the fact at 128k, emit a tool call that parses, fit the machine you own. 9 suites here, 0 community submissions so far, and every row says who ran it and on what.
Twelve filesystem-judged coding tasks. Does the model actually edit files, not describe editing them.
| Model | Result | Hardware | Items | Ran it | |
|---|---|---|---|---|---|
| measured here | Claude Sonnet 5 (cloud reference) | pass rate 100.0 %, wall s 122.0 seconds | Apple M5, 128 GB unified memory | 12 | Agent-12 |
| measured here | DeepSeek V4 Flash (2-bit, 0731 imatrix) | pass rate 100.0 %, wall s 203.2 seconds, tok s 8.4 tokens/sec | Apple M5, 128 GB unified memory | 12 | Agent-12 |
| measured here | ds4_orig_easy_s2 | pass rate 100.0 %, wall s 347.8 seconds | Apple M5, 128 GB unified memory | 12 | Agent-12 |
| measured here | DeepSeek V4 Flash (2-bit, original) · retired 8/10, weights deleted | pass rate 100.0 %, wall s 376.1 seconds, tok s 8.4 tokens/sec | Apple M5, 128 GB unified memory | 12 | Agent-12 |
| measured here | gemma_easy_s2 | pass rate 100.0 %, wall s 129.6 seconds | Apple M5, 128 GB unified memory | 12 | Agent-12 |
| measured here | gemma_easy_s4 | pass rate 100.0 %, wall s 120.3 seconds | Apple M5, 128 GB unified memory | 12 | Agent-12 |
| measured here | qwen30_easy_s2 | pass rate 100.0 %, wall s 40.3 seconds | Apple M5, 128 GB unified memory | 12 | Agent-12 |
| measured here | qwen30_easy_s3 | pass rate 100.0 %, wall s 36.3 seconds | Apple M5, 128 GB unified memory | 12 | Agent-12 |
| measured here | qwen30_easy_s4 | pass rate 100.0 %, wall s 48.6 seconds | Apple M5, 128 GB unified memory | 12 | Agent-12 |
| measured here | Qwen3-Coder-30B-A3B (MLX 8-bit) | pass rate 100.0 %, wall s 42.6 seconds, tok s 84.3 tokens/sec | Apple M5, 128 GB unified memory | 12 | Agent-12 |
| measured here | qwen36_easy_s3 | pass rate 100.0 %, wall s 59.0 seconds | Apple M5, 128 GB unified memory | 12 | Agent-12 |
| measured here | qwen36_easy_s4 | pass rate 100.0 %, wall s 82.3 seconds | Apple M5, 128 GB unified memory | 12 | Agent-12 |
| measured here | qwen36_easy_v1 | pass rate 100.0 %, wall s 553.2 seconds | Apple M5, 128 GB unified memory | 12 | Agent-12 |
| measured here | Qwen3.6-35B-A3B (MLX 8-bit) | pass rate 100.0 %, wall s 64.2 seconds, tok s 46.2 tokens/sec | Apple M5, 128 GB unified memory | 12 | Agent-12 |
| measured here | Qwen3.8-27B (MLX 8-bit, dense) + DFlash 2 drafter | pass rate 100.0 %, wall s 122.0 seconds, tok s 28.7 tokens/sec | Apple M5, 128 GB unified memory | 12 | Agent-12 |
| measured here | Qwen3.8-27B (MLX 8-bit, dense) | pass rate 100.0 %, wall s 350.7 seconds | Apple M5, 128 GB unified memory | 12 | Agent-12 |
| measured here | qwen38_easy_v1 | pass rate 100.0 %, wall s 233.4 seconds | Apple M5, 128 GB unified memory | 12 | Agent-12 |
| measured here | ds4_0731_easy_s2 | pass rate 91.7 %, wall s 239.8 seconds | Apple M5, 128 GB unified memory | 12 | Agent-12 |
| measured here | gemma_easy_s3 | pass rate 91.7 %, wall s 141.7 seconds | Apple M5, 128 GB unified memory | 12 | Agent-12 |
| measured here | Gemma 4 31B (MLX 4-bit) | pass rate 91.7 %, wall s 92.1 seconds, tok s 25.6 tokens/sec | Apple M5, 128 GB unified memory | 12 | Agent-12 |
The same harness on tasks that need multi-file reasoning and exact output.
| Model | Result | Hardware | Items | Ran it | |
|---|---|---|---|---|---|
| measured here | Claude Sonnet 5 (cloud reference) | pass rate 100.0 %, wall s 131.0 seconds | Apple M5, 128 GB unified memory | 8 | Agent-12 |
| measured here | ds4_0731_hard_s2 | pass rate 100.0 %, wall s 406.4 seconds | Apple M5, 128 GB unified memory | 8 | Agent-12 |
| measured here | DeepSeek V4 Flash (2-bit, 0731 imatrix) | pass rate 100.0 %, wall s 550.8 seconds, tok s 8.4 tokens/sec | Apple M5, 128 GB unified memory | 8 | Agent-12 |
| measured here | DeepSeek V4 Flash (2-bit, original) · retired 8/10, weights deleted | pass rate 100.0 %, wall s 842.4 seconds, tok s 8.4 tokens/sec | Apple M5, 128 GB unified memory | 8 | Agent-12 |
| measured here | gemma_hard_s4 | pass rate 100.0 %, wall s 331.9 seconds | Apple M5, 128 GB unified memory | 8 | Agent-12 |
| measured here | Gemma 4 31B (MLX 4-bit) | pass rate 100.0 %, wall s 347.9 seconds, tok s 25.6 tokens/sec | Apple M5, 128 GB unified memory | 8 | Agent-12 |
| measured here | qwen36_hard_bigbudget | pass rate 100.0 %, wall s 122.3 seconds | Apple M5, 128 GB unified memory | 8 | Agent-12 |
| measured here | qwen36_hard_s3 | pass rate 100.0 %, wall s 120.3 seconds | Apple M5, 128 GB unified memory | 8 | Agent-12 |
| measured here | qwen36_hard_s4 | pass rate 100.0 %, wall s 135.1 seconds | Apple M5, 128 GB unified memory | 8 | Agent-12 |
| measured here | Qwen3.6-35B-A3B (MLX 8-bit) | pass rate 100.0 %, wall s 125.2 seconds, tok s 46.2 tokens/sec | Apple M5, 128 GB unified memory | 8 | Agent-12 |
| measured here | qwen38_8bit_hard_bigbudget | pass rate 100.0 %, wall s 2065.3 seconds | Apple M5, 128 GB unified memory | 8 | Agent-12 |
| measured here | gemma_hard_s3 | pass rate 87.5 %, wall s 714.0 seconds | Apple M5, 128 GB unified memory | 8 | Agent-12 |
| measured here | qwen30_hard_s3 | pass rate 87.5 %, wall s 230.3 seconds | Apple M5, 128 GB unified memory | 8 | Agent-12 |
| measured here | qwen30_hard_s4 | pass rate 87.5 %, wall s 245.4 seconds | Apple M5, 128 GB unified memory | 8 | Agent-12 |
| measured here | Qwen3-Coder-30B-A3B (MLX 8-bit) | pass rate 87.5 %, wall s 391.5 seconds, tok s 84.3 tokens/sec | Apple M5, 128 GB unified memory | 8 | Agent-12 |
| measured here | qwen36_hard_s2 | pass rate 87.5 %, wall s 182.9 seconds | Apple M5, 128 GB unified memory | 8 | Agent-12 |
| measured here | Qwen3.8-27B (MLX 8-bit, dense) + DFlash 2 drafter | pass rate 87.5 %, wall s 401.4 seconds, tok s 28.7 tokens/sec | Apple M5, 128 GB unified memory | 8 | Agent-12 |
| measured here | Qwen3.8-27B (MLX 8-bit, dense) | pass rate 87.5 %, wall s 1155.9 seconds | Apple M5, 128 GB unified memory | 8 | Agent-12 |
| measured here | qwen38_8bit_hard_v1 | pass rate 75.0 %, wall s 1031.0 seconds | Apple M5, 128 GB unified memory | 8 | Agent-12 |
| measured here | qwen38_hard_v1 | pass rate 75.0 %, wall s 677.6 seconds | Apple M5, 128 GB unified memory | 8 | Agent-12 |
| measured here | qwen38_hard_s2 | pass rate 62.5 %, wall s 782.9 seconds | Apple M5, 128 GB unified memory | 8 | Agent-12 |
Unified memory a model needs, from its own file list and config. Not a speed test.
Signs, labels, handwriting, a screenshot. Exact-string scored, so there is nothing to argue about.
One fixed set of photos, one fixed prompt. Is the description right, and does it name what matters rather than everything.
Detection models on a fixed image set, scored on boxes, the test that matters for a camera that has to act.
Word error rate on the same clips, including noisy and accented ones, which is where the published numbers stop being true.
A fact buried at a known depth. Does the model find it at 8k, 32k, 128k, and what does the memory do while it tries.
How often a model emits a tool call the runtime can actually execute, which is the failure that kills local agents.
Their own numbers, their own hardware, kept separate on purpose. Useful context, not a result.
| Model | Claimed | Source | |
|---|---|---|---|
| reported | Meta Muse Glimmer 30B | MCP-Atlas 75.5 · SWE-bench Verified 76.0 · SWE-bench Pro 51.2 · Terminal-Bench 2.1 51.7 · τ3-Banking 23.5 | Meta, official model card |
| reported | NVIDIA Nemotron 3 Nano Omni 30B-A3B | OSWorld (computer use) 47.4 | NVIDIA, official model card |
python tools/build_all.py to rebuild this page.