Every test we log

Everyone publishes the big benchmarks. Nobody logs the small ones, which are the ones that decide whether a model is usable: does it read the label, find the fact at 128k, emit a tool call that parses, fit the machine you own. 9 suites here, 0 community submissions so far, and every row says who ran it and on what.

Just want to know what to run? Pick your Mac and the job and it answers with one model name.
Add yours. The submit form writes the file for you and checks it before you file it, no git needed. Or copy the template, fill it in, open a pull request, or open an issue and it gets filed for you. A number with no machine named gets rejected by the validator, not by an opinion. Suites marked draft have no harness written yet, so the first submission defines the format.

Agent tasks, easy live

Twelve filesystem-judged coding tasks. Does the model actually edit files, not describe editing them.

ModelResultHardwareItemsRan it
measured hereClaude Sonnet 5 (cloud reference)pass rate 100.0 %, wall s 122.0 secondsApple M5, 128 GB unified memory12Agent-12
measured hereDeepSeek V4 Flash (2-bit, 0731 imatrix)pass rate 100.0 %, wall s 203.2 seconds, tok s 8.4 tokens/secApple M5, 128 GB unified memory12Agent-12
measured hereds4_orig_easy_s2pass rate 100.0 %, wall s 347.8 secondsApple M5, 128 GB unified memory12Agent-12
measured hereDeepSeek V4 Flash (2-bit, original) · retired 8/10, weights deletedpass rate 100.0 %, wall s 376.1 seconds, tok s 8.4 tokens/secApple M5, 128 GB unified memory12Agent-12
measured heregemma_easy_s2pass rate 100.0 %, wall s 129.6 secondsApple M5, 128 GB unified memory12Agent-12
measured heregemma_easy_s4pass rate 100.0 %, wall s 120.3 secondsApple M5, 128 GB unified memory12Agent-12
measured hereqwen30_easy_s2pass rate 100.0 %, wall s 40.3 secondsApple M5, 128 GB unified memory12Agent-12
measured hereqwen30_easy_s3pass rate 100.0 %, wall s 36.3 secondsApple M5, 128 GB unified memory12Agent-12
measured hereqwen30_easy_s4pass rate 100.0 %, wall s 48.6 secondsApple M5, 128 GB unified memory12Agent-12
measured hereQwen3-Coder-30B-A3B (MLX 8-bit)pass rate 100.0 %, wall s 42.6 seconds, tok s 84.3 tokens/secApple M5, 128 GB unified memory12Agent-12
measured hereqwen36_easy_s3pass rate 100.0 %, wall s 59.0 secondsApple M5, 128 GB unified memory12Agent-12
measured hereqwen36_easy_s4pass rate 100.0 %, wall s 82.3 secondsApple M5, 128 GB unified memory12Agent-12
measured hereqwen36_easy_v1pass rate 100.0 %, wall s 553.2 secondsApple M5, 128 GB unified memory12Agent-12
measured hereQwen3.6-35B-A3B (MLX 8-bit)pass rate 100.0 %, wall s 64.2 seconds, tok s 46.2 tokens/secApple M5, 128 GB unified memory12Agent-12
measured hereQwen3.8-27B (MLX 8-bit, dense) + DFlash 2 drafterpass rate 100.0 %, wall s 122.0 seconds, tok s 28.7 tokens/secApple M5, 128 GB unified memory12Agent-12
measured hereQwen3.8-27B (MLX 8-bit, dense)pass rate 100.0 %, wall s 350.7 secondsApple M5, 128 GB unified memory12Agent-12
measured hereqwen38_easy_v1pass rate 100.0 %, wall s 233.4 secondsApple M5, 128 GB unified memory12Agent-12
measured hereds4_0731_easy_s2pass rate 91.7 %, wall s 239.8 secondsApple M5, 128 GB unified memory12Agent-12
measured heregemma_easy_s3pass rate 91.7 %, wall s 141.7 secondsApple M5, 128 GB unified memory12Agent-12
measured hereGemma 4 31B (MLX 4-bit)pass rate 91.7 %, wall s 92.1 seconds, tok s 25.6 tokens/secApple M5, 128 GB unified memory12Agent-12

Agent tasks, hard live

The same harness on tasks that need multi-file reasoning and exact output.

ModelResultHardwareItemsRan it
measured hereClaude Sonnet 5 (cloud reference)pass rate 100.0 %, wall s 131.0 secondsApple M5, 128 GB unified memory8Agent-12
measured hereds4_0731_hard_s2pass rate 100.0 %, wall s 406.4 secondsApple M5, 128 GB unified memory8Agent-12
measured hereDeepSeek V4 Flash (2-bit, 0731 imatrix)pass rate 100.0 %, wall s 550.8 seconds, tok s 8.4 tokens/secApple M5, 128 GB unified memory8Agent-12
measured hereDeepSeek V4 Flash (2-bit, original) · retired 8/10, weights deletedpass rate 100.0 %, wall s 842.4 seconds, tok s 8.4 tokens/secApple M5, 128 GB unified memory8Agent-12
measured heregemma_hard_s4pass rate 100.0 %, wall s 331.9 secondsApple M5, 128 GB unified memory8Agent-12
measured hereGemma 4 31B (MLX 4-bit)pass rate 100.0 %, wall s 347.9 seconds, tok s 25.6 tokens/secApple M5, 128 GB unified memory8Agent-12
measured hereqwen36_hard_bigbudgetpass rate 100.0 %, wall s 122.3 secondsApple M5, 128 GB unified memory8Agent-12
measured hereqwen36_hard_s3pass rate 100.0 %, wall s 120.3 secondsApple M5, 128 GB unified memory8Agent-12
measured hereqwen36_hard_s4pass rate 100.0 %, wall s 135.1 secondsApple M5, 128 GB unified memory8Agent-12
measured hereQwen3.6-35B-A3B (MLX 8-bit)pass rate 100.0 %, wall s 125.2 seconds, tok s 46.2 tokens/secApple M5, 128 GB unified memory8Agent-12
measured hereqwen38_8bit_hard_bigbudgetpass rate 100.0 %, wall s 2065.3 secondsApple M5, 128 GB unified memory8Agent-12
measured heregemma_hard_s3pass rate 87.5 %, wall s 714.0 secondsApple M5, 128 GB unified memory8Agent-12
measured hereqwen30_hard_s3pass rate 87.5 %, wall s 230.3 secondsApple M5, 128 GB unified memory8Agent-12
measured hereqwen30_hard_s4pass rate 87.5 %, wall s 245.4 secondsApple M5, 128 GB unified memory8Agent-12
measured hereQwen3-Coder-30B-A3B (MLX 8-bit)pass rate 87.5 %, wall s 391.5 seconds, tok s 84.3 tokens/secApple M5, 128 GB unified memory8Agent-12
measured hereqwen36_hard_s2pass rate 87.5 %, wall s 182.9 secondsApple M5, 128 GB unified memory8Agent-12
measured hereQwen3.8-27B (MLX 8-bit, dense) + DFlash 2 drafterpass rate 87.5 %, wall s 401.4 seconds, tok s 28.7 tokens/secApple M5, 128 GB unified memory8Agent-12
measured hereQwen3.8-27B (MLX 8-bit, dense)pass rate 87.5 %, wall s 1155.9 secondsApple M5, 128 GB unified memory8Agent-12
measured hereqwen38_8bit_hard_v1pass rate 75.0 %, wall s 1031.0 secondsApple M5, 128 GB unified memory8Agent-12
measured hereqwen38_hard_v1pass rate 75.0 %, wall s 677.6 secondsApple M5, 128 GB unified memory8Agent-12
measured hereqwen38_hard_s2pass rate 62.5 %, wall s 782.9 secondsApple M5, 128 GB unified memory8Agent-12

Will it fit a Mac live

Unified memory a model needs, from its own file list and config. Not a speed test.

10 models worked out so far, one page each, memory by Mac from 8 GB to 128 GB: will it fit my Mac.

Vision: read the text in the picture ready

Signs, labels, handwriting, a screenshot. Exact-string scored, so there is nothing to argue about.

Nothing logged yet. python suites/vision-read-text/run.py --model <mlx vlm repo> --hardware "..."

Vision: describe a photo draft

One fixed set of photos, one fixed prompt. Is the description right, and does it name what matters rather than everything.

Nothing logged yet. not written yet, submissions welcome

Vision: find the object draft

Detection models on a fixed image set, scored on boxes, the test that matters for a camera that has to act.

Nothing logged yet. not written yet, submissions welcome

Speech to text draft

Word error rate on the same clips, including noisy and accented ones, which is where the published numbers stop being true.

Nothing logged yet. not written yet, submissions welcome

Long context, actually retrieved draft

A fact buried at a known depth. Does the model find it at 8k, 32k, 128k, and what does the memory do while it tries.

Nothing logged yet. not written yet, submissions welcome

Tool calls that parse draft

How often a model emits a tool call the runtime can actually execute, which is the failure that kills local agents.

Nothing logged yet. not written yet, submissions welcome

Published by the vendor, not run here

Their own numbers, their own hardware, kept separate on purpose. Useful context, not a result.

ModelClaimedSource
reportedMeta Muse Glimmer 30BMCP-Atlas 75.5 · SWE-bench Verified 76.0 · SWE-bench Pro 51.2 · Terminal-Bench 2.1 51.7 · τ3-Banking 23.5Meta, official model card
reportedNVIDIA Nemotron 3 Nano Omni 30B-A3BOSWorld (computer use) 47.4NVIDIA, official model card
Agent-12 leaderboard · will it fit my Mac · source · methodology

Built 2026-10-30. Run python tools/build_all.py to rebuild this page.