Agent-12 — the local agent leaderboard
Which model can actually be your agent on your own hardware? Measured by doing real agent tasks in a sandboxed working directory — judged by the filesystem, never by the model's prose. Temperature 0, fixed caps, fresh sandbox per task, one variable moved per comparison.
Local models
| Model | Easy (12) | time | Hard (8) | time | tok/s | Hardware |
|---|---|---|---|---|---|---|
| Qwen3-Coder-30B-A3B (MLX 8-bit)30B MoE, 3B active · native tool calls | 12/12 | 42.6s | 7/8 | 391.5s | 84.3 | Apple M5, 128 GB unified memory |
| Qwen3.6-35B-A3B (MLX 8-bit)35B MoE, 3B active · native tool calls | 12/12 | 64.2s | 8/8 | 125.2s | 46.2 | Apple M5, 128 GB unified memory |
| Gemma 4 31B (MLX 4-bit)31B dense · prompted-XML dialect | 11/12 | 92.1s | 8/8 | 347.9s | 25.6 | Apple M5, 128 GB unified memory |
| DeepSeek V4 Flash (2-bit, 0731 imatrix)284B MoE · ds4.c engine | 12/12 | 203.2s | 8/8 | 550.8s | 8.4 | Apple M5, 128 GB unified memory |
| DeepSeek V4 Flash (2-bit, original) · retired 8/10, weights deleted284B MoE · ds4.c engine | 12/12 | 376.1s | 8/8 | 842.4s | 8.4 | Apple M5, 128 GB unified memory |
Cloud reference (not competing — context)
| Model | Easy (12) | time | Hard (8) | time | tok/s | Where it runs |
|---|---|---|---|---|---|---|
| Claude Sonnet 5 (cloud reference)same engine, same tasks, remote model | 12/12 | 122.0s | 8/8 | 131.0s | — | Anthropic API |
How to trust a number here:
- Every judge is smoke-tested on a known-good AND a known-bad reference solution before it judges anything real.
- Tasks that ship a test file pin its hash — editing the test scores zero.
- All rows run the same tasks, same caps, same judges, on the stated hardware.
- Cloud rows are labeled reference points, not contestants.
Tasks, judges, runner, and the validation gate are open: github.com/nicedreamzapp/agent12 · methodology · contamination policy. Cloud reference times include network/API round-trips — that is the honest end-to-end experience. Updated 2026-08-10.