MLX format of model LocoreMind/LocoOperator-4B.
Apple MacBook M3 Pro
内存：36GB
核总数：14
Intelligence Benchmark Comparison
| Benchmark | Mode | Sampled | Qwen3-8B-MLX-8bit | Nanbeige4.1-3B | Qwen3-4B-MLX-bf16 | AgentCPM-Explore | LocoOperator-4B | Qwen3.5-4B-Claude-4.6-Opus-Reasoning-Distilled |
|---|---|---|---|---|---|---|---|---|
| MMLU | Sample | 1000/14042 | 69.6% | 24.4% | 66.4% | 23.4% | 69.5% | 23.9% |
| CMMLU | Sample | 300/11582 | 77.0% | 26.3% | 65.7% | 38.3% | 67.0% | 22.0% |
| TRUTHFULQA | Full | 817 | 64.8% | 22.4% | 55.9% | 26.9% | 59.5% | 24.2% |
| HUMANEVAL | Full | 164 | 76.8% | 0.0% | 76.2% | 67.7% | 83.5% | 53.7% |
| HELLASWAG | Sample | 200/10042 | 77.5% | 21.5% | 55.0% | 25.5% | 82.5% | 24.5% |
| WINOGRANDE | Sample | 300/1267 | 58.7% | 50.7% | 56.3% | 50.0% | 64.3% | 49.0% |
--- Detail ---
Model: Qwen3-8B-MLX-8bit
Benchmark Accuracy Correct Total Time(s)
------------------------------------------------------
MMLU 69.6% 696 1000 2001.9
CMMLU 77.0% 231 300 401.7
TRUTHFULQA 64.8% 529 817 735.8
HUMANEVAL 76.8% 126 164 447.3
WINOGRANDE 58.7% 176 300 163.3
HELLASWAG 77.5% 155 200 266.1
Model: Nanbeige4.1-3B
Benchmark Accuracy Correct Total Time(s)
------------------------------------------------------
MMLU 24.4% 244 1000 11826.9
CMMLU 26.3% 79 300 1027.5
HELLASWAG 21.5% 43 200 713
TRUTHFULQA 22.4% 183 817 2768.1
WINOGRANDE 50.7% 152 300 1026.4
HUMANEVAL 0.0% 0 164 2153.9
Model: Qwen3-4B-MLX-bf16
Benchmark Accuracy Correct Total Time(s)
------------------------------------------------------
MMLU 66.4% 664 1000 1292.1
CMMLU 65.7% 197 300 616.7
HELLASWAG 55.0% 110 200 373.5
TRUTHFULQA 55.9% 457 817 785.9
WINOGRANDE 56.3% 169 300 149.5
HUMANEVAL 76.2% 125 164 745.1
Model: AgentCPM-Explore
Benchmark Accuracy Correct Total Time(s)
------------------------------------------------------
MMLU 23.4% 234 1000 4683.5
CMMLU 38.3% 115 300 1216.4
HELLASWAG 25.5% 51 200 895.8
TRUTHFULQA 26.9% 220 817 3468.7
WINOGRANDE 50.0% 150 300 1635
HUMANEVAL 67.7% 111 164 4067.7
Model: LocoOperator-4B
Benchmark Accuracy Correct Total Time(s)
------------------------------------------------------
MMLU 69.5% 695 1000 1175.1
CMMLU 67.0% 201 300 338.9
HELLASWAG 82.5% 165 200 144.2
TRUTHFULQA 59.5% 486 817 934.4
WINOGRANDE 64.3% 193 300 149.6
HUMANEVAL 83.5% 137 164 829.7
Model: Qwen3.5-4B-Claude-4.6-Opus-Reasoning-Distilled
Benchmark Accuracy Correct Total Time(s)
------------------------------------------------------
MMLU 23.9% 239 1000 5595.6
CMMLU 22.0% 66 300 1374.5
HELLASWAG 24.5% 49 200 1832.6
TRUTHFULQA 24.2% 198 817 3914.1
WINOGRANDE 49.0% 147 300 1370.1
HUMANEVAL 53.7% 88 164 2680.2