TIL: MLX is only a little faster than llama.cpp on my M1

I kept hearing that MLX, Apple's machine learning framework, is the faster way to run local models on a Mac. Before switching the runtime under LCARS Online, my All Things Open demo, I measured it on the machine I actually have: an M1 with 16 GB of memory.

TL;DR: MLX was about 6% faster end to end on my agent's real prompt. It generates text faster but reads prompts slower. On an M1, that isn't worth switching for.

• Same model in both runtimes: Qwen3.5-9B, 4-bit (Q4_K_M GGUF for llama.cpp, mlx-community/Qwen3.5-9B-4bit for MLX via mlx-lm 0.32.2).
• Same interface: each runtime's OpenAI-compatible server, run one at a time so they never compete for memory.
• Two prompts: a short one, and my Science Officer agent's real planning prompt (about 1,000 tokens of instructions, data facts and a hazard checklist).
• Same output: 256 tokens at temperature 0.
• One more check: whether each runtime returns a proper tool call, since my agents can't work without one.

Short prompt

• llama.cpp: 24.7 s total (first token 0.98 s, 10.8 tok/s)
• MLX: 22.6 s total (first token 0.93 s, 11.8 tok/s)

Agent planning prompt (~1,000 tokens)

• llama.cpp: 34.6 s total (first token 9.6 s, 10.2 tok/s)

• MLX: 32.5 s total (first token 10.7 s, 11.7 tok/s)

• Generation: MLX is about 9–15% faster.

• Reading the prompt: MLX is about 10% slower (10.7 vs. 9.6 seconds for my agent's prompt).
• End to end: MLX finished my agent's prompt about 6% sooner. On a 15–20 minute agent run, that's about a minute.
• Tool calls: both runtimes returned them correctly.

This matches what others have published: MLX's lead depends on the chip, the prompt length and the model size, and it's smallest on older chips like the M1.

My first run showed llama.cpp returning its first token in 0.4 seconds on a long prompt, which is impossible. Reading that prompt takes about 10 seconds on this machine.

The cause: I warmed each server up with the same prompt I then measured, and both runtimes cache prompts they've already seen. I was benchmarking a cache hit. The fix was to put a unique tag at the start of every request, so nothing could be reused. My first run also showed MLX at "0 tokens per second," because its server streams text under a different field name than llama.cpp's. My script just wasn't counting it.

If a local LLM benchmark shows a near-instant first token on a long prompt, check for prompt caching before you believe it.

• The 4-bit formats differ. GGUF Q4_K_M and MLX's 4-bit use different quantization schemes, so "4-bit" isn't exactly the same model in both.
• One run per cell, on one machine, plugged in.
• This tests one model size. Published results suggest MLX's advantage changes as models get bigger or smaller.

I'll run the same script, with the same model and prompts, on my 10-core M5 with 24 GB of memory. Newer chips are where MLX is supposed to pull ahead, so we'll see whether that holds up. The extra memory also leaves room to try a bigger model, which part one couldn't.

To run it yourself, the script is scripts/benchmark_runtimes.py in LCARS Online: