README.md

5.7 Benchmark on your own machine: single-stream, concurrency, and two quantization tiers

This benchmark answers three questions: how fast a small model generates tokens for a single stream on your machine; how much each stream slows down and how many total tokens per second the machine produces when several requests arrive at once; and how much speed differs between two quantization tiers (4-bit and 8-bit) of the same model. Published tok/s claims usually quote only one of those numbers. These scripts put them side by side in one table.

How it works: start a local inference server and send 1, 4, 8, and 16 concurrent requests (plus 32 on an NVIDIA GPU). Every request sends an ~800-token English input prompt asking the model to count upward from 1, with max_tokens locked to 256 so every stream does the same amount of work. We record three metrics: time to first token (TTFT), per-stream decode speed, and total throughput across all streams.

There are two scripts:

• bench.py: benchmarks any local server exposing an OpenAI-compatible HTTP endpoint, including llama.cpp's llama-server and vLLM.
• bench_mlx.py: benchmarks MLX on macOS by calling mlx-lm directly inside a Python process and prints the same table.

All runs use Qwen3.5-2B (post-trained checkpoint, thinking disabled).

What hardware you need

• Apple Silicon Mac: runs both llama.cpp and MLX; 8 GB of unified memory is enough.
• One NVIDIA GPU: runs both llama.cpp (CUDA build) and vLLM; vLLM requires Linux (or WSL2), and 8 GB of VRAM is enough.

Setup and running

cd 5_7_local_speed
uv venv && source .venv/bin/activate       # Without uv, use: python3 -m venv .venv
uv pip install -r requirements.txt         # Installs httpx only

llama.cpp (Mac or NVIDIA). On a Mac, run brew install llama.cpp; on an NVIDIA GPU, build the CUDA version following the llama.cpp repository instructions. -np sets how many parallel slots the server allocates and must be at least as large as the highest concurrency level you test:

llama-server -hf unsloth/Qwen3.5-2B-GGUF:Q4_K_M --no-mmproj -ngl 99 -c 65536 -np 16 --port 8080
python bench.py --url http://127.0.0.1:8080 --label "llama.cpp Q4_K_M"
# Stop the server, switch to Q8_0, and run again
llama-server -hf unsloth/Qwen3.5-2B-GGUF:Q8_0 --no-mmproj -ngl 99 -c 65536 -np 16 --port 8080
python bench.py --url http://127.0.0.1:8080 --label "llama.cpp Q8_0"

MLX (Mac).

uv pip install mlx-lm
python bench_mlx.py --model mlx-community/Qwen3.5-2B-4bit --label "MLX 4bit"
python bench_mlx.py --model mlx-community/Qwen3.5-2B-8bit --label "MLX 8bit"

vLLM (NVIDIA, Linux). Test two precisions: unquantized bf16, and FP8 converted by vLLM at load time:

uv pip install vllm
vllm serve Qwen/Qwen3.5-2B --port 8000 --max-model-len 4096 --language-model-only
python bench.py --url http://127.0.0.1:8000 --label "vLLM bf16" --concurrency 1 4 8 16 32
vllm serve Qwen/Qwen3.5-2B --port 8000 --max-model-len 4096 --language-model-only --quantization fp8
python bench.py --url http://127.0.0.1:8000 --label "vLLM fp8" --concurrency 1 4 8 16 32

Each run prints a summary table and saves a JSON file under results/.

What you should see

Our reference outputs for Section 5.7 are in sample_output/ (2026-10-23): four m3ultra_*.json files from the Mac Studio (M3 Ultra) runs and four rtx5090_*.json files from the RTX 5090 (32 GB) runs. All eight files are JSON records from the English-prompt runs above, with English labels and field names.

Results on a Mac Studio (M3 Ultra), where each cell shows per-stream decode speed / total throughput in tok/s:

Results on a single RTX 5090 (32 GB), again showing per-stream decode speed / total throughput in tok/s:

Key patterns in these measurements:

• Single-stream speed and total throughput are two different numbers. On the Mac, raising concurrency from 1 stream to 16 slows down each individual stream (MLX 4-bit drops from 272 to 87 tok/s), while total throughput across all streams climbs nearly 4x (from 244 to 923 tok/s). On the GPU with vLLM, going from 1 stream to 32 pushes total throughput up 18x (from 309 to 5,647 tok/s). Quoting only single-stream speed understates how many users one machine can serve; quoting only total throughput misleads readers into expecting that speed for a single user.
• Time to first token rises with concurrency. Requests arriving together queue up for prefill. On the Mac, llama.cpp time to first token climbs from 0.17 seconds at 1 stream to 2.7 seconds at 16 streams.
• 4-bit is not always faster than 8-bit. At 1 stream, 4-bit wins because smaller weights require fewer memory reads. In llama.cpp on the Mac, however, Q8_0 pulls ahead of Q4_K_M from 4 concurrent streams onward. The usual explanation is that higher concurrency shifts the bottleneck from reading weights toward compute, and unpacking (dequantizing) 4-bit weights before every matrix operation costs more compute than the smaller memory read saves. We did not verify that mechanism separately in this benchmark.
• The inference stack determines how performance scales under concurrency. On the GPU, llama.cpp Q4_K_M is the fastest at 1 stream (490 tok/s), ahead of vLLM. Yet llama.cpp total throughput plateaus around 880 tok/s by 8 streams; adding more concurrency only slices the same pie thinner, leaving 39 tok/s per stream and a 2.6-second time to first token at 32 streams. With vLLM, per-stream decode speed drops slowly: at 32 streams each stream still gets 226 to 254 tok/s, total throughput reaches 5,600 to 6,100 tok/s, and time to first token stays around 0.3 seconds. Serving one person at a desk and serving a group of concurrent users call for different inference stacks.
• On the GPU, FP8 beats bf16 at every concurrency level (390 vs. 323 tok/s at 1 stream; 6,073 vs. 5,647 tok/s at 32 streams), because weight reads shrink by half and the RTX 5090 has native FP8 compute units.
• The same model behaves very differently across inference stacks. On the same Mac at roughly 4-bit precision, MLX delivers about 2.5x the total throughput of llama.cpp. We also tested mlx-lm's built-in HTTP server (mlx_lm.server, version 0.32.2): single-stream speed dropped to 41 to 74 tok/s, and at 8 concurrent streams it refused connections. That is why we benchmark MLX in-process with bench_mlx.py, which measures the engine itself. In real deployments, the HTTP serving layer can eat into speed as well.

These numbers only measure how fast each setup runs, not how much output quality changes after quantization. For quality trade-offs across quantization tiers, check the published evaluations cited in Section 5.7 or run a side-by-side check on your own tasks.

Test with your own workload

To swap models: change the GGUF repository and quantization tag after -hf in llama.cpp, --model in bench_mlx.py, or the model name after vllm serve. To change the workload: adjust --max-tokens for output length; to test longer inputs, edit FILLER in bench.py (currently ~800 tokens); and set --concurrency to the number of simultaneous requests you expect. The combination worth measuring most is the one you will actually deploy: your target inference stack, your chosen quantization tier, and your expected concurrency.