# 5.7 Benchmark on your own machine: single-stream, concurrency, and two quantization tiers

This benchmark answers three questions: how fast a small model generates tokens for a single stream on your machine; how much each stream slows down and how many total tokens per second the machine produces when several requests arrive at once; and how much speed differs between two quantization tiers (4-bit and 8-bit) of the same model. Published `tok/s` claims usually quote only one of those numbers. These scripts put them side by side in one table.

How it works: start a local inference server and send 1, 4, 8, and 16 concurrent requests (plus 32 on an NVIDIA GPU). Every request sends an ~800-token English input prompt asking the model to count upward from 1, with `max_tokens` locked to 256 so every stream does the same amount of work. We record three metrics: time to first token (TTFT), per-stream decode speed, and total throughput across all streams.

There are two scripts:

- `bench.py`: benchmarks any local server exposing an OpenAI-compatible HTTP endpoint, including `llama.cpp`'s `llama-server` and `vLLM`.
- `bench_mlx.py`: benchmarks MLX on macOS by calling `mlx-lm` directly inside a Python process and prints the same table.

All runs use `Qwen3.5-2B` (post-trained checkpoint, thinking disabled).

## What hardware you need

- Apple Silicon Mac: runs both `llama.cpp` and MLX; 8 GB of unified memory is enough.
- One NVIDIA GPU: runs both `llama.cpp` (CUDA build) and `vLLM`; `vLLM` requires Linux (or WSL2), and 8 GB of VRAM is enough.

## Setup and running

```bash
cd 5_7_local_speed
uv venv && source .venv/bin/activate       # Without uv, use: python3 -m venv .venv
uv pip install -r requirements.txt         # Installs httpx only
```

**llama.cpp (Mac or NVIDIA).** On a Mac, run `brew install llama.cpp`; on an NVIDIA GPU, build the CUDA version following the `llama.cpp` repository instructions. `-np` sets how many parallel slots the server allocates and must be at least as large as the highest concurrency level you test:

```bash
llama-server -hf unsloth/Qwen3.5-2B-GGUF:Q4_K_M --no-mmproj -ngl 99 -c 65536 -np 16 --port 8080
python bench.py --url http://127.0.0.1:8080 --label "llama.cpp Q4_K_M"
# Stop the server, switch to Q8_0, and run again
llama-server -hf unsloth/Qwen3.5-2B-GGUF:Q8_0 --no-mmproj -ngl 99 -c 65536 -np 16 --port 8080
python bench.py --url http://127.0.0.1:8080 --label "llama.cpp Q8_0"
```

**MLX (Mac).**

```bash
uv pip install mlx-lm
python bench_mlx.py --model mlx-community/Qwen3.5-2B-4bit --label "MLX 4bit"
python bench_mlx.py --model mlx-community/Qwen3.5-2B-8bit --label "MLX 8bit"
```

**vLLM (NVIDIA, Linux).** Test two precisions: unquantized `bf16`, and `FP8` converted by `vLLM` at load time:

```bash
uv pip install vllm
vllm serve Qwen/Qwen3.5-2B --port 8000 --max-model-len 4096 --language-model-only
python bench.py --url http://127.0.0.1:8000 --label "vLLM bf16" --concurrency 1 4 8 16 32
vllm serve Qwen/Qwen3.5-2B --port 8000 --max-model-len 4096 --language-model-only --quantization fp8
python bench.py --url http://127.0.0.1:8000 --label "vLLM fp8" --concurrency 1 4 8 16 32
```

Each run prints a summary table and saves a JSON file under `results/`.

## What you should see

Our reference outputs for Section 5.7 are in `sample_output/` (`2026-10-23`): four `m3ultra_*.json` files from the Mac Studio (M3 Ultra) runs and four `rtx5090_*.json` files from the RTX 5090 (32 GB) runs. All eight files are JSON records from the English-prompt runs above, with English labels and field names.

Results on a Mac Studio (M3 Ultra), where each cell shows `per-stream decode speed / total throughput` in `tok/s`:

| Concurrent streams | llama.cpp Q4_K_M | llama.cpp Q8_0 | MLX 4-bit | MLX 8-bit |
|---|---|---|---|---|
| 1 | 183 / 163 | 160 / 146 | 272 / 244 | 193 / 178 |
| 4 | 82 / 269 | 94 / 301 | 197 / 606 | 168 / 537 |
| 8 | 48 / 307 | 56 / 344 | 149 / 830 | 127 / 738 |
| 16 | 30 / 360 | 31 / 380 | 87 / 923 | 83 / 896 |

Results on a single RTX 5090 (32 GB), again showing `per-stream decode speed / total throughput` in `tok/s`:

| Concurrent streams | vLLM bf16 | vLLM FP8 | llama.cpp Q4_K_M | llama.cpp Q8_0 |
|---|---|---|---|---|
| 1 | 323 / 309 | 390 / 368 | 490 / 401 | 414 / 348 |
| 4 | 264 / 951 | 372 / 1,280 | 290 / 851 | 264 / 804 |
| 8 | 256 / 1,771 | 369 / 2,416 | 160 / 881 | 159 / 873 |
| 16 | 229 / 3,112 | 319 / 4,081 | 76 / 855 | 74 / 845 |
| 32 | 226 / 5,647 | 254 / 6,073 | 39 / 879 | 39 / 895 |

Key patterns in these measurements:

- **Single-stream speed and total throughput are two different numbers.** On the Mac, raising concurrency from 1 stream to 16 slows down each individual stream (MLX 4-bit drops from 272 to 87 `tok/s`), while total throughput across all streams climbs nearly 4x (from 244 to 923 `tok/s`). On the GPU with `vLLM`, going from 1 stream to 32 pushes total throughput up 18x (from 309 to 5,647 `tok/s`). Quoting only single-stream speed understates how many users one machine can serve; quoting only total throughput misleads readers into expecting that speed for a single user.
- **Time to first token rises with concurrency.** Requests arriving together queue up for prefill. On the Mac, `llama.cpp` time to first token climbs from 0.17 seconds at 1 stream to 2.7 seconds at 16 streams.
- **4-bit is not always faster than 8-bit.** At 1 stream, 4-bit wins because smaller weights require fewer memory reads. In `llama.cpp` on the Mac, however, `Q8_0` pulls ahead of `Q4_K_M` from 4 concurrent streams onward. The usual explanation is that higher concurrency shifts the bottleneck from reading weights toward compute, and unpacking (dequantizing) 4-bit weights before every matrix operation costs more compute than the smaller memory read saves. We did not verify that mechanism separately in this benchmark.
- **The inference stack determines how performance scales under concurrency.** On the GPU, `llama.cpp Q4_K_M` is the fastest at 1 stream (490 `tok/s`), ahead of `vLLM`. Yet `llama.cpp` total throughput plateaus around 880 `tok/s` by 8 streams; adding more concurrency only slices the same pie thinner, leaving 39 `tok/s` per stream and a 2.6-second time to first token at 32 streams. With `vLLM`, per-stream decode speed drops slowly: at 32 streams each stream still gets 226 to 254 `tok/s`, total throughput reaches 5,600 to 6,100 `tok/s`, and time to first token stays around 0.3 seconds. Serving one person at a desk and serving a group of concurrent users call for different inference stacks.
- **On the GPU, FP8 beats bf16 at every concurrency level** (390 vs. 323 `tok/s` at 1 stream; 6,073 vs. 5,647 `tok/s` at 32 streams), because weight reads shrink by half and the RTX 5090 has native FP8 compute units.
- **The same model behaves very differently across inference stacks.** On the same Mac at roughly 4-bit precision, MLX delivers about 2.5x the total throughput of `llama.cpp`. We also tested `mlx-lm`'s built-in HTTP server (`mlx_lm.server`, version 0.32.2): single-stream speed dropped to 41 to 74 `tok/s`, and at 8 concurrent streams it refused connections. That is why we benchmark MLX in-process with `bench_mlx.py`, which measures the engine itself. In real deployments, the HTTP serving layer can eat into speed as well.

These numbers only measure how fast each setup runs, not how much output quality changes after quantization. For quality trade-offs across quantization tiers, check the published evaluations cited in Section 5.7 or run a side-by-side check on your own tasks.

## Test with your own workload

To swap models: change the GGUF repository and quantization tag after `-hf` in `llama.cpp`, `--model` in `bench_mlx.py`, or the model name after `vllm serve`. To change the workload: adjust `--max-tokens` for output length; to test longer inputs, edit `FILLER` in `bench.py` (currently ~800 tokens); and set `--concurrency` to the number of simultaneous requests you expect. The combination worth measuring most is the one you will actually deploy: your target inference stack, your chosen quantization tier, and your expected concurrency.