oMLX Review, Auditing the 90s to 5s Agent Claim

The oMLX launch page makes one promise: a Mac LLM server that cuts agent wait times from 90s to 5s. Taken at face value, that is an 18x improvement delivered by a laptop. This oMLX review treats the number as a hypothesis rather than a verdict, for two reasons. Nobody has published the methodology behind it, and LLM agent latency stacks four separable costs, model load, prompt prefill, decode, and round trips, only some of which a warm local server can eliminate.

The defensible reading is that 18x is a cold-start and round-trip story, not a speed story. Keep the model resident in unified memory and you delete model load time, provider queuing, and most network overhead, and against a baseline dominated by those costs the arithmetic works. In a steady-state agent loop, wait time converges on prefill and decode, and there Apple Silicon rarely delivers anything close to 18x against a fast hosted endpoint. This article decomposes the budget, runs five worked latency scenarios, and hands you an afternoon benchmark, so the ratio you act on is one you measured yourself.

What This oMLX Review Audits

oMLX is an open-source project that serves LLMs locally on a Mac through Apple's MLX stack; the oMLX GitHub repository covers installation and setup. The product is the easy part of this audit. The claim is harder.

The 90 seconds to 5 seconds figure comes from the launch page, and the page publishes no test conditions: no model, no quantization, no chip, no description of what the agent was doing, and no account of what the API baseline actually suffered through. Without those details the figure is a best-case anecdote. That is not a dismissal; field anecdotes show the extreme case, and they say nothing about the typical one.

The implicit baseline is a hosted API agent loop where each task pays cold start, queueing, or both. The implicit local condition is a warm server, where the model never leaves memory. Those two words carry the whole claim. Cold model load means streaming multi-gigabyte weights from disk into unified memory before the first token, typically seconds to tens of seconds depending on model size and storage. Warm means resident, effectively zero per request. The 5s half of the claim quietly assumes warm. If the 90s half assumed a cold or heavily queued provider, the comparison is a load-time comparison dressed as a speed comparison.

One scoping note: oMLX replaces the serving layer, the endpoint your agent client calls. It does not replace model quality, context handling, or your framework's own tool-dispatch overhead, all of which show up in your end-to-end numbers anyway.

Where Agent Wait Time Goes on Apple Silicon

MLX inference on Apple Silicon runs on Apple's purpose-built array framework; the MLX documentation covers the GPU and unified memory model underneath. Auditing the claim requires the prefill versus decode decomposition that serving engineers use, because the two phases stress different hardware and reward different fixes.

Model load, cold versus warm

Cold load is a one-time transfer of quantized weights from disk into unified memory. A 4-bit 8B model occupies roughly 4 to 5 GB; 70B-class models run to tens of GB. On fast internal storage that transfer typically costs seconds to tens of seconds. Warm load costs nothing per request. This is the one component a local server eliminates outright, and it is also a component some API baselines pay repeatedly, serverless endpoints after every idle timeout, or any setup that swaps models between calls. Against a warm dedicated endpoint, though, neither side pays it, and the 18x loses its main fuel.

Prompt prefill, the cost that grows every turn

Every turn of an agent loop resends the whole conversation: system prompt, tool schemas, prior turns, tool outputs. Prefill compute grows with context length, so the local bill for history grows every turn. Community figures for mid-size quantized models on Pro and Max chips commonly put prompt processing in the low thousands of tokens per second; call it 1,500. A 2k token turn costs about 1.3 seconds. A 50k token history costs more than 30 seconds, paid again on every turn. In multi-turn loops, prefill of the growing context is often the dominant local cost, which makes context discipline (trimming, summarizing, capping tool output size) more consequential than peak tokens per second.

Decode, bound by memory bandwidth

Generating each token means streaming weights and the KV cache through the memory system, so decode is largely memory-bandwidth bound. M-series memory bandwidth spans roughly 120 GB/s on base chips to several hundred GB/s on Max and Ultra parts, while datacenter GPUs push multiple terabytes per second. In practice, mid-size quantized models commonly decode at tens of tokens per second on a Mac, so a larger or less-quantized model can decode slower locally than a fast hosted API even with zero network latency. Research on KV cache growth covers the mechanics: it scales with context length and stacks memory pressure on top of the weights, which on memory-tight machines invites mid-loop failures.

Round trips and queueing

An API loop pays network round trip time, TLS setup, provider queueing under load, and rate-limit stalls, commonly tens to hundreds of milliseconds per turn before any compute happens. A localhost call is about a millisecond, and the only queue is your own GPU. A warm local server genuinely deletes this component. Its weight in the total shrinks, though, whenever decode dominates the turn, which in long-output agent tasks is most of the time.

Five Worked Budgets Show When 18x Holds

The oMLX 90s to 5s claim decomposes cleanly once you write out component budgets. The figures below are illustrative arithmetic with every assumption shown, not measurements; the point is the method, and every constant is one you will replace on your own machine.

Start with the marketing scenario. The API baseline pays about 80 seconds of cold start and queue plus 10 seconds of inference. The Mac is warm: under a second of prefill for a 1k token prompt and about 4 seconds to decode 150 tokens at 40 tokens per second, call it 5 seconds total. That is the shape of the whole claim: an 18x ratio is arithmetically plausible only when the baseline is dominated by costs a warm local server avoids, cold loads, queues, network round trips. In steady state, the gap narrows toward raw compute differences.

Run the same loop warm on both sides and the arithmetic flips. Assume a ten-turn tool loop, 2k tokens of context per turn, 300 output tokens per turn. The API turn costs about 3.8 seconds (0.1 network, 0.2 queue, 0.2 prefill, 3.3 decoding at ~90 tok/s), so 38 seconds total. The local turn costs about 8.8 seconds (1.3 prefill at ~1,500 tok/s plus 7.5 decoding at ~40 tok/s), so 88 seconds. Local is well over 2x slower, with zero network in the picture.

When is 18x arithmetically honest? The list is short:

• The baseline pays cold start, model swap, or a long queue per task (serverless endpoints, cold dedicated instances).
• Contexts stay small and outputs stay short, so local prefill and decode never dominate.
• The model is small and quantized enough to decode briskly on your chip.
• The workload is many tiny round-trip-bound calls, where deleting network and queue is most of the win.

Where does it collapse?

• Warm steady-state loops against a warm API endpoint.
• Contexts that grow past tens of thousands of tokens, where per-turn prefill eats the local budget.
• Parallel agents contending for one GPU.
• Any comparison that includes frontier model quality, which no local open-weight run replicates.

An oMLX Benchmark Protocol You Can Run in an Afternoon

To benchmark a local LLM server on your Mac fairly, fix everything fixable and measure one component at a time. The protocol below fits in an afternoon and produces numbers you can compare across weeks and model swaps.

1. Fix the workload. Build a context ladder of prompts at roughly 500, 4k, 16k, and 64k input tokens, cap output at a fixed 256 tokens, and set temperature to 0. A benchmark with a drifting workload measures nothing.
2. Measure cold, then warm. Kill the server, restart it, and time from launch to the first completed request. Immediately repeat the same request and time that. Record the model's size on disk next to both numbers.
3. Capture TTFT and inter-token latency. Use a streaming client that timestamps the first token and every token after it. The standard serving metrics exist because time to first token, inter-token latency, and end-to-end latency each answer a different question; report all three rather than a single average.
4. Compute decode rate. Tokens between first and last token, divided by the interval. This is the number hardware reviews stop at, and the one your agent feels least.
5. Run the end-to-end tool loop. One fixed task that forces five tool calls, with per-turn wall time logged. This number, not tokens per second, is what your agent user experiences.
6. Sanity-check against the community. The mlx-lm package ships generation tooling you can time directly, and the long llama.cpp Metal benchmarks thread collects per-chip figures from many machines. If your numbers are wildly off those ballparks, suspect your harness before your hardware.

Record every run in one template:

A run that omits chip, memory, model, quantization, or context length cannot be compared to anything, because each of those fields changes the answer. And since laptops throttle, run repeats, note the drift, and keep the machine plugged in.

Reading Your Numbers and Choosing Local vs API

The oMLX vs cloud API decision is really a workload archetype question, and the local LLM vs API choice for agent loops falls straight out of your measurements.

Your numbers point local when:

• TTFT stays within roughly 2x of your API endpoint at your real context length, and decode rate is livable for your output lengths.
• Contexts stay bounded. Summarize or trim history before prefill does the damage; to reduce LLM agent wait time on your Mac, context discipline beats any server swap.
• You run one agent or low concurrency, and the loop runs all day, where per-token API spend and data egress dominate the decision.
• Privacy or air-gap requirements are hard constraints.

They point back to the API when contexts grow past tens of thousands of tokens, when you fan out parallel agents, or when the task needs frontier model quality. On quantization: 4-bit weights decode faster and fit comfortably in unified memory, 8-bit buys quality back at a decode cost, so benchmark your quant rather than assuming it. And test thermals under your real cadence, because a loop that runs for hours meets a different machine than a five-minute benchmark does.

Verdict on oMLX for Agent Workloads

What oMLX demonstrably does: it deletes cold load from steady-state loops on one machine, removes provider queueing and network round trips, and keeps every token on your side of the wire. For short-context, single-agent, latency- and privacy-sensitive loops, that is a real win, and the server is open enough to audit, which the launch claim is not.

What remains unverified: the 90-second baseline (whose cold start, which provider, what queue), the conditions behind the 5-second local figure (which chip, model, quantization), and therefore the 18x ratio as a general statement. An honest oMLX review ends where the evidence ends.

Guidance by archetype:

• Adopt: interactive single-agent tools, short contexts, privacy-bound or offline work, all-day loops where API cost dominates.
• Test first: medium contexts (8k to 32k), tool-heavy loops, any quantization you have not personally benchmarked.
• Skip for now: long-context work, multi-agent fan-outs, and anything that genuinely requires frontier models.

The 18x headline is best read as a true story about cold starts told in the language of speed. Warm both sides, measure your own loop, and let the ratio you can reproduce decide.