Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio
Apple published a research post describing how its open-source MLX array framework now supports running frontier-scale large language models across multiple Mac Studio computers, and it shipped a new MLX release on the day of publication. The headline demonstration is Muse Spark Open, a mixture-of-experts model with nearly a trillion parameters, which Apple reports can be run from a desk across a cluster of four Mac Studios at over 36 tokens per second — faster than average human reading speed. The measurements were made on Mac Studio systems with the M5 Ultra chip at a context size of 8,192 tokens.
The post also benchmarks two smaller open-weight models: Mistral Medium 3.5 at 128 billion parameters and Qwen 3.8 at 27 billion parameters. Apple evaluated Qwen 3.8 and Mistral Medium 3.5 in their native precisions, BF16 and FP8 respectively, and Muse Spark Open in 4-bit NVFP4. The larger two of those models fit on a single Mac Studio with 512GB of unified memory: Qwen 3.8 occupies up to 68GB and Mistral Medium 3.5 up to 158GB at a 65,536-token context. Muse Spark Open does not fit — Apple states it would need more than 1.6TB of memory in BF16 for the parameters alone, dropping to roughly 493GB when quantized to NVFP4, still too much once intermediate computation is counted, so it must be split across machines.
Benchmarks come from the mlx_lm.benchmark tool and cover three metrics: time to first token, generation throughput, and peak power, at context sizes of 8,192 and 65,536 tokens. Apple separates two hardware generations, the new M5 Ultra and the previous M3 Ultra, and attributes the differences to two architectural properties. Time to first token is described as compute-bound, which is where the M5 GPU Neural Accelerator — hardware dedicated to matrix multiplication — pays off, yielding up to 4.6x improvement over M3 Ultra. Decode speed is memory-bandwidth-bound, and the M5 Ultra's 1.2 TB/s of unified memory bandwidth against the M3 Ultra's 819 GB/s translates into roughly 1.6x faster generation. Peak power at the 65,536-token context is reported at about 300W for the M5 Ultra and about 200W for the M3 Ultra, which Apple notes keeps the machine usable in ordinary home or office environments.
The distributed side is the more technically interesting part. Apple describes two ways MLX can split a model. Pipeline parallelization splits the model by depth: one Mac runs the first half of the layers, hands the intermediate output to the second Mac, which runs the rest. Tensor parallelization splits by width, so every machine holds a fraction of each layer and per-layer computation is parallelized. The trade-off is explicit in the post: pipelining demands only minimal code changes, while tensor parallelization is more complex and involves many small communication operations per layer per token, making low-latency interconnect essential.
That interconnect arrives via macOS 26.2, which introduced a low-latency RDMA driver over Thunderbolt 5. MLX v0.30.1 and later builds on it and adds communication collectives called jaccl. In Apple's setup, machines are joined by two Thunderbolt cables per pair to double link bandwidth. Jobs are launched with mlx.launch, whose --backend jaccl flag selects the RDMA Thunderbolt backend, and mlx_lm.benchmark or mlx_lm.server can be prefixed with mlx.launch to spread work across nodes. An experimental flag, MLX_METAL_FAST_SYNCH=1, enables faster CPU-GPU synchronization to cut communication overhead.
Results differ by scheme. Pipeline parallelization leaves decode essentially unchanged, which follows from layers still being evaluated sequentially, but can beat a single node on prefill for very large contexts — up to about 1.8x faster at 65,536 tokens. The reason, per Apple, is that MLX segments long contexts into chunks (8,192 tokens via the prefill-step-size setting) and overlaps consecutive chunks across the two machines. Tensor parallelization, by contrast, speeds up decode meaningfully and scales better with dense architectures. At very large contexts with only two machines, its communication overhead can offset compute gains during prefill; the post notes Muse Spark Open on two machines lags two-machine pipelining there because of extra communication from normalizing expert outputs. Adding machines changes the picture, and the four-M5-Ultra configuration is where tensor parallelization pays off. Bandwidth also matters: on four M5 Ultras at 65,536 tokens, a second cable per link cut Muse Spark Open's time to first token from about 50.5 seconds to 40.6 seconds.
For local agentic workloads, Apple points to mlx_lm.server, which can be launched across Mac Studios and exposes an endpoint on a chosen port. The post gives a configuration example wiring OpenCode to a server on 127.0.0.1:8080 running Muse Spark Open, using an OpenAI-compatible provider block. Apple's stated motivation is privacy — the framework's unified memory architecture lets researchers run inference and fine-tuning on their own hardware rather than in a hosted environment.
A few limitations are worth flagging. The reported 36-plus tokens per second applies at an 8,192-token context on four machines; the 65,536-token numbers are shown as speedups rather than as an absolute rate, and prefill latencies in the tens of seconds are normal for a model of this size. The fast-synchronization flag is described as experimental. The gains quoted for M5 Ultra over M3 Ultra are for the two dense models, not for Muse Spark Open, whose baseline is the two-machine pipeline configuration. And the parallelization features depend on hardware and OS versions — Thunderbolt 5 for RDMA and macOS 26.2 or later — so this is not a path for older machines.
Why it matters: this pushes the practical ceiling of what a developer can run on owned hardware up by roughly an order of magnitude in model size, without a datacenter GPU or a paid API, which matters for teams with privacy or data-residency constraints. The mlx.launch workflow is a modest change for anyone already using mlx_lm, though multi-machine tensor parallelization brings real operational complexity. Inference — this is my own read — the main near-term beneficiaries are agentic coding setups that need long contexts and can tolerate tens of seconds of prefill, and the main open question is how these numbers hold up for larger clusters and newer model architectures.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
MLX on Mac with Apple silicon lets AI researchers and developers experiment locally with the latest models and methods in a private environment. The open-source array framework leverages the unified memory architecture of Apple silicon, enabling anyone to run new inference and fine-tuning methods on their own hardware and explore other AI techniques. In a previous post, we highlighted the advantages of the M5 architecture and its GPU Neural Accelerators.