AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Beyond Kubernetes at Modal: How to Scale 1 Million Concurrent Sandboxes in Seconds

Collected Sep 30, 2026

Modal staff engineers Colin Weld and Connor Adams described rebuilding the company's sandbox infrastructure from the ground up to support millions of concurrent sandboxes and tens of thousands of sandbox creations per second, in an article covered by InfoQ.

The engineers said traditional container orchestration systems such as Kubernetes struggle at this scale because they rely heavily on centralized coordination and strongly consistent state. Running 1 million sandboxes requires many tens of thousands of compute nodes, producing operations whose cost grows with the number of containers, nodes, or both, they said. In Kubernetes' case, they noted, load on the scheduling algorithm and on the central durable store etcd grows with nodes and pods, and both write to etcd multiple times, which they said can cause serious issues under high pod creation rates or churn. They added that etcd is not natively shardable within a keyspace, and that overcoming these limits is feasible but requires serious work, including rewriting or replacing etcd and parallelizing scheduling.

Modal's change was to stop coordinating globally and make scheduling resemble load balancing. Instead of a central datastore as the source of truth, each worker became its own source of truth, and a fleet of scheduling servers operates in parallel. A scheduling server picks a worker and contacts it directly via RPC; workers accept if they have free resources and otherwise reject.

The engineers said the resulting architecture has one bottleneck: all workers publish state as a single Redis stream, which load testing suggested remains viable until well over 100,000 workers. In their benchmark, they created 1 million sandboxes in under a minute, with median startup-to-code time under 0.5 seconds.

Commenting on LinkedIn, Hopsworks CEO Jim Dowling said new technical problems arise at every order-of-magnitude increase in scale, suggesting the team iterated multiple times to reach a reliable 50,000 sandbox creations per second. AWS principal AI engineer Alex Jones said a key part was not extending Kubernetes but walking around it after understanding its limitations, calling it the first credible signal that Kubernetes isn't adapting fast enough to what GenAI infrastructure needs, and arguing coordination is decoupling from execution.

Modal is a serverless compute platform for AI workloads, providing access to CPUs, GPUs, containers, inference, training, batch jobs, and isolated sandboxes. Other projects cited as pursuing similar goals include Unikraft, Google Substrate, and Overdrive.

Read at InfoQ · AI, ML & Data Engineering

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

In a recent article, Colin Weld and Connor Adams, staff engineers at Modal, describe how they rebuilt their sandbox infrastructure from the ground up to support millions of concurrent sandboxes and tens of thousands of sandbox creations per second. By Sergio De Simone