Training a 1T-Parameter MoE on EU public infrastructure with PyTorch and TorchTitan

The EUROPA consortium — formed by Domyn and Fraunhofer IAIS and IIS — has described how it scaled pretraining of a roughly 1-trillion-parameter Mixture-of-Experts model on JUPITER, the EuroHPC exascale supercomputer hosted at Forschungszentrum Jülich. The work took place under the EuroHPC Frontier AI Grand Challenge, which was set up to test whether sovereign, open-source AI could compete at frontier levels on shared public resources. Throughput rose from 90 to about 1,300 tokens/s per GPU across 384 NVIDIA GH200 GPUs per data-parallel replica (96 nodes); the final setup runs 1,536 GPUs over four replicas. The consortium's brief also calls for an open European MoE of at least 400B parameters, support for all 24 official EU languages and strict EU AI Act compliance.
The configuration used in the experiments lists 951.9B total parameters with 32.8B active per token, 74 layers, a hidden size of 6,144, 384 routed experts with top-8 selection, expert hidden size of 1,792, a 256,000-token vocabulary and a pretraining sequence length of 65,536 tokens. Optimization is split: Muon for about 99.5% of parameters (every weight matrix except the router and embeddings) and AdamW for the remainder.
What makes JUPITER unusual is that it is publicly funded infrastructure shared by hundreds of projects rather than a cluster built for one model. Each Booster node has four GH200 Grace Hopper Superchips, each pairing a 96 GB HBM3 Hopper GPU with a 72-core Grace CPU and 120 GB of LPDDR5X over a 900 GB/s NVLink-C2C link. GPUs within a node talk over NVLink 4 at 150 GB/s per direction, but each node has only four InfiniBand NDR200 adapters, about 25 GB/s per GPU. The write-up notes InfiniBand is 36 times slower than the GPU-to-CPU link. The compute nodes also use the 120 GB host-memory configuration, not the 480 GB option, leaving roughly 108 GiB usable per GPU after the kernel's share.
Those constraints shaped the parallelism: 384-way FSDP, 4-way expert parallelism inside each node and no pipeline parallelism. The MoE token all-to-all stays on NVLink because it is hard to overlap, and the absence of pipeline parallelism keeps steps short and feedback fast. Fixed costs are amortized with a batch of 384 sequences of 65,536 tokens, about 25M tokens per step. The remaining effort went into three directions: longer sequences so compute runs longer per gather, fewer bytes over InfiniBand, and using Grace memory to hold model, gradient and optimizer state.
Muon keeps one fp32 state per parameter versus two for Adam, which the team says was instrumental in fitting the 65,536-token sequence. Its Newton-Schulz orthogonalization works on whole matrices, and a naive FSDP implementation would gather each matrix on one rank and scatter it back — too expensive here. FlexShard, now in TorchTitan, lets them choose parameter ownership: experts are sharded whole, so each rank runs Newton-Schulz on complete expert matrices without communication, while dense matrices are gathered on an owner rank and the communication overlaps with expert work. Initially Newton-Schulz ran on Grace's 72 Arm cores at about 35 seconds per step; moving it to the GPU while parameters stayed offloaded cut the optimizer step from 41.7 s to 2.2 s and lifted throughput from 441 to 597 tokens/s per GPU, a 35% gain.
Activation checkpointing came next. Storing activations only at layer boundaries and recomputing inside layers (FullAC in TorchTitan) raised throughput from 90 to 370 tokens/s at 32,768 tokens. At 65,536 tokens the boundary activations alone come to about 55 GiB per GPU, so they are copied to Grace after the forward pass and returned one layer before recomputation during backward, with a parameter controlling how many layers offload. That took throughput from 370 to 508 tokens/s (+37%) and allowed sequences of 49,152 and then 65,536 tokens. In practice this is a clean illustration of the GH200 design: the 900 GB/s C2C link is fast enough to treat host memory as a slower tier of accelerator memory.
A CUDA allocator problem produced the third major win. After a software upgrade, allocations began retrying on every rank at about 70% memory use despite 25-30 GiB free. The cause was FSDP's use of one compute stream and four side streams with per-stream memory pools: freed memory can only be reused by its owning stream. At peak, side streams held 27.3 GiB of which 6.4 GiB was live, leaving 20.9 GiB unreachable when the compute stream needed a 4 GiB MoE buffer. FSDP2's private unshard_async_op flag moved all-gather setup onto the compute stream; retrying ranks fell from 384 to zero, peak reserved memory dropped from 88.1 to 74.6 GiB and throughput rose from 996 to 1,210 tokens/s. The flag is being promoted to a public FSDP API (PyTorch #202343), which matters beyond this project: any FSDP user mixing communication streams can hit the same fragmentation.
Other upstream contributions include a shared RoPE cache in TorchTitan (2.56 GiB per GPU at this sequence length), skipping the embedding re-gather in backward, configurable seed-offset mesh axes, and PyTorch fixes for CPU initialization resetting the RNG state and for trunc_normal_ all-gathering DTensors. The team also reported unattributed host memory at job start to the Jülich Supercomputing Centre, since it affects all users of the machine. The companion post is said to cover data and expert load balancing, reducing InfiniBand bytes, node-local resharding, dynamic tile scheduling in SonicMoE and capping NCCL's SM usage.
Why it matters: Teams training large MoE models on shared or rented clusters face the same bottlenecks — thin inter-node links, limited host memory and stream-level allocator fragmentation — and the fixes here are being merged into PyTorch and TorchTitan rather than staying proprietary. Developers using FSDP should watch for the public unshard_async_op API, which could recover usable HBM without code changes; Muon users get a placement strategy that avoids gathering matrices. A reasonable inference is that public HPC systems can host competitive frontier training when software is flexible enough, though these results cover a single model configuration on one machine.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
TL;DR Scaling pretraining for a ~1-trillion-parameter Mixture-of-Experts (MoE) model on shared, public infrastructure was previously considered unfeasible due to unique architectural and interconnect bottlenecks. However, by leveraging the fine-grained flexibility... The post Training a 1T-Parameter MoE on EU public infrastructure with PyTorch and TorchTitan appeared first on PyTorch .