Train Qwen3 on AWS Trainium with torchtitan

AWS published a walkthrough for training Qwen3 on Trainium using torchtitan, the PyTorch-native training stack. The post covers three configurations, all run on a single trn2.48xlarge instance, and all sharing the same torchrun launch pattern and config structure. The key enabler is TorchNeuron, which registers Trainium as a native PyTorch device when torch_neuronx is installed, letting torchtitan detect it at startup without a separate abstraction layer.
The instance has 16 Trainium2 accelerators and 128 physical NeuronCores; at LNC=2 that presents as 64 logical NeuronCores, so the launch commands start 64 training processes. The walkthrough pins torch and torch_neuronx at 2.14.0 and 2.14.3, transformers at 5.16.0, and torchtitan-neuron at release-2.33 (0.2.2+neuron.1), and advises pinning these because the Transformers v5 integration and Trainium override modules can shift between torchtitan releases. Installation is from the aws-neuron/torchtitan-neuron repo via a requirements file, and model artifacts come from the Hub (Qwen/Qwen3-4B-Instruct-2507) into a local directory referenced by the config.
In the first configuration, Transformers v5 acts as the modeling backend: torchtitan builds the model through AutoModelForCausalLM, with AutoConfig.from_pretrained reading the architecture from the Hub. You set the Hub ID and an assets path, pick a dataset (the example uses c4_test), fix steps at 10 and local batch size at 1, and set data_parallel_shard_degree to -1 so parameters are sharded across all 64 ranks. Enabling checkpoint options with an absolute initial_load_path fine-tunes from published weights; an adapter adjusts the model. prefix and handles tied embeddings, and the final checkpoint is written as Hugging Face safetensors. The post reports a step-1 loss of 6.54 from published weights versus 12.43 from random initialization, a useful smoke test, and says a reload round trip shows no missing or unexpected keys apart from lm_head.weight, which Qwen3-4B ties to embeddings.
The second configuration swaps in torchtitan's native Qwen3Model. Because the tested build lacked a generic 4B config builder, the example starts from qwen3_32b() and overrides config.model_spec with model_registry("4B", attn_backend="sdpa"). That field selects the model class and also determines sharding and checkpoint key mapping, which is why the global batch size of 64 appears explicitly here. In my reading, the distinction matters for teams comparing the two paths: the Transformers route leans on upstream Hub code, while the native spec gives torchtitan more control over parallelism and checkpoint layout.
The third configuration exercises mixture-of-experts with kernels from torchtitan-neuron. Using qwen3_nki_moe_config for "30B-A3B" with n_layers=4, num_experts=64 and top_k=4, each token routes to four experts. The model is trimmed to four layers and 64 experts so it fits on one instance. Data parallel sharding spans 64 ranks with tensor parallel degree 1, and there is no expert parallelism; num_experts counts experts in the model, not the expert parallel degree. CompileConfig(enable=False) runs the example in eager mode, though TorchNeuron supports registered NKI operators in both eager execution and torch.compile. The recipe also selects SDPA and a gather-free rotary embedding, and if you turn on torch.compile you should add grad_clip to CompileConfig.components so gradient-norm clipping is compiled too.
The custom-kernel section explains how NKI operators enter training as ordinary PyTorch operators. wrap_nki makes a kernel callable from PyTorch, nki_op registers it, and register_autograd attaches a backward pass using routing metadata saved during the forward pass. The example uses rope_hf from nkilib.core.embeddings, which expects Hugging Face layout [batch, heads, seq, head_dim]; torchtitan stores query and key as [batch, seq, heads, head_dim], so the operator permutes and calls contiguous, allocating outputs with torch.empty_like. A second operator handles the backward mode via a backward=True flag, with the rotary cache stashed in setup_context and None returned for its gradient since it is not differentiable.
The kernel needs a supported sequence length, so the example subclasses CosSinRoPE as NKIRoPE and raises an error unless the sequence length is a multiple of 128 * LNC (256 at LNC=2); the rotary cache is reshaped to [seq, head_dim * 2] with cosine in the first half and sine in the second. Registration uses the @override decorator with derive() to copy shared config fields, and the module is loaded through --override.imports. Torchtitan logs each replaced node before training, so you should see one nki_rope entry per layer. The post is explicit that this demonstrates registration and autograd integration, not a speedup, and reports that float32 numerical comparison of rope_hf against the PyTorch rotary embedding gave a maximum absolute difference of 0.0 for both query and key. When stacking overrides, fqns patterns limit each to intended instances, and torchtitan errors if two overrides claim one node and warns if an override matches nothing.
Output review is part of the workflow: startup should show 64 ranks and the expected model, the first step takes longer because it includes compilation, and each step reports loss, gradient norm, HBM use, throughput and MFU. Scaling guidance is to watch per-rank HBM as model size, sequence length or local batch size grow, and to combine tensor parallelism (each rank computes a slice of every matmul, with sequence parallelism on by default for normalization) with FSDP on a multidimensional DeviceMesh.
Why it matters: teams that want Qwen3 fine-tuning or pretraining on non-NVIDIA silicon now have documented, reproducible recipes rather than a porting project, and the override registry gives kernel authors a way to drop in NKI code without forking the trainer. The practical caveats are version pinning and the need to validate custom kernels numerically; whether these paths deliver competitive throughput is something the post leaves to your own benchmarking, an inference rather than a reported result.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Train Qwen3 on AWS Trainium with torchtitan three ways on a single trn2.48xlarge: the Hugging Face Transformers v5 modeling backend, torchtitan's native Qwen3, and NKI mixture-of-experts kernels. The same torchrun launcher and config structure work unchanged across all three paths, including loading and saving published Hugging Face safetensors.