AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Welcome Beam: a 501B open model you can own, customize, and scale

Collected Oct 16, 2026

Reflection, a new US lab, has released its first model, Beam, on Hugging Face under the Apache 2.0 license. Beam is a text-in, text-out mixture-of-experts (MoE) language model with 501 billion total parameters, of which about 23 billion are active per token. It targets coding and agentic workloads and is positioned for teams that want to run a frontier-scale model on their own hardware, infrastructure or cloud and customize it. The release includes a dedicated Rust-based harness, inference recipes, and supervised fine-tuning and reinforcement learning recipes built with the Reflection team.

The architecture is spelled out in some detail. The decoder has 52 layers with a hidden size of 5,120 and uses hybrid attention: 43 layers apply sliding-window attention over the previous 512 tokens, while every sixth layer plus the final one attends to the full context, giving 9 global layers. Sliding-window layers use RoPE for position; global layers use no positional encoding and instead scale each query by a factor that grows slowly with visible tokens, which the implementation calls scalable-softmax query scaling and describes as acting like a temperature that keeps softmax precision as context lengthens. Attention uses grouped-query attention with 48 query heads and 8 key-value heads, RMSNorm on queries and keys, and a learned sigmoid gate that scales each channel of each head before the output projection.

The first 3 layers use a dense SwiGLU MLP. The remaining 49 use a Mixture-of-Experts block with 256 experts each, every one a SwiGLU MLP with intermediate size 2,560 (roughly 39M parameters). Those experts hold 98.4 percent of the model's parameters. Routing follows DeepSeek-V3's group-limited scheme: score all 256 experts with a sigmoid, split into 8 groups of 32, keep the 4 best groups by the sum of each group's top two scores, then pick the top 8 experts inside them. Beam uses the same group sizes as DeepSeek-V3 but no shared expert, combines expert outputs with normalized router weights, and scales by 2.5. Because only 8 experts run per token, each token is computed with roughly 23B parameters, but all 256 experts must stay in memory: deployment needs the memory footprint of a 501B model while running at the speed of a 23B one. The BF16 checkpoint is about 1 TB in safetensors format, which is the central practical constraint of the release.

Residual connections are the most unusual design choice. Where a standard transformer adds each sublayer output with weight 1, Beam counts the input embedding as step 1 and at step k adds the sublayer output with weight 1/sqrt(k) while keeping the existing stream with weight sqrt((k-1)/k). The squares of the two weights sum to one, so the stream stays at constant scale from first layer to last, and after all 104 additions every sublayer output and the input embedding carry equal weight in the final hidden state. Every sublayer is normalized before and after, and final logits pass through a tanh softcap at plus or minus 30 computed in FP32. The implementation fixes FP32 computation for the router, the SwiGLU product, the norms, RoPE, the query scaling and the softcap.

Long-context behavior gets a concrete accounting. Each layer caches 2 x 8 KV heads x 128 dims, or 2,048 values per token. If all 52 layers kept every token, the cache would grow by 208 KiB per token; because only the 9 global layers keep the whole sequence and the 43 sliding-window layers hold a fixed 512-token window, growth is 36 KiB per token in BF16, roughly a sixth of the naive rate under the transformers dynamic cache.

The model ships in three checkpoint formats. BF16 is the full-precision option at roughly 1 TB. FP8 checkpoints suit Hopper and Ampere-class chips. NVFP4 stores most parameters in 4-bit precision for NVIDIA Blackwell GPUs, with some parameters kept in BF16 for accuracy, shrinking the weights from about 1 TB to roughly 280 GB while retaining most of the BF16 quality according to the Reflection-provided results.

Hardware guidance follows from those sizes. A B200 has 180 GB of HBM3e, so two would hold the weights with little room for activations and KV cache; four B200s (720 GB) is the practical plan, and a full 8xB200 node also works, as do two B300s at 288 GB each or 8xRTX Pro 6000 cards. The FP8 path is for pre-Blackwell hardware such as the H200. Under transformers, Beam's config defines an expert-parallel plan that places each expert on one device while replicating attention, dense layers, embeddings and the LM head (8B parameters, 16 GB in BF16) everywhere; the number of devices must divide 256. A 16xH200 torchrun launch puts 78 GB of weights per GPU, whereas 8 GPUs would need 139 GB each and not fit on a 141 GB H200.

Tooling support is broad from day one. Transformers (version 5.20.1 or later), vLLM and SGLang all support the model on release, and full-parameter RL is available with Miles. Inference can run through the text-generation pipeline or AutoModelForCausalLM, and a response template lives in the tokenizer so that parse_response can split reasoning, answer and tool calls into separate fields rather than relying on regexes; passing tool schemas lets the parser cast inline arguments to their declared types. Beam supports five reasoning effort values: low, medium (the default), high, xhigh and max, and it can preserve thinking across turns.

A DFlash2 speculative decoding drafter is provided, proposing up to 15 tokens ahead for Beam to verify in one forward pass, at the cost of extra memory for the drafter. Serving examples cover vLLM with tensor parallelism across 16 GPUs (two 8xH200 nodes) and an OpenAI-compatible endpoint, and SGLang with DFlash2 enabled, where the NVFP4 checkpoint fits on 4 B300s with the trtllm_mha attention backend, while FP8 on 8 H200s uses the fa3 backend instead. Beam is also served on Hugging Face Inference Providers through Baseten, Fireworks and Together, and can be deployed on Inference Endpoints in FP8 or NVFP4, including via the hf cli command for a catalog deployment.

Reflection and Hugging Face also published post-training recipes. For supervised fine-tuning, TRL's SFTTrainer with LoRA adapters and PEFT keeps memory manageable rather than updating all 501B parameters; the setup combines sharded loading, expert parallelism, FlashAttention 4 and sequence packing, fitting on a single 8xB300 node at around 19,000 tokens per second, with an equivalent H100 setup needing four nodes. After one epoch on Multilingual-Thinking, Beam reasoned in the requested language on 16 of 16 held-out prompts versus 3 of 16 for the base model, a narrow evaluation of language adherence rather than reasoning correctness. For reinforcement learning, a GRPO recipe built with RadixArk trains in BF16 through Miles, with Megatron training, SGLang generating rollouts and weights published to rollout engines after each update; it runs on DAPO-Math-17K and evaluates on AIME25 using a Ray cluster of twelve 8xH200 nodes.

Hugging Face is also running a $10,000 Hub credits program for the best projects built with Beam, covering demos on Spaces, quantizations, synthetic datasets, RL environments, benchmarks or anything else, submitted by sharing a Space, model or dataset repo on the model's repo page. Separately, Hugging Face shared five tasks run once each at low and high reasoning effort through the Reflection platform, describing them explicitly as vibe evals rather than benchmarks, with inputs, outputs, API usage, fixtures, graders and traces published in a public bucket. In those runs, the low-effort long-context response got all code behavior and all six source citations right but did not match the parser's format, which the writeup attributes partly to prompt ambiguity, and low omitted a requested justification that high supplied with a reviewed roots-of-unity proof.

Why it matters: An Apache 2.0 frontier-scale model with a 1M-token window changes the build-versus-buy calculus for teams with strict data or latency requirements, since they can now self-host, fine-tune and quantize rather than route everything through an API. The practical friction is memory: full BF16 weights are about 1 TB, so most teams will need FP8 on Hopper-class GPUs or NVFP4 on Blackwell, and expert parallelism imposes device-count constraints. For product teams, the tokenizer-level response template and dedicated tool-call grammar could reduce parsing work in agent pipelines, and the published SFT and RL recipes lower the cost of adapting Beam to a specific domain, though the benchmark and evaluation evidence to date is thin enough that independent testing still matters.

Read at Hugging Face Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt