Teaching Everyone to Fish for Tokens

Nathan Lambert, writing in Interconnects, describes Nvidia's investment in nearly open-source models, including Nemotron, where the company releases all the data it legally can along with training code, as an effort toward a world where many companies can build their own token machines rather than a world where intelligence is monopolized. He reports that Nvidia is spending $26 billion on this endeavor, and says it is not clear whether it will work.

The post distinguishes an open-source language model, which comes with a full training recipe, data and code, from open weight models, which ship with model weights and inference code. It cites the Olmo models Lambert helped build at Ai2 and the earlier Pythia models from EleutherAI as examples of the open-source recipe, and notes that many companies still use workflows built on Llama 3 despite later agentic behaviors.

Lambert writes that open-source AI faces a tricky future because building the best models is extremely capital intensive, and that the default expectation is that the open-source recipe is too far behind for a new lab to be tractable. He says the companies bowing out, such as Databricks and 01.ai, seem like anomalies.

He lays out two futures. In the first, if the open-source recipe works for Nvidia, it creates more demand for its chips than the models cost to build. In the second, if neither financially positive path plays out, open models fork toward efficiency, modifiability and specialization, filling a long-tail ecosystem relative to closed counterparts that hold monopoly stakes in areas such as knowledge work collaboration, drug discovery and software engineering. He calls the second outcome his most likely.

Lambert also describes a shift in which training a base model into a general agentic reasoner is becoming opaque, potentially changing the standard pretraining, midtraining and post-training lexicon toward pretraining, reasoning training and post-training. He notes declining numbers of open model builders releasing base models, alongside experiments with revenue share licenses.