Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline

NVIDIA and AI4Bharat, the Indian language research lab at IIT Madras, announced Nemotron-pretraining-Indic-v1, a pretraining corpus of roughly 2.8 trillion tokens spanning 22 Indian languages, plus the open, GPU-accelerated data generation pipeline used to build it. The release is described as coming soon. The authors include Mohammed Safi Ur Rahman Khan, Mitesh Khapra and Anoop Kunchukuttan from AI4Bharat, and Niket Agarwal, Raviraj Joshi, Anusha Kamath, Markus Kliegl, Yev Meyer, Sriharsha Niverty, Kiran Praveen, Utkarsh Vaidya and others from NVIDIA.
The 22 languages are those listed in India's Constitution, which the announcement notes are first languages for 97% of Indians, or nearly 1.2 billion citizens at the 2011 Census. The corpus contains about 2,601.0 billion native-script tokens and 224.6 billion romanized tokens, counted with the Gemma-4 tokenizer, covering 34 language-script configurations.
Twelve languages get both native-script and romanized data: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu and Urdu. The remaining ten are native-script only: Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali and Sindhi. Native-script here means the writing system used for text translated from English; romanized means Indic-language text written in Latin script rather than an English translation. For the twelve dual-script languages, romanization covers a subset of accepted native-script translations. The token counts describe the retained corpus, including accepted rewrites, not a training run's budget, and they vary with the tokenizer chosen.
The pipeline starts from documents in 15 Nemotron-CC-v2 subsets—web text, question-answer material, mathematics, code and other specialized content—and runs translation, quality assessment and targeted repair through NeMo Data Designer, NVIDIA's open-source synthetic data framework. Each output retains a source identifier so it can be traced back to the original document.
For the twelve-language workflow, documents are translated in full so the model can use context across paragraphs, tables and equations. Gemma-4-31B-it was chosen after comparing models on complete documents for translation quality and preservation of technical content; prompts separate prose to translate from content to preserve, such as code, equations, URLs and formatting. The same model then scores each output against its English source using a rubric inspired by Multidimensional Quality Metrics (MQM), an open framework for classifying translation errors by type and severity, covering meaning, fluency, terminology, style and formatting. Only outputs scoring at least 75 out of 100—a rubric score, not an accuracy percentage—are kept; lower-scoring outputs are repaired using the assessment feedback and reassessed against the same threshold, a step that recovered 438.4 billion tokens already included in the corpus total. Sampled human reviews and deliberately introduced test errors informed prompt and check improvements.
Accepted native-script translations also seed romanized data: a subset is converted to Latin script and checked against its native-script input, separating translation from script conversion. In an initial human comparison, this two-step approach produced fewer hallucinated additions and less residual English than generating romanized text directly from English.
The ten remaining languages use a different workflow built around Indic Translate, a document-level English-to-22-Indian-languages translation model from Bodhan AI, a nonprofit AI-in-education center incubated at IIT Madras, with Gemma-4-31B-it assessing translations against the English source. Eight languages keep the whole-document approach. For longer Santali documents the pipeline translates overlapping windows and reconstructs the document, while Manipuri is translated into Bengali script and then converted to Meitei Mayek. These flows keep outputs scoring at least 75 out of 100 and produce native-script data only, without feedback-based rewriting.
On results, the Nemotron 3.5 Super VL technical report compares the checkpoint immediately before Indic continued pretraining with the checkpoint after it. The configured budget was 100 billion training tokens, 10% of it (10 billion tokens) drawn from a twelve-language subset of this corpus plus natural Indic text, with the other 90% keeping the preceding mixture's relative proportions. Scores rose on all five reported Indic knowledge and reasoning benchmarks: MILU, Indic MMLU-Pro, Indic ARC-Challenge, Indic ARC-Easy and Indic GSM8K, with gains from 4.98 to 9.82 percentage points. GSM8K excludes Odia because evaluation exceeded the serving context limit. The comparison covers a stage combining synthetic and natural Indic data, so it does not isolate the synthetic corpus's contribution, and the benchmarks differ in language coverage.
Comparing released models, Nemotron 3.5 Super VL scores higher than Nemotron 3 Super on five of six task benchmarks spanning knowledge, reasoning, mathematics, code and translation, while IFEval 1.5 instruction following dipped 2.04 points. It improves on all four IndiVibe dimensions—usefulness, fluency, language and script fidelity, and appropriate verbosity. IndiVibe is Sarvam AI's open benchmark of assistant-style prompts in Indian languages, and the reported scores are conditional preference scores, not direct head-to-head win rates.
For developers, configurations use language, script and region tags such as hi-Deva-IN for Hindi in Devanagari and hi-Latn-IN for romanized Hindi. The posted example uses Hugging Face Datasets to list splits and stream a sample so records can be inspected without downloading the whole corpus; the guidance is to pin a dataset revision for reproducible experiments, compare mixtures at a fixed token budget while recording natural/synthetic and native/romanized shares, and account for records derived from the same source across languages and scripts when creating held-out splits. Pairing options include Nemotron post-training data collections and Nemotron-Personas-India personas.
The generation recipe is shipped as the open-source Nemotron Bharat translation plugin for NeMo Data Designer, covering translation and evaluation prompts, optional feedback-based rewriting, document windowing and script remapping. The advice is to sample your own source documents first, check translation quality plus preservation of code, mathematics and formatting, refine prompts with native-language reviewers before scaling, and treat generation and corpus acceptance as separate steps, since the plugin returns candidates and scores while your filtering policy decides what enters training.
Limitations are stated plainly. Translation broadens access but does not replace text originally written in Indian languages or guarantee coverage of local knowledge and culture, so the corpus should be combined with suitably sourced natural Indic text. Quality assessment is mainly automated with sampled human review of varying coverage per language, and in the twelve-language workflow the translator also serves as judge. Retained text can still contain errors and the ten-language workflow needs broader native-speaker validation. Source-ID checks remove processing duplicates, not semantic duplicates. Alongside the corpus, AI4Bharat with NVIDIA support is building Indic-LLM-Eval-Suite, an open and evolving benchmark collection for Indic model capabilities.
Why it matters: teams building Indic language models have had far less open training data than English-focused projects, and this release combines a large synthetic corpus with the recipe used to produce it. In practice, the pipeline may matter more than the fixed token counts, because it lets a team point the same translation, scoring and repair steps at its own documents and scaling choices; that is my reading, not a claim from the announcement. The main caveats to plan around are the stated gaps around natural-language coverage, automated quality judging and unvalidated outputs in the ten-language track.
Based on reporting from the original publisher. Visit the source for full context and later updates.