docs: update pretraining docs (#981)

akoumpa · jgerh · web-flow · commit b32dc1fbc605 · 2025-12-19T09:25:14.000-08:00
Signed-off-by: Alexandros Koumparoulis &lt;akoumparouli@nvidia.com&gt;
Co-authored-by: jgerh &lt;163925524+jgerh@users.noreply.github.com&gt;
diff --git a/docs/guides/dataset-overview.md b/docs/guides/dataset-overview.md
@@ -1,6 +1,6 @@
 # Dataset Overview: LLM and VLM Datasets in NeMo Automodel
 
-This page summarizes the datasets already supported in NeMo Automodel for LLM and VLM, and shows how to plug in your own datasets via simple Python functions or purely through YAML using the `_target_` mechanism.
+This page summarizes the datasets already supported in NeMo Automodel for LLM and VLM, and shows how to plug in your own datasets using simple Python functions or directly through YAML using the `_target_` mechanism.
 
 - See also: [LLM datasets](llm/dataset.md) and [VLM datasets](vlm/dataset.md) for deeper, task-specific guides.
 
@@ -84,7 +84,7 @@ dataset:
   split: "0.99, 0.01, 0.00"  # train, validation, test
   splits_to_build: "train"
 ```
- - See the detailed pretraining guide, [Megatron MCore Pretraining](llm/mcore-pretraining.md), which uses MegatronPretraining data.
+ - See the detailed pretraining guide, [Megatron Core Dataset Pretraining](llm/pretraining.md), which uses MegatronPretraining data.
 
 > ⚠️ Note: Multi-turn conversational and tool-calling/function-calling dataset support is coming soon.
 
diff --git a/docs/guides/llm/mcore-pretraining.md b/docs/guides/llm/mcore-pretraining.md
diff --git a/docs/guides/llm/nanogpt-pretraining.md b/docs/guides/llm/nanogpt-pretraining.md
diff --git a/docs/guides/llm/pretraining.md b/docs/guides/llm/pretraining.md
diff --git a/docs/guides/overview.md b/docs/guides/overview.md
@@ -1,8 +1,8 @@
-## Recipes and E2E Examples
+## Recipes and End-to-End Examples
 
 NeMo Automodel is organized around two key concepts: recipes and components.
 
-Recipes are executable scripts configured via YAML files. Each recipe defines its own training and validation loop, orchestrated through a `step_scheduler`. It specifies the model, dataset, loss function, optimizer and scheduler, checkpointing, and distributed training settings—allowing end-to-end training with a single command.
+Recipes are executable scripts configured with YAML files. Each recipe defines its own training and validation loop, orchestrated through a `step_scheduler`. It specifies the model, dataset, loss function, optimizer, scheduler, checkpointing, and distributed training settings—allowing end-to-end training with a single command.
 
 Components are modular, plug-and-play building blocks referenced using the `_target_` field. These include models, datasets, loss functions, and distribution managers. Recipes assemble these components, making it easy to swap them out to change precision, distribution strategy, dataset, or task—without modifying the training loop itself.
 
@@ -14,7 +14,7 @@ This page maps the ready-to-run recipes found in the `examples/` directory to th
 ## Large Language Models (LLM)
 This section provides practical recipes and configurations for working with large language models across three core workflows: fine-tuning, pretraining, and knowledge distillation.
 
-### Fine-tuning
+### Fine-Tuning
 
 End-to-end fine-tuning recipes for many open models. Each subfolder contains YAML configurations showing task setups (e.g., SQuAD, HellaSwag), precision options (e.g., FP8), and parameter-efficient methods (e.g., LoRA/QLoRA).
 
@@ -30,7 +30,7 @@ Starter configurations and scripts for pretraining with datasets from different
 - Example models: GPT-2 baseline, NanoGPT, DeepSeek-V3, Moonlight 16B TE (Slurm)
 - How-to guides:
   - [LLM pretraining](llm/pretraining.md)
-  - [Pretraining with Megatron-Core datasets](llm/mcore-pretraining.md)
+  - [Pretraining with NanoGPT](llm/nanogpt-pretraining.md)
 
 ### Knowledge Distillation (KD)
 
@@ -49,11 +49,11 @@ Curated configurations for benchmarking different training stacks and settings (
 
 
 ## Vision Language Models (VLM)
-This section provides practical recipes and configurations for working with vision-language models, covering fine-tuning and generation workflows for multimodal tasks.
+This section provides practical recipes and configurations for working with vision language models, covering fine-tuning and generation workflows for multimodal tasks.
 
-### Fine-tuning
+### Fine-Tuning
 
-Vision-language model fine-tuning recipes.
+Fine-tuning recipes for VLMs.
 
 - Folder: [examples/vlm_finetune](https://github.com/NVIDIA-NeMo/Automodel/tree/main/examples/vlm_finetune)
 - Representative family: Gemma 3 (various configurations)
@@ -73,4 +73,4 @@ WAN 2.2 example for diffusion-based image generation.
 
 ---
 
-If you are new to the project, begin with the [Installation](installation.md) guide. Then, select a recipe category above and follow its linked how-to guide(s). The provided YAML configurations can serve as templates—customize them by adapting model names, datasets, and precision settings to match your specific needs.
+If you are new to the project, begin with the [Installation](installation.md) guide. Then, select a recipe category above and follow its linked how-to guide(s). The provided YAML configurations can serve as templates—customize them by adapting model names, datasets, and precision settings to match your specific needs.
diff --git a/docs/index.md b/docs/index.md
@@ -31,8 +31,8 @@ Fine-tune Hugging Face Models Instantly with Day-0 Support with NVIDIA NeMo Auto
 guides/overview.md
 guides/llm/finetune.md
 guides/llm/toolcalling.md
-guides/llm/mcore-pretraining.md
 guides/llm/pretraining.md
+guides/llm/nanogpt-pretraining.md
 guides/llm/sequence-classification.md
 guides/omni/gemma3-3n.md
 ```
diff --git a/examples/llm_pretrain/megatron_pretrain_gpt2.yaml b/examples/llm_pretrain/megatron_pretrain_gpt2.yaml
@@ -14,7 +14,7 @@
 
 
 # To run this recipe, please use the following command:
-# torchrun --nproc-per-node=8 recipes/llm_pretrain/pretrain.py --config recipes/llm_pretrain/megatron_pretrain.yaml
+# torchrun --nproc-per-node=8 examples/llm_pretrain/pretrain.py --config examples/llm_pretrain/megatron_pretrain_gpt2.yaml
 # Adjust --nproc-per-node to the number of GPUs available on your host machine.
 
 
diff --git a/examples/llm_pretrain/pretrain.py b/examples/llm_pretrain/pretrain.py
@@ -17,7 +17,7 @@
 from nemo_automodel.recipes.llm.train_ft import TrainFinetuneRecipeForNextTokenPrediction
 
 
-def main(default_config_path="examples/llm/nanogpt_pretrain.yaml"):
+def main(default_config_path="examples/llm_pretrain/nanogpt_pretrain.yaml"):
     """Entry-point for launching NanoGPT-style pre-training.
 
     The script follows the same invocation pattern as *examples/llm_finetune/finetune.py*:
diff --git a/tools/nanogpt_data_processor.py b/tools/nanogpt_data_processor.py
@@ -16,7 +16,7 @@
 FineWeb dataset preprocessing script
 
 This tool downloads a dataset from the Hugging Face Hub (default: FineWeb),
-tokenizes the data (default: tiktoken + gpt2), and writes memory-mapped binary shards compatible
+tokenizes the data (default: GPT-2 via transformers.AutoTokenizer), and writes memory-mapped binary shards compatible
 with `BinTokenDataset` for efficient streaming pre-training.
 
 Usage (typical):
