Merge branch 'main' into 2.8-RC-Tutorial-Test

Author: svekars

intermediate_source/ddp_series_minGPT.rst

@@ -26,10 +26,11 @@ Authors: `Suraj Subramanian <https://github.com/subramen>`__
    .. grid-item-card:: :octicon:`list-unordered;1em;` Prerequisites
       :class-card: card-prerequisites
 
-      - Familiarity with `multi-GPU training <../beginner/ddp_series_multigpu.html>`__ and `torchrun <../beginner/ddp_series_fault_tolerance.html>`__ 
-      - [Optional] Familiarity with `multinode training <ddp_series_multinode.html>`__
-      - 2 or more TCP-reachable GPU machines (this tutorial uses AWS p3.2xlarge instances)
       - PyTorch `installed <https://pytorch.org/get-started/locally/>`__ with CUDA on all machines
+      - Familiarity with `multi-GPU training <../beginner/ddp_series_multigpu.html>`__ and `torchrun <../beginner/ddp_series_fault_tolerance.html>`__
+      - [Optional] Familiarity with `multinode training <ddp_series_multinode.html>`__
+      - 2 or more TCP-reachable GPU machines for multi-node training (this tutorial uses AWS p3.2xlarge instances)
+
 
 Follow along with the video below or on `youtube <https://www.youtube.com/watch/XFsFDGKZHh4>`__.
 
@@ -63,25 +64,23 @@ from any node that has access to the cloud bucket.
 
 Using Mixed Precision
 ~~~~~~~~~~~~~~~~~~~~~~~~
-To speed things up, you might be able to use `Mixed Precision <https://pytorch.org/docs/stable/amp.html>`__ to train your models. 
-In Mixed Precision, some parts of the training process are carried out in reduced precision, while other steps 
-that are more sensitive to precision drops are maintained in FP32 precision. 
+To speed things up, you might be able to use `Mixed Precision <https://pytorch.org/docs/stable/amp.html>`__ to train your models.
+In Mixed Precision, some parts of the training process are carried out in reduced precision, while other steps
+that are more sensitive to precision drops are maintained in FP32 precision.
 
 
 When is DDP not enough?
 ~~~~~~~~~~~~~~~~~~~~~~~~
 A typical training run's memory footprint consists of model weights, activations, gradients, the input batch, and the optimizer state.
-Since DDP replicates the model on each GPU, it only works when GPUs have sufficient capacity to accomodate the full footprint. 
+Since DDP replicates the model on each GPU, it only works when GPUs have sufficient capacity to accomodate the full footprint.
 When models grow larger, more aggressive techniques might be useful:
 
--  `activation checkpointing <https://pytorch.org/docs/stable/checkpoint.html>`__: Instead of saving intermediate activations during the forward pass, the activations are recomputed during the backward pass. In this approach, we run more compute but save on memory footprint.
--  `Fully-Sharded Data Parallel <https://pytorch.org/docs/stable/fsdp.html>`__: Here the model is not replicated but "sharded" across all the GPUs, and computation is overlapped with communication in the forward and backward passes. Read our `blog <https://medium.com/pytorch/training-a-1-trillion-parameter-model-with-pytorch-fully-sharded-data-parallel-on-aws-3ac13aa96cff>`__ to learn how we trained a 1 Trillion parameter model with FSDP.
-
+-  `Activation checkpointing <https://pytorch.org/docs/stable/checkpoint.html>`__: Instead of saving intermediate activations during the forward pass, the activations are recomputed during the backward pass. In this approach, we run more compute but save on memory footprint.
+-  `Fully-Sharded Data Parallel <https://docs.pytorch.org/docs/stable/distributed.fsdp.fully_shard.html>`__: Here the model is not replicated but "sharded" across all the GPUs, and computation is overlapped with communication in the forward and backward passes. Read our `blog <https://medium.com/pytorch/training-a-1-trillion-parameter-model-with-pytorch-fully-sharded-data-parallel-on-aws-3ac13aa96cff>`__ to learn how we trained a 1 Trillion parameter model with FSDP.
 
 Further Reading
 ---------------
 -  `Multi-Node training with DDP <ddp_series_multinode.html>`__ (previous tutorial in this series)
 -  `Mixed Precision training <https://pytorch.org/docs/stable/amp.html>`__
--  `Fully-Sharded Data Parallel <https://pytorch.org/docs/stable/fsdp.html>`__
+-  `Fully-Sharded Data Parallel tutorial <https://docs.pytorch.org/tutorials/intermediate/FSDP_tutorial.html>`__
 -  `Training a 1T parameter model with FSDP <https://medium.com/pytorch/training-a-1-trillion-parameter-model-with-pytorch-fully-sharded-data-parallel-on-aws-3ac13aa96cff>`__
--  `FSDP Video Tutorial Series <https://www.youtube.com/playlist?list=PL_lsbAsL_o2BT6aerEKgIoufVD_fodnuT>`__ 

recipes_source/recipes/profiler_recipe.py

@@ -5,31 +5,29 @@
 """
 
 ######################################################################
-"""
-This recipe explains how to use PyTorch profiler and measure the time and
-memory consumption of the model's operators.
-
-Introduction
-------------
-PyTorch includes a simple profiler API that is useful when user needs
-to determine the most expensive operators in the model.
-
-In this recipe, we will use a simple Resnet model to demonstrate how to
-use profiler to analyze model performance.
-
-Prerequisites
----------------
-- ``torch >= 1.9``
-
-Setup
------
-To install ``torch`` and ``torchvision`` use the following command:
-
-.. code-block:: sh
-
-   pip install torch torchvision
-
-"""
+# This recipe explains how to use PyTorch profiler and measure the time and
+# memory consumption of the model's operators.
+#
+# Introduction
+# ------------
+# PyTorch includes a simple profiler API that is useful when the user needs
+# to determine the most expensive operators in the model.
+#
+# In this recipe, we will use a simple Resnet model to demonstrate how to
+# use the profiler to analyze model performance.
+#
+# Prerequisites
+# ---------------
+# - ``torch >= 2.3.0``
+#
+# Setup
+# -----
+# To install ``torch`` and ``torchvision`` use the following command:
+#
+# .. code-block:: sh
+#
+#    pip install torch torchvision
+#
 
 ######################################################################
 # Steps