Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/source-fabric/advanced/model_parallel/tp_fsdp.rst
Original file line number Diff line number Diff line change
Expand Up @@ -276,7 +276,7 @@ Next steps

.. displayitem::
:header: Pipeline Parallelism
:description: Coming sooon
:description: Coming soon
:col_css: col-md-4
:height: 160
:tag: advanced
Expand Down
4 changes: 2 additions & 2 deletions docs/source-pytorch/advanced/compile.rst
Original file line number Diff line number Diff line change
Expand Up @@ -262,7 +262,7 @@ Avoid graph breaks
When ``torch.compile`` looks at the code in your model's ``forward()`` or ``*_step()`` method, it will try to compile as much of the code as possible.
If there are regions in the code that it doesn't understand, it will introduce a so-called "graph break" that essentially splits the code in optimized and unoptimized parts.
Graph breaks aren't a deal breaker, since the optimized parts should still run faster.
But if you want to get the most out of ``torch.compile``, you might want to invest rewriting the problematic section of the code that produce the breaks.
But if you want to get the most out of ``torch.compile``, you might want to invest rewriting the problematic section of the code that produces the breaks.

You can check whether your model produces graph breaks by calling ``torch.compile`` with ``fullgraph=True``:

Expand Down Expand Up @@ -332,7 +332,7 @@ Enabling CUDA Graphs often results in a significant speedup, but sometimes also

**Shape padding:** The specific shape/size of the tensors involved in the computation of your model (input, activations, weights, gradients, etc.) can have an impact on the performance.
With shape padding enabled, ``torch.compile`` can extend the tensors by padding to a size that gives a better memory alignment.
Naturally, the tradoff here is that it will consume a bit more memory.
Naturally, the tradeoff here is that it will consume a bit more memory.

.. code-block:: python

Expand Down
2 changes: 1 addition & 1 deletion docs/source-pytorch/advanced/model_parallel/tp_fsdp.rst
Original file line number Diff line number Diff line change
Expand Up @@ -282,7 +282,7 @@ Next steps

.. displayitem::
:header: Pipeline Parallelism
:description: Coming sooon
:description: Coming soon
:col_css: col-md-4
:height: 160
:tag: advanced
Expand Down
4 changes: 2 additions & 2 deletions docs/source-pytorch/advanced/post_training_quantization.rst
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ The "approach" parameter in PostTrainingQuantConfig is defined by the user to ma
Quantize the model
==================

The model can be qutized by Intel® Neural Compressor with:
The model can be quantized by Intel® Neural Compressor with:

.. code-block:: python

Expand All @@ -126,7 +126,7 @@ At last, the quantized model can be saved by:
Hands-on Examples
*****************

Based on the `given example code <https://lightning.ai/docs/pytorch/2.1.0/notebooks/lightning_examples/text-transformers.html>`_, we show how Intel Neural Compressor conduct model quantization on PyTorch Lightning. We first define the basic config of the quantization process.
Based on the `given example code <https://lightning.ai/docs/pytorch/2.1.0/notebooks/lightning_examples/text-transformers.html>`_, we show how Intel Neural Compressor conducts model quantization on PyTorch Lightning. We first define the basic config of the quantization process.

.. code-block:: python

Expand Down
4 changes: 2 additions & 2 deletions docs/source-pytorch/advanced/pruning_quantization.rst
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ You can also perform iterative pruning, apply the `lottery ticket hypothesis <ht
.. code-block:: python

def compute_amount(epoch):
# the sum of all returned values need to be smaller than 1
# the sum of all returned values needs to be smaller than 1
if epoch == 10:
return 0.5

Expand All @@ -43,7 +43,7 @@ You can also perform iterative pruning, apply the `lottery ticket hypothesis <ht
return 0.01


# the amount can be also be a callable
# the amount can also be a callable
trainer = Trainer(callbacks=[ModelPruning("l1_unstructured", amount=compute_amount)])


Expand Down
2 changes: 1 addition & 1 deletion docs/source-pytorch/advanced/training_tricks.rst
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ If the Trainer's ``gradient_clip_algorithm`` is set to ``'value'`` (``'norm'`` b
# clip gradients' maximum magnitude to <=0.5
trainer = Trainer(gradient_clip_val=0.5, gradient_clip_algorithm="value")

Read more about :ref:`Configuring Gradient Clipping <configure_gradient_clipping>` for advanced use-cases.
Read more about :ref:`Configuring Gradient Clipping <configure_gradient_clipping>` for advanced use cases.

----------

Expand Down
2 changes: 1 addition & 1 deletion src/lightning/fabric/connector.py
Original file line number Diff line number Diff line change
Expand Up @@ -239,7 +239,7 @@ def _check_config_and_set_final_flags(
else:
raise TypeError(
f"Found invalid type for plugin {plugin}. Expected one of: Precision, "
"CheckpointIO, ClusterEnviroment."
"CheckpointIO, ClusterEnvironment."
)

duplicated_plugin_key = [k for k, v in plugins_flags_types.items() if v > 1]
Expand Down
2 changes: 1 addition & 1 deletion src/lightning/fabric/plugins/precision/bitsandbytes.py
Original file line number Diff line number Diff line change
Expand Up @@ -403,7 +403,7 @@ class _NF4DQLinear(_Linear4bit):
def __init__(self, *args: Any, **kwargs: Any) -> None:
super().__init__(*args, quant_type="nf4", compress_statistics=True, **kwargs)

# these classes are defined programatically like this to avoid importing bitsandbytes in environments that have
# these classes are defined programmatically like this to avoid importing bitsandbytes in environments that have
# it available but will not use it
classes = {
"_Linear8bitLt": _Linear8bitLt,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -248,7 +248,7 @@ def _check_config_and_set_final_flags(
else:
raise MisconfigurationException(
f"Found invalid type for plugin {plugin}. Expected one of: Precision, "
"CheckpointIO, ClusterEnviroment, or LayerSync."
"CheckpointIO, ClusterEnvironment, or LayerSync."
)

duplicated_plugin_key = [k for k, v in plugins_flags_types.items() if v > 1]
Expand Down
5 changes: 3 additions & 2 deletions tests/tests_fabric/utilities/test_data.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,9 @@ def test_has_len():
def test_replace_dunder_methods_multiple_loaders_without_init():
"""In case of a class, that inherits from a class that we are patching, but doesn't define its own `__init__`
method (the one we are wrapping), it can happen, that `hasattr(cls, "__old__init__")` is True because of parent
class, but it is impossible to delete, because that method is owned by parent class. Furthermore, the error occured
only sometimes because it depends on the order in which we are iterating over a set of classes we are patching.
class, but it is impossible to delete, because that method is owned by parent class. Furthermore, the error
occurred only sometimes because it depends on the order in which we are iterating over a set of classes we are
patching.

This test simulates the behavior by generating sufficient number of dummy classes, which do not define `__init__`
and are children of `DataLoader`. We are testing that a) context manager `_replace_dunder_method` exits cleanly, and
Expand Down
Loading