You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Jun 3, 2025. It is now read-only.
Copy file name to clipboardExpand all lines: src/content/get-started/sparsify-a-model.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,7 +11,7 @@ index: 4000
11
11
SparseML enables you to create a sparse model from scratch. The library contains state-of-the-art sparsification algorithms, including pruning, distillation, and quantization techniques.
12
12
13
13
These algorithms are built on top of sparsification recipes, enabling easy integration into custom ML training pipelines to sparsify most neural networks.
14
-
Additionally, SparseML integrates with popular ML repositories like HuggingFace Transformers and Ultralytics YOLO. With these integrations, creating a recipe and passing it to a CLI is all you need to sparsify a model.
14
+
Additionally, SparseML integrates with popular ML repositories like Hugging Face Transformers and Ultralytics YOLO. With these integrations, creating a recipe and passing it to a CLI is all you need to sparsify a model.
15
15
16
16
Aside from sparsification algorithms, SparseML contains generic export pathways for performant deployments.
17
17
These export pathways ensure the model saves in the correct format and rewrites the inference graphs for performance, such as quantized operator folding.
This page walks through an example of creating a sparsification recipe to prune a dense model from scratch and applying a recipe to a supported integration.
12
12
13
-
SparseML has pre-made integrations with many popular model repositories, such as with HuggingFace Transformers and Ultralytics YOLOv5.
13
+
SparseML has pre-made integrations with many popular model repositories, such as with Hugging Face Transformers and Ultralytics YOLOv5.
14
14
For these integrations, a sparsification recipe is all you need, and you can apply state-of-the-art sparsification algorithms, including
15
15
pruning, distillation, and quantization, with a single command line call.
16
16
@@ -31,17 +31,17 @@ Important hyperparameters that need to be set are the following:
31
31
- The layers to prune and their target sparsity levels
32
32
- The number of epochs for pruning
33
33
- The frequency of pruning
34
-
- The length of time to finetune after pruning
35
-
- The the learning rates to (LR) for pruning and finetuning
34
+
- The length of time to fine-tune after pruning
35
+
- The learning rates to (LR) for pruning and fine-tuning
36
36
37
37
The proper hyperparameter values will differ for different model architectures, training schemes, and domains, but there is some general intuition for safe starting values.
38
38
The following are reasonably default values to start with:
39
39
- The final sparsity is set to 80% sparsity applied globally across all layers.
40
40
- The running frequency is set to pruning once per epoch (up to a few times per epoch for shorter schedules).
41
41
- The number of pruning epochs is set to 1/3 the original training epochs.
42
-
- The number of finetuning epochs is set to 1/4 the original epochs.
42
+
- The number of fine-tuning epochs is set to 1/4 the original epochs.
43
43
- The pruning LR is set to the midrange from the model's training start and final LRs.
44
-
- The finetuning LRs cycle from the pruning LR to the final LR is used for training.
44
+
- The fine-tuning LRs cycle from the pruning LR to the final LR is used for training.
45
45
46
46
SparseML conveniently encodes these hyperparameters into a YAML-based **Recipe** file. The rest of the system parses the arguments in the YAML file to set the parameters of the algorithm.
47
47
@@ -76,16 +76,16 @@ In this recipe:
76
76
- `GlobalMagnitudePruningModifier` applies gradual magnitude pruning globally across all the prunable parameters/weights in a model.
77
77
- `GlobalMagnitudePruningModifier`starts at 5% sparsity at epoch 0 and gradually ramps up to 80% sparsity at epoch 30, pruning at the start of each epoch.
78
78
- `SetLearningRateModifier`sets the pruning LR to 0.05 (midpoint between the original 0.1 and 0.001 training LRs).
79
-
- `LearningRateFunctionModifier`cycles the finetuning LR from the pruning LR to 0.001 with a cosine curve (0.001 was the final original training LR).
80
-
- `EpochRangeModifier`expands the training time to continue finetuning for an additional 20 epochs after pruning has ended.
81
-
- 30 pruning epochs and 20 finetuning epochs were chosen based on a 90 epoch training schedule -- be sure to adjust based on the number of epochs used for the initial training for your use case.
79
+
- `LearningRateFunctionModifier`cycles the fine-tuning LR from the pruning LR to 0.001 with a cosine curve (0.001 was the final original training LR).
80
+
- `EpochRangeModifier`expands the training time to continue fine-tuning for an additional 20 epochs after pruning has ended.
81
+
- 30 pruning epochs and 20 fine-tuning epochs were chosen based on a 90 epoch training schedule -- be sure to adjust based on the number of epochs used for the initial training for your use case.
82
82
83
83
## Quantization and Quantization Recipes
84
84
85
85
A quantization recipe systematically reduces the precision for weights and activations within a neural network, generally from `FP32` to `INT8`. Running a quantized
86
86
model increases speed and reduces memory consumption while sacrificing very little in terms of accuracy.
87
87
88
-
Quantizationaware training (QAT) is the standard algorithm. With QAT, fake quantization operators are injected into the graph before quantizable nodes for activations, and weights are wrapped with fake quantization operators.
88
+
Quantization-aware training (QAT) is the standard algorithm. With QAT, fake quantization operators are injected into the graph before quantizable nodes for activations, and weights are wrapped with fake quantization operators.
89
89
The fake quantization operators interpolate the weights and activations down to INT8 on the forward pass but enable a full update of the weights at FP32 on the backward pass.
90
90
The updates to the weights at FP32 throughout the training process allow the model to adapt to the loss of information from quantization on the forward pass.
91
91
QAT generally guarantees better recovery for a given model compared with post-training quantization (PTQ), where training is not used.
@@ -124,7 +124,7 @@ Note the `model` is used here as a general placeholder; to determine the name of
124
124
## Pruning plus Quantization Recipe
125
125
126
126
To create a pruning and quantization recipe, the pruning and quantization recipes are merged from the previous sections.
127
-
Quantization is added after pruning and finetuning are complete such that the training cycles end with it.
127
+
Quantization is added after pruning and fine-tuning are complete such that the training cycles end with it.
128
128
This prevents stability issues from lacking precision when pruning and utilizing larger LRs.
129
129
130
130
Combining the two previous recipes creates the following new recipe.yaml file:
# Question Answering with HuggingFace Transformers and SparseML
9
+
# Question Answering with Hugging Face Transformers and SparseML
10
10
11
11
This page explains how to create and deploy a sparse Transformer for Question Answering.
12
12
@@ -19,7 +19,7 @@ This integration enables you to create a sparse model in two ways:
19
19
-**Sparse Transfer Learning** - fine-tune a sparse model (or use one of our [sparse pre-trained models](https://sparsezoo.neuralmagic.com/?domain=nlp&sub_domain=question_answering)) on your own private dataset.
20
20
21
21
Each option is useful in different situations:
22
-
-**Sparsification from Scratch** enables you to create a sparse version of any model (even those not in the SparseZoo), but requires hand-tuning the hyperparameters of the Sparsification algorithm.
22
+
-**Sparsification from Scratch** enables you to create a sparse version of any model (even those not in the SparseZoo), but requires hand-tuning the hyperparameters of the sparsification algorithm.
23
23
-**Sparse Transfer Learning** is the easiest path to creating a sparse model trained on your data. Simply pull a pre-sparsified model and transfer learning recipe from the SparseZoo and fine-tune on your data with a single command.
The SparseML train script is a wrapper around a [HuggingFace script](https://huggingface.co/docs/transformers/run_scripts),
57
-
and usage for most arguments follows the HuggingFace. The most important arguments for SparseML are:
56
+
The SparseML train script is a wrapper around a [Hugging Face script](https://huggingface.co/docs/transformers/run_scripts),
57
+
and usage for most arguments follows the Hugging Face. The most important arguments for SparseML are:
58
58
59
59
-`--model_name_or_path` indicates which model to start the pruning process from. It can be a SparseZoo stub, HF model identifier, or a path to a local model.
60
60
-`--recipe` points to recipe file containing the sparsification hyperparamters. It can be a SparseZoo stub or a local file. For more on creating a recipe see [here](/user-guide/recipes/creating).
61
61
-`--dataset_name` indicates that we should fine tune on the SQuAD dataset.
62
62
63
-
To utilize a custom dataset, use the `--train_file` and `--validation_file` arguments. To use a dataset from the HuggingFace hub, use `--dataset_name`.
64
-
See the [HF Docs](https://huggingface.co/docs/transformers/run_scripts#run-a-script) for more details.
63
+
To utilize a custom dataset, use the `--train_file` and `--validation_file` arguments. To use a dataset from the Hugging Face hub, use `--dataset_name`.
64
+
See the [Hugging Face Docs](https://huggingface.co/docs/transformers/run_scripts#run-a-script) for more details.
65
65
66
66
Run the following to see the full list of options:
# Text Classification with HuggingFace Transformers and SparseML
9
+
# Text Classification with Hugging Face Transformers and SparseML
10
10
11
11
This page explains how to create and deploy a sparse Transformer for Text Classification.
12
12
@@ -19,7 +19,7 @@ This integration enables you to create a sparse model in two ways:
19
19
-**Sparse Transfer Learning** - fine-tune a sparse model (or use one of our [sparse pre-trained models](https://sparsezoo.neuralmagic.com/?domain=nlp&sub_domain=text_classification)) on your own private dataset.
20
20
21
21
Each option is useful in different situations:
22
-
-**Sparsification from Scratch** enables you to create a sparse version of any model (even those not in the SparseZoo), but requires hand-tuning the hyperparameters of the Sparsification algorithm.
22
+
-**Sparsification from Scratch** enables you to create a sparse version of any model (even those not in the SparseZoo), but requires hand-tuning the hyperparameters of the sparsification algorithm.
23
23
-**Sparse Transfer Learning** is the easiest path to creating a sparse model trained on your data. Simply pull a pre-sparsified model and transfer learning recipe from the SparseZoo and fine-tune on your data with a single command.
The SparseML train script is a wrapper around a [HuggingFace script](https://huggingface.co/docs/transformers/run_scripts), and
56
-
usage for most arguments follows the HuggingFace. The most important arguments for SparseML are:
57
-
-`model_name_or_path`: specifies starting model. It can be a SparseZoo stub, HF model identifier, or a local directory
55
+
The SparseML train script is a wrapper around a [Hugging Face script](https://huggingface.co/docs/transformers/run_scripts), and
56
+
usage for most arguments follows the Hugging Face. The most important arguments for SparseML are:
57
+
-`model_name_or_path`: specifies starting model. It can be a SparseZoo stub, Hugging Face model identifier, or a local directory
58
58
with `model.pt`, `tokenizer.json` and `config.json`
59
59
-`recipe`: recipe containing the training hyperparamters (SparseZoo stub or a local file)
60
60
-`task_name`: specifies the sentiment analysis task for the MNLI dataset
61
61
62
-
To utilize a custom dataset, use the `--train_file` and `--validation_file` arguments. To use a dataset from the HuggingFace hub, use `--dataset_name`.
63
-
See the [HF Docs](https://huggingface.co/docs/transformers/run_scripts#run-a-script) for more details.
62
+
To utilize a custom dataset, use the `--train_file` and `--validation_file` arguments. To use a dataset from the Hugging Face hub, use `--dataset_name`.
63
+
See the [Hugging Face Docs](https://huggingface.co/docs/transformers/run_scripts#run-a-script) for more details.
64
64
65
65
Run the following to see the full list of options:
The SparseML train script is a wrapper around a [HuggingFace script](https://huggingface.co/docs/transformers/run_scripts),
56
-
and usage for most arguments follows the HuggingFace. The most important arguments for SparseML are:
55
+
The SparseML train script is a wrapper around a [Hugging Face script](https://huggingface.co/docs/transformers/run_scripts),
56
+
and usage for most arguments follows the Hugging Face. The most important arguments for SparseML are:
57
57
58
-
-`--model_name_or_path` indicates which model to start the pruning process from. It can be a SparseZoo stub, HF model identifier, or a path to a local model.
58
+
-`--model_name_or_path` indicates which model to start the pruning process from. It can be a SparseZoo stub, Hugging Face model identifier, or a path to a local model.
59
59
-`--recipe` points to recipe file containing the sparsification hyperparamters. It can be a SparseZoo stub or a local file. For more on creating a recipe see [here](/user-guide/recipes/creating).
60
60
-`--dataset_name` indicates that we should fine tune on the CoNLL-2003 dataset.
61
61
62
-
To utilize a custom dataset, use the `--train_file` and `--validation_file` arguments. To use a dataset from the HuggingFace hub, use `--dataset_name`.
63
-
See the [HF Docs](https://huggingface.co/docs/transformers/run_scripts#run-a-script) for more details.
62
+
To utilize a custom dataset, use the `--train_file` and `--validation_file` arguments. To use a dataset from the Hugging Face hub, use `--dataset_name`.
63
+
See the [Hugging Face Docs](https://huggingface.co/docs/transformers/run_scripts#run-a-script) for more details.
64
64
65
65
Run the following to see the full list of options:
0 commit comments