You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
"**Description:** Overview of quantization in Keras (int8, float8, int4, GPTQ)."
15
+
]
16
+
},
17
+
{
18
+
"cell_type": "markdown",
19
+
"metadata": {
20
+
"colab_type": "text"
21
+
},
22
+
"source": [
19
23
"## Introduction\n",
20
24
"\n",
21
25
"Modern large models are often **memory- and bandwidth-bound**: most inference time is spent moving tensors between memory and compute units rather than doing math. Quantization reduces the number of bits used to represent the model's weights and (optionally) activations, which:\n",
@@ -31,12 +35,19 @@
31
35
"* Joint weight + activation PTQ in `int4`, `int8`, and `float8`.\n",
32
36
"* Weight-only PTQ via **GPTQ** (2/3/4/8-bit) to maximize compression with minimal accuracy impact, especially for large language models (LLMs).\n",
33
37
"\n",
34
-
"**Terminology**\n",
38
+
"### Terminology\n",
39
+
"\n",
35
40
"* *Scale / zero-point:* Quantization maps real values `x` to integers `q` using a scale (and optionally a zero-point). Symmetric schemes use only a scale.\n",
36
41
"* *Per-channel vs per-tensor:* A separate scale per output channel (e.g., per hidden unit) usually preserves accuracy better than a single scale for the whole tensor.\n",
37
-
"* *Calibration:* A short pass over sample data to estimate activation ranges (e.g., max absolute value).\n",
38
-
"\n",
39
-
"\n",
42
+
"* *Calibration:* A short pass over sample data to estimate activation ranges (e.g., max absolute value)."
43
+
]
44
+
},
45
+
{
46
+
"cell_type": "markdown",
47
+
"metadata": {
48
+
"colab_type": "text"
49
+
},
50
+
"source": [
40
51
"## Quantization Modes\n",
41
52
"\n",
42
53
"Keras currently focuses on the following numeric formats. Each mode can be applied selectively to layers or to the whole model via the same API.\n",
@@ -67,10 +78,18 @@
67
78
"\n",
68
79
"### Implementation notes\n",
69
80
"\n",
70
-
"* For `int4`, Keras packs signed 4-bit values (range = [-8, 7]) and stores per-channel scales such as `kernel_scale`. Dequantization happens on the fly, and matmuls use 8-bit (unpacked) kernels.\n",
71
-
"* Activation scaling for `int4` / `int8` / `float8` uses **AbsMax calibration** by default (range set by the maximum absolute value observed). Alternative calibration methods (e.g., percentile) may be added in future releases.\n",
72
-
"* Per-channel scaling is the default for weights where supported, because it materially improves accuracy at negligible overhead.\n",
73
-
"\n",
81
+
"* **Dynamic activation quantization**: In the `int4`, `int8` PTQ path, activation scales are computed on-the-fly at runtime (per tensor and per batch) using an AbsMax estimator. This avoids maintaining a separate, fixed set of activation scales from a calibration pass and adapts to varying input ranges.\n",
82
+
"* **4-bit packing**: For `int4`, Keras packs signed 4-bit values (range = [-8, 7]) and stores per-channel scales such as `kernel_scale`. Dequantization happens on the fly, and matmuls use 8-bit (unpacked) kernels.\n",
83
+
"* **Calibration Strategy**: Activation scaling for `int4` / `int8` / `float8` uses **AbsMax calibration** by default (range set by the maximum absolute value observed). Alternative calibration methods (e.g., percentile) may be added in future releases.\n",
84
+
"* Per-channel scaling is the default for weights where supported, because it materially improves accuracy at negligible overhead."
85
+
]
86
+
},
87
+
{
88
+
"cell_type": "markdown",
89
+
"metadata": {
90
+
"colab_type": "text"
91
+
},
92
+
"source": [
74
93
"## Quantizing Keras Models\n",
75
94
"\n",
76
95
"Quantization is applied explicitly after layers or models are built. The API is designed to be predictable: you call quantize, the graph is rewritten, the weights are replaced, and you can immediately run inference or save the model.\n",
"**What this does:** Quantizes the weights of the supported layers, and re-wires their forward paths to be compatible with the quantized kernels and quantization scales.\n",
"layer.quantize(\"int4\") # Or \"int8\", \"float8\", etc."
144
169
]
145
170
},
146
171
{
147
172
"cell_type": "markdown",
148
-
"id": "249deef4",
149
-
"metadata": {},
173
+
"metadata": {
174
+
"colab_type": "text"
175
+
},
150
176
"source": [
151
-
"**When to use layer-wise quantization**\n",
177
+
"### When to use layer-wise quantization\n",
152
178
"\n",
153
179
"* To keep numerically sensitive blocks (e.g., small residual paths, logits) at higher precision while quantizing large projection layers.\n",
154
180
"* To mix modes (e.g., attention projections in int4, feed-forward in int8) and measure trade-offs incrementally.\n",
155
-
"* Always validate on a small eval set after each step; mixing precisions across residual connections can shift distributions.\n",
156
-
"\n",
181
+
"* Always validate on a small eval set after each step; mixing precisions across residual connections can shift distributions."
182
+
]
183
+
},
184
+
{
185
+
"cell_type": "markdown",
186
+
"metadata": {
187
+
"colab_type": "text"
188
+
},
189
+
"source": [
157
190
"## Layer & model coverage\n",
158
191
"\n",
159
192
"Keras supports the following core layers in its quantization framework:\n",
@@ -165,26 +198,42 @@
165
198
"\n",
166
199
"Any composite layers that are built from the above (for example, `MultiHeadAttention`, `GroupedQueryAttention`, feed-forward blocks in Transformers) inherit quantization support by construction. This covers the majority of modern encoder-only and decoder-only Transformer architectures.\n",
167
200
"\n",
168
-
"Since all KerasHub models subclass `keras.Model`, they automatically support the `model.quantize(...)` API. In practice, this means you can take a popular LLM preset, call a single function to obtain an int8/int4/GPTQ-quantized variant, and then save or serve it—without changing your training code.\n",
201
+
"Since all KerasHub models subclass `keras.Model`, they automatically support the `model.quantize(...)` API. In practice, this means you can take a popular LLM preset, call a single function to obtain an int8/int4/GPTQ-quantized variant, and then save or serve it\u2014without changing your training code.\n",
169
202
"\n",
170
203
"## Practical guidance\n",
171
204
"\n",
172
205
"* For GPTQ, use a calibration set that matches your inference domain (a few hundred to a few thousand tokens is often enough to see strong retention).\n",
173
206
"* Measure both **VRAM** and **throughput/latency**: memory savings are immediate; speedups depend on the availability of fused low-precision kernels on your device."
Copy file name to clipboardExpand all lines: guides/md/quantization_overview.md
+28-17Lines changed: 28 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,10 +5,12 @@
5
5
**Last modified:** 2025/10/09<br>
6
6
**Description:** Overview of quantization in Keras (int8, float8, int4, GPTQ).
7
7
8
+
8
9
<imgclass="k-inline-icon"src="https://colab.research.google.com/img/colab_favicon.ico"/> [**View in Colab**](https://colab.research.google.com/github/keras-team/keras-io/blob/master/guides/ipynb/quantization_overview.ipynb) <spanclass="k-dot">•</span><imgclass="k-inline-icon"src="https://github.com/favicon.ico"/> [**GitHub source**](https://github.com/keras-team/keras-io/blob/master/guides/quantization_overview.py)
9
10
10
-
---
11
11
12
+
13
+
---
12
14
## Introduction
13
15
14
16
Modern large models are often **memory- and bandwidth-bound**: most inference time is spent moving tensors between memory and compute units rather than doing math. Quantization reduces the number of bits used to represent the model's weights and (optionally) activations, which:
@@ -24,14 +26,13 @@ At a high level, Keras supports:
24
26
* Joint weight + activation PTQ in `int4`, `int8`, and `float8`.
25
27
* Weight-only PTQ via **GPTQ** (2/3/4/8-bit) to maximize compression with minimal accuracy impact, especially for large language models (LLMs).
26
28
27
-
> **Terminology**
28
-
>
29
-
> **Scale / zero-point:* Quantization maps real values `x` to integers `q` using a scale (and optionally a zero-point). Symmetric schemes use only a scale.
30
-
> **Per-channel vs per-tensor:* A separate scale per output channel (e.g., per hidden unit) usually preserves accuracy better than a single scale for the whole tensor.
31
-
> **Calibration:* A short pass over sample data to estimate activation ranges (e.g., max absolute value).
29
+
### Terminology
32
30
33
-
---
31
+
**Scale / zero-point:* Quantization maps real values `x` to integers `q` using a scale (and optionally a zero-point). Symmetric schemes use only a scale.
32
+
**Per-channel vs per-tensor:* A separate scale per output channel (e.g., per hidden unit) usually preserves accuracy better than a single scale for the whole tensor.
33
+
**Calibration:* A short pass over sample data to estimate activation ranges (e.g., max absolute value).
34
34
35
+
---
35
36
## Quantization Modes
36
37
37
38
Keras currently focuses on the following numeric formats. Each mode can be applied selectively to layers or to the whole model via the same API.
@@ -62,12 +63,12 @@ Keras currently focuses on the following numeric formats. Each mode can be appli
62
63
63
64
### Implementation notes
64
65
65
-
* For `int4`, Keras packs signed 4-bit values (range = [−8, 7]) and stores per-channel scales such as `kernel_scale`. Dequantization happens on the fly, and matmuls use 8-bit (unpacked) kernels.
66
-
* Activation scaling for `int4` / `int8` / `float8` uses **AbsMax calibration** by default (range set by the maximum absolute value observed). Alternative calibration methods (e.g., percentile) may be added in future releases.
66
+
***Dynamic activation quantization**: In the `int4`, `int8` PTQ path, activation scales are computed on-the-fly at runtime (per tensor and per batch) using an AbsMax estimator. This avoids maintaining a separate, fixed set of activation scales from a calibration pass and adapts to varying input ranges.
67
+
***4-bit packing**: For `int4`, Keras packs signed 4-bit values (range = [-8, 7]) and stores per-channel scales such as `kernel_scale`. Dequantization happens on the fly, and matmuls use 8-bit (unpacked) kernels.
68
+
***Calibration Strategy**: Activation scaling for `int4` / `int8` / `float8` uses **AbsMax calibration** by default (range set by the maximum absolute value observed). Alternative calibration methods (e.g., percentile) may be added in future releases.
67
69
* Per-channel scaling is the default for weights where supported, because it materially improves accuracy at negligible overhead.
68
70
69
71
---
70
-
71
72
## Quantizing Keras Models
72
73
73
74
Quantization is applied explicitly after layers or models are built. The API is designed to be predictable: you call quantize, the graph is rewritten, the weights are replaced, and you can immediately run inference or save the model.
/Users/jyotindersingh/miniconda3/envs/keras-io-env-3.10/lib/python3.10/site-packages/keras/src/models/model.py:455: UserWarning: Layer InputLayer does not have a `quantize` method implemented.
114
+
warnings.warn(str(e))
115
+
```
116
+
</div>
117
+
106
118
**What this does:** Quantizes the weights of the supported layers, and re-wires their forward paths to be compatible with the quantized kernels and quantization scales.
107
119
108
120
**Note**: Throughput gains depend on backend/hardware kernels; in cases where kernels fall back to dequantized matmul, you still get memory savings but smaller speedups.
@@ -111,11 +123,12 @@ model.quantize("int8")
111
123
112
124
The Keras quantization framework allows you to quantize each layer separately, without having to quantize the entire model using the same unified API.
layer.quantize("int4") # Or "int8", "float8", etc.
@@ -128,7 +141,6 @@ layer.quantize("int4") # Or "int8", "float8", etc.
128
141
* Always validate on a small eval set after each step; mixing precisions across residual connections can shift distributions.
129
142
130
143
---
131
-
132
144
## Layer & model coverage
133
145
134
146
Keras supports the following core layers in its quantization framework:
@@ -143,7 +155,6 @@ Any composite layers that are built from the above (for example, `MultiHeadAtten
143
155
Since all KerasHub models subclass `keras.Model`, they automatically support the `model.quantize(...)` API. In practice, this means you can take a popular LLM preset, call a single function to obtain an int8/int4/GPTQ-quantized variant, and then save or serve it—without changing your training code.
144
156
145
157
---
146
-
147
158
## Practical guidance
148
159
149
160
* For GPTQ, use a calibration set that matches your inference domain (a few hundred to a few thousand tokens is often enough to see strong retention).
0 commit comments