Quantization Aware Training won't work

[Hikaru-13915]

Now, I'm trying to quantize a model using callbacks, Quantization Aware Training.
But it won't work for testing.
Please somebody help me.
The cord bellow.
autoencoder = LitAutoEncoder(args)
custumised_callback=QuantizationAwareTraining(
observer_type="histogram",
modules_to_fuse=[(f"model.conv{i}",f"model.relu{i}") for i in range(1,3,1)])
trainer = pl.Trainer(gpus=1, callbacks=custumised_callback)
trainer.fit(autoencoder)
torch.save(autoencoder.state_dict(), "qat_lightning.pt")
test_loader=iter(DataLoader(MNIST(os.getcwd(), download=True, train=False, transform=transforms.ToTensor()), batch_size=args.batch_size)).next()
autoencoder(autoencoder.quant(test_loader[0]))

Though I had obtained a quantized model successfully, got an error at autoencoder(autoencoder.quant(test_loader[0]))
The error tells me

File "c:\Users\quant_with_only_lightning.py", line 209, in main autoencoder(autoencoder.quant(test_loader[0]))
File "C:\Users\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\lib\site-packages\pytorch_lightning\callbacks\quantization.py", line 76, in wrapper
data = model.quant(data)
File "C:\Users\lib\site-packages\torch\nn\modules\module.py", line 1051, in call_impl
return forward_call(*input, **kwargs)
File "C:\Users\lib\site-packages\torch\nn\quantized\modules_init.py", line 52, in forward
return torch.quantize_per_tensor(X, float(self.scale),
NotImplementedError: Could not run 'aten::quantize_per_tensor' with arguments from the 'QuantizedCPU' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::quantize_per_tensor' is only available for
these backends: [CPU, CUDA, BackendSelect, Named, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, UNKNOWN_TENSOR_TYPE_ID, AutogradMLC, AutogradHPU, AutogradNestedTensor, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, Autocast, Batched, VmapMode].
CPU: registered at aten\src\ATen\RegisterCPU.cpp:16286 [kernel]
CUDA: registered at aten\src\ATen\RegisterCUDA.cpp:20674 [kernel]
BackendSelect: fallthrough registered at ..\aten\src\ATen\core\BackendSelectFallbackKernel.cpp:3 [backend fallback]
Named: registered at ..\aten\src\ATen\core\NamedRegistrations.cpp:7 [backend fallback]
ADInplaceOrView: fallthrough registered at ..\aten\src\ATen\core\VariableFallbackKernel.cpp:60 [backend fallback]
AutogradOther: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:9848 [autograd kernel]
AutogradCPU: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:9848 [autograd kernel]
AutogradCUDA: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:9848 [autograd kernel]
AutogradXLA: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:9848 [autograd kernel]
UNKNOWN_TENSOR_TYPE_ID: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:9848 [autograd kernel]
AutogradMLC: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:9848 [autograd kernel]
AutogradHPU: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:9848 [autograd kernel]
l]
AutogradPrivateUse1: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:9848 [autograd kernel]
AutogradPrivateUse2: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:9848 [autograd kernel]
AutogradPrivateUse3: registered at ..\torch\csrc\autograd\generated\VariableType_0.cpp:9848 [autograd kernel]
Tracer: registered at ..\torch\csrc\autograd\generated\TraceType_0.cpp:9750 [kernel]
Autocast: fallthrough registered at ..\aten\src\ATen\autocast_mode.cpp:255 [backend fallback]
Batched: registered at ..\aten\src\ATen\BatchingRegistrations.cpp:1019 [backend fallback]
VmapMode: fallthrough registered at ..\aten\src\ATen\VmapModeRegistrations.cpp:33 [backend fallback]

and trainer.test() won't work as well

I just followed the example only to find it won't work at all
How do I quantize and test by a quantized model correctly?

[tchaton]

Dear @Hikaru-13915,

Mind sharing your PyTorch version. It seems PyTorch doesn't support some operation to be quantized.

[Hikaru-13915]

Thank you man!
I've never heard of it.
but it deserves a nod
Here is a pip list of my environment.

pip list
Package Version Location

---

absl-py 1.0.0
aiohttp 3.8.1
aiosignal 1.2.0
alembic 1.4.1
async-timeout 4.0.1
attrs 21.2.0
boto3 1.20.13
botocore 1.23.13
cachetools 4.2.4
certifi 2021.10.8
charset-normalizer 2.0.8
click 8.0.3
cloudpickle 2.0.0
colorama 0.4.4
cycler 0.10.0
databricks-cli 0.16.2
docker 5.0.3
entrypoints 0.3
Flask 2.0.2
frozenlist 1.2.0
fsspec 2021.11.0
future 0.18.2
gitdb 4.0.9
GitPython 3.1.24
google-auth 2.3.3
google-auth-oauthlib 0.4.6
greenlet 1.1.2
grpcio 1.42.0
idna 3.3
importlib-metadata 4.8.2
itsdangerous 2.0.1
Jinja2 3.0.3
jmespath 0.10.0
kiwisolver 1.3.2
Mako 1.1.6
Markdown 3.3.6
MarkupSafe 2.0.1
matplotlib 3.4.3
mlflow 1.21.0
multidict 5.2.0
numpy 1.21.2
oauthlib 3.1.1
onnx 1.10.2
opencv-python 4.5.3.56
packaging 21.3
pandas 1.3.3
Pillow 8.3.2
pip 20.2.4
prometheus-client 0.12.0
prometheus-flask-exporter 0.18.6
protobuf 3.19.1
pyasn1 0.4.8
pyasn1-modules 0.2.8
pyDeprecate 0.3.1
pyparsing 2.4.7
python-dateutil 2.8.2
python-editor 1.0.4
pytorch-lightning 1.5.3
pytz 2021.1
pywin32 227
PyYAML 6.0
querystring-parser 1.2.4
requests 2.26.0
requests-oauthlib 1.3.0
rsa 4.8
s3transfer 0.5.0
setuptools 57.4.0
six 1.16.0
smmap 5.0.0
SQLAlchemy 1.4.27
sqlparse 0.4.2
tabulate 0.8.9
tensorboard 2.7.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.0
torch 1.9.1+cu111
torchmetrics 0.6.0
torchvision 0.10.1
tqdm 4.62.3
typing-extensions 3.10.0.2
urllib3 1.26.7
waitress 2.0.0
websocket-client 1.2.1
Werkzeug 2.0.2
wheel 0.37.0
yarl 1.7.2
zipp 3.6.0

If you have any ideas about ways out to solve this matter, let me hear about them