How to make AutoQ produce model of bit sizes other than (2, 4, and 8)?

[i3abghany]

Hello,

I am having a trouble making AutoQ produce a model with bit widths other than 2, 4, and 8. I slightly modify the nncf/examples/torch/classification/configs/mixed_precision/resnet50_imagenet_mixed_int_autoq_staged.json to be as follows:

{
    "model": "resnet18_cifar10",
    "dataset": "cifar10"
    "pretrained": false,
    "input_info": {
        "sample_size": [
            2,
            3,
            32,
            32
        ]
    },
    "num_classes": 10,
    "batch_size": 150,
    "epochs": 8,
    "workers": 6,
    "multiprocessing_distributed": true,
    "target_device": "VPU",
    "compression": {
        "algorithm": "quantization",
        "initializer": {
            "batchnorm_adaptation": {
                "num_bn_adaptation_samples": 3500
            },
            "range":
            {
                "type": "mean_min_max",
                "num_init_samples": 1000
            },
            "precision": {
                "type": "autoq",
                "bits": [2, 4, 8, 12, 16, 20, 24, 28, 32],
                "iter_number": 10,
                "compression_ratio": 0.9,
                "eval_subset_ratio": 0.20,
                "dump_init_precision_data": true
            }
        },
        "params": {
            "base_lr": 2e-4,
            "base_wd": 1e-5,
            "batch_multiplier": 1,
            "activations_quant_start_epoch": 0,
            "weights_quant_start_epoch": 0,
            "lr_poly_drop_start_epoch": 0,
            "lr_poly_drop_duration_epochs": 10,
        }
    }

The differences from the example file are:

• I use resnet18 using the CIFAR10 dataset.
• pretrained=false
• compression.precision.num_iters=10
• compression.precision.bits=[2, 4, 8, 12, 16, 20, 24, 28, 32]

I use a small number of episodes just to have a quick run.

I suspect that target_device having the value of VPU makes it only produce weights and activations using specific sizes. I am not interested in a specific hardware architecture.

When I totally remove the target_device property from the JSON file, I get an error saying that Automatic Precision Initialization is only supported for target devices VPU and NONE (as the default is set to CPU).

I am just interested in producing a model that's maybe 90% the size of a model that uses only 32-bit floats with no specific hardware requirements in mind.

[ljaljushkin]

Hello, @i3abghany!

There's a TRIAL device for that purpose. But I had to note that AutoQ wasn't tested for bitwidth more than 8.
I expect some difficulties with convergence of the algorithm. Some hyperparameter tuning might be required.
@vuiseng9 could you please comment on that?

[i3abghany]

Thank you for the reply. I also tried the TRIAL device and the bitwidth distribution logged only consists of N=2, 4, and 8. Please find the following config and output.log files:

They were generated by the run:

config.json
output.log

[i3abghany]

Also, does HAWQ have a stable implementation for bitwidths that are more than 8?

[ljaljushkin]

Thank you for the reply. I also tried the TRIAL device and the bitwidth distribution logged only consists of N=2, 4, and 8. Please find the following config and output.log files:

They were generated by the run:

config.json output.log

I've reproduced the same behavior on my side with AutoQ. Seems like there's some bug for set of bitwidth different from {2,4,8}. Created an issue for that: #1385

[ljaljushkin]

Also, does HAWQ have a stable implementation for bitwidths that are more than 8?

It wasn't tested in such setup, but according to the neighbor discussion thread it should work: #1380