Why isn't HAWQ using the full search space I provide in the config file?

[i3abghany]

Config file:

{
   {
    "model": "resnet18_cifar10",
    "dataset": "cifar10",
    "pretrained": false,
    "input_info": {
        "sample_size": [
            2,
            3,
            32,
            32
        ]
    },
    "num_classes": 10,
    "batch_size": 150,
    "workers": 6,
    "multiprocessing_distributed": true,
    "epochs": 4,
    "target_device": "TRIAL",
    "compression": {
        "algorithm": "quantization",
        "initializer": {
            "precision": {
                "type": "hawq",
                "bits": [4, 8, 12, 16, 20],
                "compression_ratio": 1.9
            }
        },
        "params": {
            "base_lr": 3.1e-4,
            "base_wd": 1e-5,
            "batch_multiplier": 1,
            "activations_quant_start_epoch": 3,
            "weights_quant_start_epoch": 12,
            "lr_poly_drop_start_epoch": 20,
            "lr_poly_drop_duration_epochs": 10,
            "disable_wd_start_epoch": 20
        }
    }
}

The bitwidths distribution in the output.log file looks like this after 4 epochs:

Statistics of the quantization algorithm:
+--------------------------------+--------+
|        Statistic's name        | Value  |
+================================+========+
| Ratio of enabled quantizations | 58.824 |
+--------------------------------+--------+

Statistics of the quantization share:
+----------------------------------+--------------------+
|         Statistic's name         |       Value        |
+==================================+====================+
| Symmetric WQs / All placed WQs   | 100.00 % (21 / 21) |
+----------------------------------+--------------------+
| Asymmetric WQs / All placed WQs  | 0.00 % (0 / 21)    |
+----------------------------------+--------------------+
| Signed WQs / All placed WQs      | 100.00 % (21 / 21) |
+----------------------------------+--------------------+
| Unsigned WQs / All placed WQs    | 0.00 % (0 / 21)    |
+----------------------------------+--------------------+
| Per-tensor WQs / All placed WQs  | 100.00 % (21 / 21) |
+----------------------------------+--------------------+
| Per-channel WQs / All placed WQs | 0.00 % (0 / 21)    |
+----------------------------------+--------------------+
| Placed WQs / Potential WQs       | 51.22 % (21 / 41)  |
+----------------------------------+--------------------+
| Symmetric AQs / All placed AQs   | 100.00 % (30 / 30) |
+----------------------------------+--------------------+
| Asymmetric AQs / All placed AQs  | 0.00 % (0 / 30)    |
+----------------------------------+--------------------+
| Signed AQs / All placed AQs      | 40.00 % (12 / 30)  |
+----------------------------------+--------------------+
| Unsigned AQs / All placed AQs    | 60.00 % (18 / 30)  |
+----------------------------------+--------------------+
| Per-tensor AQs / All placed AQs  | 100.00 % (30 / 30) |
+----------------------------------+--------------------+
| Per-channel AQs / All placed AQs | 0.00 % (0 / 30)    |
+----------------------------------+--------------------+

Statistics of the bitwidth distribution:
+--------------+---------------------+--------------------+--------------------+
| Num bits (N) | N-bits WQs / Placed |    N-bits AQs /    | N-bits Qs / Placed |
|              |         WQs         |     Placed AQs     |         Qs         |
+==============+=====================+====================+====================+
| 16           | 9.52 % (2 / 21)     | 3.33 % (1 / 30)    | 5.88 % (3 / 51)    |
+--------------+---------------------+--------------------+--------------------+
| 8            | 0.00 % (0 / 21)     | 23.33 % (7 / 30)   | 13.73 % (7 / 51)   |
+--------------+---------------------+--------------------+--------------------+
| 4            | 90.48 % (19 / 21)   | 73.33 % (22 / 30)  | 80.39 % (41 / 51)  |
+--------------+---------------------+--------------------+--------------------+

Clearly, it uses only 4-, 8-, and 16-bit values. Why isn't it using the full range of bit widths I am providing in the config file?

[ljaljushkin]

It's not using the full range of bit widths, because the compression ratio is too aggressive - 1.9.
Compression ratio 2 refers to int4 uniform quantization. There are not many configurations with such big values of bit widths.
Please try some value lower than 1.
Added some clarifications to the documentation. Hope it's more clear now: #1383

I recommend using a pretrained model in order to approximate Hessian traces more correctly and get a more reliable metric of sensitivity based on that.
dump_init_precision_data parameter is useful for debugging the choice of algorithm.
For instance, I've tried the following config for pre-trained resnet50 on cifar10 and got the following plot (hawq_dumps/Pareto_Frontier_compress_ratio.png) of all configurations and the chosen one.

 "compression": {
        "algorithm": "quantization",
        "initializer": {
            "batchnorm_adaptation": {
                "num_bn_adaptation_samples": 3500
            },
            "range": {
                "type": "mean_min_max",
                "num_init_samples": 1000
            },
            "precision": {
                "type": "hawq",
                "bits": [4, 8, 12, 16, 20],
                "iter_number": 300,
                "compression_ratio": 0.75,
                "eval_subset_ratio": 0.20,
                "dump_init_precision_data": true
            }
        },

[i3abghany]

Thank you so much for the help! do we have access to the pre-trained models in NNCF or must I get them somewhere else? And if I need to provide them, do you happen to know where to get the pre-trained ResNets for CIFAR-10?

[ljaljushkin]

For our experiments on cifar10, we used models from this repo