Dl/ov/tiny gpt2 example callbacks by daniil-lyakhov · Pull Request #20 · daniil-lyakhov/nncf

daniil-lyakhov · 2023-05-31T08:43:31Z

Changes

New custom forward api is introduced
Adapters for tensor collectors are introduced
tiny_gpt2 and mozilla_deepspeech examples are introduced

TensorReducerSequence Reducer adapter inside reducer TensorCollectorAdapter

daniil-lyakhov · 2023-05-31T08:44:32Z

examples/post_training_quantization/openvino/tiny_gpt2/main.py

@AlexKoff88 - main file

daniil-lyakhov · 2023-05-31T08:45:25Z

nncf/openvino/statistics/aggregator.py

+    @staticmethod
+    def _get_callback(model, sequence_container):
+        original_model_outputs_names = {op.node.friendly_name for op in model.outputs}
+
+        def complition_callback(outputs):
+            for op, value in outputs.items():
+                if op.node.friendly_name in original_model_outputs_names:
+                    continue
+                if not isinstance(value, np.ndarray):
+                    value = value.data
+                sequence_container[op.node.friendly_name].append(OVNNCFTensor(value))
+
+        return complition_callback


@AlexKoff88 - callback creation for OpenVino

daniil-lyakhov · 2023-05-31T08:46:15Z

nncf/common/tensor_statistics/aggregator.py

@AlexKoff88 - custom inference in use here

daniil-lyakhov · 2023-05-31T08:46:36Z

examples/post_training_quantization/openvino/mozilla-deepspeech/main.py

@AlexKoff88 - this example is working too

AlexKoff88 · 2023-05-31T09:39:24Z

nncf/common/factory.py

I didn't get why you need BackendType.OPTIMUM?

This is redundant code from my previous experiments, please ignore

daniil-lyakhov · 2023-05-31T09:49:16Z

nncf/common/tensor_statistics/aggregator.py

+        if self._is_custom_inference:
+            sequence_container = defaultdict(list)
+            custom_forward = self.dataset.get_custom_forward(
+                engine.compiled_model, self._get_callback(model, sequence_container)


engine.compiled_model is supposed to be in the backend specific part

AlexKoff88 · 2023-05-31T09:51:46Z

examples/post_training_quantization/openvino/tiny_gpt2/main.py

+
+def set_ov_model_in_hf_model(hf_model, ov_model):
+    hf_model.model = ov_model
+    hf_model.request = ov_model.create_infer_request()


I assume that ov_model has a type of ov::Model. If so, .create_infer_request() works only for CompiledModel

You are right

AlexKoff88 · 2023-05-31T09:55:49Z

examples/post_training_quantization/openvino/tiny_gpt2/main.py

+    set_ov_model_in_hf_model(hf_model, ov_model)
+
+    def _callback_fn(info):
+        outputs = {k: v for k, v in zip(info["infer_request"].model_outputs, info["infer_request"].outputs)}


Does the InferRequest object have .model_outputs property?

Yes, and this attribute is used in HF integration https://github.com/huggingface/optimum-intel/blob/main/optimum/intel/openvino/modeling_decoder.py#L284-L287

AlexKoff88 · 2023-05-31T10:03:10Z

examples/post_training_quantization/openvino/tiny_gpt2/main.py

+    return data_item
+
+
+dataset = nncf.CustomInferenceDataset([tokens] * 10, transform_fn, get_custom_forward)


I don't think we should make get_custom_forward a part of Dataset API. I propose:

rename it to get_forward_fn(model: ov.Model, output_processing_callback: Callable) -> Callable

make it an optional argument of nncf.quantize() API

I absolutely agree that it should not be part of Dataset API.

Comments from my side:
I have some concerns about get_forward_fn:

output_processing_callback is not needed for Torch and Keras TF models. It can confuse developer because they will call output_processing_callback that does not do it anything.

signature of output_processing_callback is not clear for different frameworks.

Proposal:

Introduce get_forward_fn(model: ov.Model) -> Callable Torch and Keras TF and get_forward_fn(model: ov.Model, statistic_aggregator: StatisticsAggregator) -> Callable for OpenVINO, ONNX and TF.

Pros:

It is addressed to 1 via explicit introduction different signatures for different frameworks because different frameworks collect statistics with using different approaches.

It is addressed to 2 because methods of a class can be easily documented + sugar from IDE. It also can provide several interfaces to register model output. statistic_collector.register_model_output(name, tensor) ``statistic_collector.register_model_outputs(outputs: Dict[str, tensor])
Request changes:

def get_custom_forward(ov_model, statistic_aggregator): hf_model = model_with_pkv set_ov_model_in_hf_model(hf_model, ov_model) def _callback_fn(info): outputs = {k.key.get_any_name(): v.value for k, v in zip(info["infer_request"].model_outputs, info["infer_request"].outputs)} statistic_aggregator.register_model_outputs(outputs)

Introduce a different classes to join framework model and custom forward function for each framework. For examplenncf.OVModelWithCustomForward(model: ov.Model, get_forward_fn: Callable) for OV

Pros:

nncf.quantize and nncf.quantize_with_accuracy_control w/o extending signature

The class explicitly specified signature of get_forward_fn for framework model.

Easy reuse in other algorithms

ov_model_with_custom_forward = nncf.OVModelWithCustomForward(model_with_pkv.model, get_forward_fn) quantized_model_with_custom_forward = nncf.quantize(ov_model_with_custom_forward, dataset, subset_size=3)

IMHO: rename get_forward_fn -> make_forward_fn

To tell the truth, I am still skeptical about the whole approach of collecting recurrent states and how this is applicable to other models. Now, I am looking at the Whisper notebook and I would not use this API since it requires much more effort and code rewriting to use the proposed API.

daniil-lyakhov added 7 commits May 17, 2023 17:44

Sequential tensor reducer

0cce7af

TensorReducerSequence Reducer adapter inside reducer TensorCollectorAdapter

Move responsibility to reducers

4e8db62

Docstrings

ef0cf35

Renaming

b2eeef1

Hack to quantize gpt2

1ffeab2

gpt 2 quantization via callbacks

0c95e25

pre-commit

8df0777

github-actions bot added experimental NNCF Common NNCF ONNX NNCF OpenVINO NNCF PT NNCF TF labels May 31, 2023

Cleanup

ba40347

daniil-lyakhov commented May 31, 2023

View reviewed changes

examples/post_training_quantization/openvino/tiny_gpt2/main.py

Copy link

Owner Author

daniil-lyakhov May 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AlexKoff88 - main file

daniil-lyakhov commented May 31, 2023

View reviewed changes

nncf/common/tensor_statistics/aggregator.py

Copy link

Owner Author

daniil-lyakhov May 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AlexKoff88 - custom inference in use here

daniil-lyakhov commented May 31, 2023

View reviewed changes

examples/post_training_quantization/openvino/mozilla-deepspeech/main.py

Copy link

Owner Author

daniil-lyakhov May 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AlexKoff88 - this example is working too

AlexKoff88 reviewed May 31, 2023

View reviewed changes

daniil-lyakhov commented May 31, 2023

View reviewed changes

Remove redundant Optimum backend

61fc3db

AlexKoff88 reviewed May 31, 2023

View reviewed changes

		return data_item


		dataset = nncf.CustomInferenceDataset([tokens] * 10, transform_fn, get_custom_forward)

Conversation

daniil-lyakhov commented May 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexsu52 May 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

daniil-lyakhov commented May 31, 2023 •

edited

Loading

alexsu52 May 31, 2023 •

edited

Loading