|
47 | 47 | "metadata": {}, |
48 | 48 | "outputs": [], |
49 | 49 | "source": [ |
50 | | - "!python inference_commonVoice.py -p /data/commonVoice/test" |
| 50 | + "!python inference_commonVoice.py -p ${COMMON_VOICE_PATH}/processed_data/test" |
51 | 51 | ] |
52 | 52 | }, |
53 | 53 | { |
54 | 54 | "cell_type": "markdown", |
55 | 55 | "metadata": {}, |
56 | 56 | "source": [ |
57 | 57 | "## inference_custom.py for Custom Data \n", |
58 | | - "To generate an overall results output summary, the audio_ground_truth_labels.csv file needs to be modified with the name of the audio file and expected audio label (i.e. en for English). By default, this is disabled but if desired, the *--ground_truth_compare* can be used. To run inference on custom data, you must specify a folder with WAV files and pass the path in as an argument. " |
| 58 | + "To run inference on custom data, you must specify a folder with .wav files and pass the path in as an argument. You can do so by creating a folder named `data_custom` and then copy 1 or 2 .wav files from your test dataset into it. .mp3 files will NOT work. " |
59 | 59 | ] |
60 | 60 | }, |
61 | 61 | { |
|
65 | 65 | "### Randomly select audio clips from audio files for prediction\n", |
66 | 66 | "python inference_custom.py -p DATAPATH -d DURATION -s SIZE\n", |
67 | 67 | "\n", |
68 | | - "An output file output_summary.csv will give the summary of the results." |
| 68 | + "An output file `output_summary.csv` will give the summary of the results." |
69 | 69 | ] |
70 | 70 | }, |
71 | 71 | { |
|
104 | 104 | "### Optimizations with Intel® Extension for PyTorch (IPEX) \n", |
105 | 105 | "python inference_custom.py -p data_custom -d 3 -s 50 --vad --ipex --verbose \n", |
106 | 106 | "\n", |
| 107 | + "This will apply ipex.optimize to the model(s) and TorchScript. You can also add the --bf16 option along with --ipex to run in the BF16 data type, supported on 4th Gen Intel® Xeon® Scalable processors and newer.\n", |
| 108 | + "\n", |
107 | 109 | "Note that the *--verbose* option is required to view the latency measurements. " |
108 | 110 | ] |
109 | 111 | }, |
|
121 | 123 | "metadata": {}, |
122 | 124 | "source": [ |
123 | 125 | "## Quantization with Intel® Neural Compressor (INC)\n", |
124 | | - "To improve inference latency, Intel® Neural Compressor (INC) can be used to quantize the trained model from FP32 to INT8 by running quantize_model.py. The *-datapath* argument can be used to specify a custom evaluation dataset but by default it is set to */data/commonVoice/dev* which was generated from the data preprocessing scripts in the *Training* folder. " |
| 126 | + "To improve inference latency, Intel® Neural Compressor (INC) can be used to quantize the trained model from FP32 to INT8 by running quantize_model.py. The *-datapath* argument can be used to specify a custom evaluation dataset but by default it is set to `$COMMON_VOICE_PATH/processed_data/dev` which was generated from the data preprocessing scripts in the `Training` folder. " |
125 | 127 | ] |
126 | 128 | }, |
127 | 129 | { |
|
137 | 139 | "cell_type": "markdown", |
138 | 140 | "metadata": {}, |
139 | 141 | "source": [ |
140 | | - "After quantization, the model will be stored in *lang_id_commonvoice_model_INT8* and *neural_compressor.utils.pytorch.load* will have to be used to load the quantized model for inference. " |
| 142 | + "After quantization, the model will be stored in lang_id_commonvoice_model_INT8 and neural_compressor.utils.pytorch.load will have to be used to load the quantized model for inference. If self.language_id is the original model and data_path is the path to the audio file:\n", |
| 143 | + "\n", |
| 144 | + "```\n", |
| 145 | + "from neural_compressor.utils.pytorch import load\n", |
| 146 | + "model_int8 = load(\"./lang_id_commonvoice_model_INT8\", self.language_id)\n", |
| 147 | + "signal = self.language_id.load_audio(data_path)\n", |
| 148 | + "prediction = self.model_int8(signal)\n", |
| 149 | + "```" |
| 150 | + ] |
| 151 | + }, |
| 152 | + { |
| 153 | + "cell_type": "markdown", |
| 154 | + "metadata": {}, |
| 155 | + "source": [ |
| 156 | + "The code above is integrated into inference_custom.py. You can now run inference on your data using this INT8 model:" |
| 157 | + ] |
| 158 | + }, |
| 159 | + { |
| 160 | + "cell_type": "code", |
| 161 | + "execution_count": null, |
| 162 | + "metadata": {}, |
| 163 | + "outputs": [], |
| 164 | + "source": [ |
| 165 | + "!python inference_custom.py -p data_custom -d 3 -s 50 --vad --int8_model --verbose" |
| 166 | + ] |
| 167 | + }, |
| 168 | + { |
| 169 | + "cell_type": "markdown", |
| 170 | + "metadata": {}, |
| 171 | + "source": [ |
| 172 | + "### (Optional) Comparing Predictions with Ground Truth\n", |
| 173 | + "\n", |
| 174 | + "You can choose to modify audio_ground_truth_labels.csv to include the name of the audio file and expected audio label (like, en for English), then run inference_custom.py with the --ground_truth_compare option. By default, this is disabled." |
141 | 175 | ] |
142 | 176 | }, |
143 | 177 | { |
|
0 commit comments