You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<p>DEIMKit is a Python wrapper for <ahref="https://github.com/ShihuaHuang95/DEIM">DEIM: DETR with Improved Matching for Fast Convergence</a>. Check out the original repo for more details.</p>
> I recommend using [Pixi](https://pixi.sh) to run this package. Pixi makes it easy to install the right version of Python and the dependencies to run this package on any platform!
@@ -85,7 +129,7 @@ This will download a toy dataset with 8 images, and train a model on it for 3 ep
85
129
86
130
If this runs without any issues, you've got a working Python environment with all the dependencies installed. This also installs DEIMKit in editable mode for development. See the [pixi cheatsheet](#-pixi-cheat-sheet) below for more.
87
131
88
-
## Usage
132
+
## 🚀 Usage
89
133
90
134
List models supported by DEIMKit
91
135
@@ -103,7 +147,7 @@ list_models()
103
147
'deim_hgnetv2_x']
104
148
```
105
149
106
-
### Inference
150
+
### 💡 Inference
107
151
108
152
Load a pretrained model by the original authors
109
153
@@ -157,7 +201,7 @@ Stomata Dataset
157
201
158
202
See the [demo notebook on using pretrained models](nbs/pretrained-model-inference.ipynb) and [custom model inference](nbs/custom-model-inference.ipynb) for more details.
159
203
160
-
### Training
204
+
### 🏋️ Training
161
205
162
206
DEIMKit provides a simple interface for training your own models.
163
207
@@ -225,7 +269,8 @@ Navigate to the http://localhost:6006/ in your browser to view the training prog
225
269
226
270
227
271
228
-
### Export
272
+
### 💾 Export
273
+
Currently, the export function is only used for exporting the model to ONNX and run it using ONNXRuntime (see [Live Inference](#-live-inference) for more details). I think one could get pretty far with this even on a low resource machine. Drop an issue if you think this should be extended to other formats.
> The exported model will accept raw BGR images of any size. It will also handle the preprocessing internally. Credit to [PINTO0309](https://github.com/PINTO0309/DEIM) for the implementation.
290
+
>
291
+
> 
292
+
293
+
> [!TIP]
294
+
> If you want to export to OpenVINO you can do so directly from the ONNX model.
295
+
>
296
+
>
297
+
> ```python
298
+
>import onnx
299
+
>from onnx import helper
300
+
>
301
+
> model = onnx.load("best.onnx")
302
+
>
303
+
># Change the mode attribute of the GridSample node to bilinear as this operation is not supported in OpenVINO
304
+
>for node in model.graph.node:
305
+
>if node.op_type =='GridSample':
306
+
>for i, attr inenumerate(node.attribute):
307
+
>if attr.name =='mode'and attr.s ==b'linear':
308
+
># Replace 'linear' with 'bilinear'
309
+
> node.attribute[i].s =b'bilinear'
310
+
>
311
+
># Save the modified model
312
+
> onnx.save(model, "best_prep_openvino.onnx")
313
+
>```
314
+
> You can then use the live inference script to run inference on the OpenVINO model.
315
+
316
+
### 🖥️ Gradio App
317
+
Run a Gradio app to interact with your model. The app will accept raw BGR images of any size. It will also handle the preprocessing internally using the exported ONNX model.
245
318
246
319
```bash
247
-
python scripts/gradio_demo.py
320
+
python scripts/gradio_demo.py \
321
+
--model "best.onnx" \
322
+
--classes "classes.txt" \
323
+
--examples "Rock Paper Scissors SXSW.v14i.coco/test"
248
324
```
249
325
250
326
251
-
### Live Inference
327
+
> [!NOTE]
328
+
> The demo app uses onnx model and onnxruntime for inference. Additionally, I have also made it that the ONNX model to accept any input size, despite the original model was trained on 640x640 images.
329
+
> This means you can use any image size you want. Play around with the input size slider to see what works best for your model.
330
+
> Some objects are visible even at lower input sizes, this means you can use a lower input size to speed up inference.
331
+
332
+
### 🎥 Live Inference
252
333
Run live inference on a video, image or webcam using ONNXRuntime. This runs on CPU by default.
253
-
If you would like to use the CUDA backend, you can install the `onnxruntime-gpu` package and uninstall the `onnxruntime` package.
334
+
If you would like to use the CUDA backend, install the `onnxruntime-gpu` package and uninstall the `onnxruntime` package.
254
335
255
-
For video inference, specify the path to the video file as the input. Output video will be saved as `onnx_result.mp4` in the current directory.
336
+
For running inference on a webcam, set the `--webcam` flag.
256
337
257
338
```bash
258
339
python scripts/live_inference.py
259
-
--onnx model.onnx # Path to the ONNX model file
260
-
--input video.mp4 # Path to the input video file
261
-
--class-names classes.txt # Path to the classes file with each name on a new row
262
-
--input-size 320 # Input size for the model
340
+
--model model.onnx # Path to the ONNX model file
341
+
--webcam # Use webcam as input source
342
+
--classes classes.txt # Path to the classes file with each name on a new row
The following is a demo of video inference after training for about 50 epochs on the vehicles dataset with image size 320x320.
348
+
Because we are handling the preprocessing internally in the ONNX model, the input size is not limited to the original 640x640. You can use any input size you want for inference. The model was trained on 640x640 images. Integrating the preprocessing internally in the ONNX model also lets us run inference at very high FPS as it uses more efficient onnx operators.
The following is a model I trained on a custom dataset using the deim_hgnetv2_s model and exported to ONNX. Here are some examples of inference on a webcam at different video resolutions.
268
351
352
+
Webcam video width at 1920x1080 pixels (1080p):
269
353
270
-
You can also run live inference on a webcam by setting the `webcam` flag.
The following is an inference using the pre-trained model `deim_hgnetv2_x` trained on COCO. See how I exported the pre-trained model to onnx in this notebook [here](nbs/export.ipynb).
I'm not affiliated with the original DEIM authors. I just found the model interesting and wanted to try it out. The changes made here are of my own. Please cite and star the original repo if you find this useful.
0 commit comments