|
| 1 | +{ |
| 2 | + "cells": [ |
| 3 | + { |
| 4 | + "cell_type": "markdown", |
| 5 | + "metadata": {}, |
| 6 | + "source": [ |
| 7 | + "## Object Tracking with `arcgis.learn`\n", |
| 8 | + "\n", |
| 9 | + "Object tracking is the process of:\n", |
| 10 | + "\n", |
| 11 | + "* Taking an initial set of object detections (such as an input set of bounding box coordinates)\n", |
| 12 | + "* Creating a unique ID for each of the initial detections\n", |
| 13 | + "* And then tracking each of the objects as they move around frames in a video, maintaining the assignment of unique IDs\n", |
| 14 | + "\n", |
| 15 | + "Multiple-objects tracking can be performed using `predict_video` function of the `arcgis.learn` module." |
| 16 | + ] |
| 17 | + }, |
| 18 | + { |
| 19 | + "cell_type": "markdown", |
| 20 | + "metadata": {}, |
| 21 | + "source": [ |
| 22 | + "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n", |
| 23 | + "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Prerequisites\" data-toc-modified-id=\"Prerequisites-1\">Prerequisites</a></span></li><li><span><a href=\"#How-Object-Tracking-Works\" data-toc-modified-id=\"How-Object-Tracking-Works\">How Object Tracking Works</a></span><ul class=\"toc-item\"><li><span><a href=\"#Kalaman-Filter\" data-toc-modified-id=\"Kalman-Filter\">Kalman Filter</a></span></li><li><span><a href=\"#Hungarian-Assignment-Algorithm\" data-toc-modified-id=\"Hungarian-Assignment-Algorithm\">Hungarian Assignment Algorithm</a></span></li></ul></li><li><span><a href=\"#Track-Objects-Using-arcgis.learn\" data-toc-modified-id=\"Track-Objects-Using-arcgis.learn\">Track Objects Using arcgis.learn</a></span></li><li><span><a href=\"#Vehicle-Tracking-Example\" data-toc-modified-id=\"Vehicle-Tracking-Example\">Vehicle Tracking Example</a></span></li></ul></div>" |
| 24 | + ] |
| 25 | + }, |
| 26 | + { |
| 27 | + "cell_type": "markdown", |
| 28 | + "metadata": {}, |
| 29 | + "source": [ |
| 30 | + "## Prerequisites\n", |
| 31 | + "\n", |
| 32 | + "- Please refer to the prerequisites section in our [guide](https://developers.arcgis.com/python/guide/geospatial-deep-learning/) for more information. This sample demonstrates how to do object tracking using arcgis.learn.\n", |
| 33 | + "- Please refer to [guide](https://developers.arcgis.com/python/guide/object-detection/) to understand how object detection works." |
| 34 | + ] |
| 35 | + }, |
| 36 | + { |
| 37 | + "cell_type": "markdown", |
| 38 | + "metadata": {}, |
| 39 | + "source": [ |
| 40 | + "## How Object Tracking Works\n", |
| 41 | + "\n", |
| 42 | + "Object tracking in `arcgis.learn` is based SORT(Simple Online Realtime Tracking) Algorithm. This Algorithm combines __Kalman-filtering and Hungarian Assignment Algorithm__\n", |
| 43 | + "\n", |
| 44 | + "__Kalman Filter__ is used to estimate the position of a tracker while __Hungarian Algorithm__ is used to assign trackers to a new detection.\n", |
| 45 | + "Following sections briefly describe __Kalman Filter__ and __Hungarian Algorithm__." |
| 46 | + ] |
| 47 | + }, |
| 48 | + { |
| 49 | + "cell_type": "markdown", |
| 50 | + "metadata": {}, |
| 51 | + "source": [ |
| 52 | + "## Kalman Filter\n", |
| 53 | + "\n", |
| 54 | + "Kalman filtering uses a series of measurements observed over time and produces estimates of unknown variables by estimating a joint probability distribution over the variables for each timeframe. The filter is named after Rudolf E. Kálmán, one of the primary developers of its theory.\n", |
| 55 | + "\n", |
| 56 | + "Our state contains 8 variables; `(u,v,a,h,u’,v’,a’,h’)` where `(u,v)` are centres of the bounding boxes, a is the aspect ratio and h, the height of the image. The other variables are the respective velocities of the variables.\n", |
| 57 | + "\n", |
| 58 | + "A Kalman Filter is used on every bounding box, so it comes after a box has been matched with a tracker. When the association is made, predict and update functions are called. \n", |
| 59 | + "\n", |
| 60 | + "#### Predict\n", |
| 61 | + "\n", |
| 62 | + "Prediction step is matrix multiplication that will tell us the position of our bounding box at time t based on its position at time t-1.\n", |
| 63 | + "\n", |
| 64 | + "#### Update\n", |
| 65 | + "\n", |
| 66 | + "Update phase is a correction step. It includes the new measurement from the Object Detection model and helps improve our filter.\n" |
| 67 | + ] |
| 68 | + }, |
| 69 | + { |
| 70 | + "cell_type": "markdown", |
| 71 | + "metadata": {}, |
| 72 | + "source": [ |
| 73 | + "## Hungarian Assignment Algorithm\n", |
| 74 | + "\n", |
| 75 | + "The Hungarian algorithm, also known as Kuhn-Munkres algorithm, can associate an obstacle from one frame to another, based on a score such as Intersection over Union (IoU). \n", |
| 76 | + "\n", |
| 77 | + "We iterate through the list of trackers and detections and assign a tracker to each detection on the basis of IoU scores.\n" |
| 78 | + ] |
| 79 | + }, |
| 80 | + { |
| 81 | + "cell_type": "markdown", |
| 82 | + "metadata": {}, |
| 83 | + "source": [ |
| 84 | + "\n", |
| 85 | + "__The general process is to detect obstacles using an object detection algorithm, match these bounding box with former bounding boxes we have using The Hungarian Algorithm and then predict future bounding box positions or actual positions using Kalman Filters.__ " |
| 86 | + ] |
| 87 | + }, |
| 88 | + { |
| 89 | + "cell_type": "markdown", |
| 90 | + "metadata": {}, |
| 91 | + "source": [ |
| 92 | + "## Track Objects Using arcgis.learn\n", |
| 93 | + "\n", |
| 94 | + "Multiple-object tracking can be performed using `predict_video` function of the `arcgis.learn` module. To enable tracking, set the `track` parameter in the `predict_video` function as `track = True`.\n", |
| 95 | + "\n", |
| 96 | + "The following options/parameters are available in the predict video function for the user to decide:-\n", |
| 97 | + "\n", |
| 98 | + "* `vanish_frames` i.e. the number of frames the object remains absent from the frame to be considered as vanished.\n", |
| 99 | + "\n", |
| 100 | + "* `detect_frames` i.e. the number of frames an object remains present in the frame to start tracking.\n", |
| 101 | + "\n", |
| 102 | + "* `assignment_iou_thrd` i.e. There might be multiple trackers detecting and tracking objects. The Intersection over Union (iou) threshold can be set to assign a tracker with the mentioned threshold value.\n" |
| 103 | + ] |
| 104 | + }, |
| 105 | + { |
| 106 | + "cell_type": "markdown", |
| 107 | + "metadata": {}, |
| 108 | + "source": [ |
| 109 | + "## Vehicle Tracking Example\n", |
| 110 | + "\n", |
| 111 | + "The following video has been created using `predict_video()` function of a `Retinanet` model from `arcgis.learn`. \n", |
| 112 | + "\n", |
| 113 | + "*The data is collected from a lamp post in Berlin.*\n", |
| 114 | + "\n" |
| 115 | + ] |
| 116 | + }, |
| 117 | + { |
| 118 | + "cell_type": "code", |
| 119 | + "execution_count": 6, |
| 120 | + "metadata": {}, |
| 121 | + "outputs": [ |
| 122 | + { |
| 123 | + "data": { |
| 124 | + "text/html": [ |
| 125 | + "\n", |
| 126 | + " <video alt=\"test\" controls>\n", |
| 127 | + " <source src=\"data/test_predictions.mp4\" type=\"video/mp4\">\n", |
| 128 | + " </video>\n" |
| 129 | + ], |
| 130 | + "text/plain": [ |
| 131 | + "<IPython.core.display.HTML object>" |
| 132 | + ] |
| 133 | + }, |
| 134 | + "execution_count": 6, |
| 135 | + "metadata": {}, |
| 136 | + "output_type": "execute_result" |
| 137 | + } |
| 138 | + ], |
| 139 | + "source": [ |
| 140 | + "from IPython.display import HTML\n", |
| 141 | + "HTML(\"\"\"\n", |
| 142 | + " <video alt=\"test\" controls>\n", |
| 143 | + " <source src=\"data/test_predictions.mp4\" type=\"video/mp4\">\n", |
| 144 | + " </video>\n", |
| 145 | + "\"\"\")" |
| 146 | + ] |
| 147 | + }, |
| 148 | + { |
| 149 | + "cell_type": "code", |
| 150 | + "execution_count": null, |
| 151 | + "metadata": {}, |
| 152 | + "outputs": [], |
| 153 | + "source": [] |
| 154 | + } |
| 155 | + ], |
| 156 | + "metadata": { |
| 157 | + "kernelspec": { |
| 158 | + "display_name": "Python 3", |
| 159 | + "language": "python", |
| 160 | + "name": "python3" |
| 161 | + }, |
| 162 | + "language_info": { |
| 163 | + "codemirror_mode": { |
| 164 | + "name": "ipython", |
| 165 | + "version": 3 |
| 166 | + }, |
| 167 | + "file_extension": ".py", |
| 168 | + "mimetype": "text/x-python", |
| 169 | + "name": "python", |
| 170 | + "nbconvert_exporter": "python", |
| 171 | + "pygments_lexer": "ipython3", |
| 172 | + "version": "3.6.9" |
| 173 | + } |
| 174 | + }, |
| 175 | + "nbformat": 4, |
| 176 | + "nbformat_minor": 2 |
| 177 | +} |
0 commit comments