Skip to content

Commit 613618a

Browse files
shilpakancharlacopybara-github
shilpakancharla
authored and
copybara-github
committed
Update video tutorials with links and evaluation to MoViNet
PiperOrigin-RevId: 492496066
1 parent cf5f08d commit 613618a

File tree

3 files changed

+167
-13
lines changed

3 files changed

+167
-13
lines changed

‎site/en/tutorials/load_data/video.ipynb

Lines changed: 24 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -68,15 +68,21 @@
6868
"id": "F-SqCosJ6-0H"
6969
},
7070
"source": [
71-
"This tutorial demonstrates how to load and preprocess [AVI](https://en.wikipedia.org/wiki/Audio_Video_Interleave){:.external} video data using the [UCF101 human action dataset](https://www.tensorflow.org/datasets/catalog/ucf101). Once you have preprocessed the data, it can be used for such tasks as video classification/recognition, captioning or clustering. The original dataset contains realistic action videos collected from YouTube with 101 categories, including playing cello, brushing teeth, and applying eye makeup. You will learn how to:\n",
71+
"This tutorial demonstrates how to load and preprocess [AVI](https://en.wikipedia.org/wiki/Audio_Video_Interleave) video data using the [UCF101 human action dataset](https://www.tensorflow.org/datasets/catalog/ucf101). Once you have preprocessed the data, it can be used for such tasks as video classification/recognition, captioning or clustering. The original dataset contains realistic action videos collected from YouTube with 101 categories, including playing cello, brushing teeth, and applying eye makeup. You will learn how to:\n",
7272
"\n",
7373
"* Load the data from a zip file.\n",
7474
"\n",
7575
"* Read sequences of frames out of the video files.\n",
7676
"\n",
7777
"* Visualize the video data.\n",
7878
"\n",
79-
"* Wrap the frame-generator [`tf.data.Dataset`](https://www.tensorflow.org/guide/data)."
79+
"* Wrap the frame-generator [`tf.data.Dataset`](https://www.tensorflow.org/guide/data).\n",
80+
"\n",
81+
"This video loading and preprocessing tutorial is the first part in a series of TensorFlow video tutorials. Here are the other three tutorials:\n",
82+
"\n",
83+
"- [Build a 3D CNN model for video classification](https://www.tensorflow.org/tutorials/video/video_classification): Note that this tutorial uses a (2+1)D CNN that decomposes the spatial and temporal aspects of 3D data; if you are using volumetric data such as an MRI scan, consider using a 3D CNN instead of a (2+1)D CNN.\n",
84+
"- [MoViNet for streaming action recognition](https://www.tensorflow.org/hub/tutorials/movinet): Get familiar with the MoViNet models that are available on TF Hub.\n",
85+
"- [Transfer learning for video classification with MoViNet](https://www.tensorflow.org/tutorials/video/transfer_learning_with_movinet): This tutorial explains how to use a pre-trained video classification model trained on a different dataset with the UCF-101 dataset."
8086
]
8187
},
8288
{
@@ -88,7 +94,7 @@
8894
"## Setup\n",
8995
"\n",
9096
"Begin by installing and importing some necessary libraries, including:\n",
91-
"[remotezip](https://github.com/gtsystem/python-remotezip){:.external} to inspect the contents of a ZIP file, [tqdm](https://github.com/tqdm/tqdm){:.external} to use a progress bar, [OpenCV](https://opencv.org/){:.external} to process video files, and [`tensorflow_docs`](https://github.com/tensorflow/docs/tree/master/tools/tensorflow_docs){:.external} for embedding data in a Jupyter notebook."
97+
"[remotezip](https://github.com/gtsystem/python-remotezip) to inspect the contents of a ZIP file, [tqdm](https://github.com/tqdm/tqdm) to use a progress bar, [OpenCV](https://opencv.org/) to process video files, and [`tensorflow_docs`](https://github.com/tensorflow/docs/tree/master/tools/tensorflow_docs) for embedding data in a Jupyter notebook."
9298
]
9399
},
94100
{
@@ -257,7 +263,7 @@
257263
" files: List of files in the dataset.\n",
258264
"\n",
259265
" Returns:\n",
260-
" Dictionary of class names (key) and files (values).\n",
266+
" Dictionary of class names (key) and files (values). \n",
261267
" \"\"\"\n",
262268
" files_for_class = collections.defaultdict(list)\n",
263269
" for fname in files:\n",
@@ -989,6 +995,20 @@
989995
" validation_data = val_ds,\n",
990996
" callbacks = tf.keras.callbacks.EarlyStopping(patience = 2, monitor = 'val_loss'))"
991997
]
998+
},
999+
{
1000+
"cell_type": "markdown",
1001+
"metadata": {
1002+
"id": "DdJm7ojgGxtT"
1003+
},
1004+
"source": [
1005+
"\n",
1006+
"To learn more about working with video data in TensorFlow, check out the following tutorials:\n",
1007+
"\n",
1008+
"* [Build a 3D CNN model for video classification](https://www.tensorflow.org/tutorials/video/video_classification)\n",
1009+
"* [MoViNet for streaming action recognition](https://www.tensorflow.org/hub/tutorials/movinet)\n",
1010+
"* [Transfer learning for video classification with MoViNet](https://www.tensorflow.org/tutorials/video/transfer_learning_with_movinet)"
1011+
]
9921012
}
9931013
],
9941014
"metadata": {

‎site/en/tutorials/video/transfer_learning_with_movinet.ipynb

Lines changed: 118 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,13 @@
6868
"* Replace the classifier head with the number of labels of a new dataset\n",
6969
"* Perform transfer learning on the [UCF101 dataset](https://www.crcv.ucf.edu/data/UCF101.php)\n",
7070
"\n",
71-
"The model downloaded in this tutorial is from [official/projects/movinet](https://github.com/tensorflow/models/tree/master/official/projects/movinet). This repository contains a collection of MoViNet models that TF Hub uses in the TensorFlow 2 SavedModel format."
71+
"The model downloaded in this tutorial is from [official/projects/movinet](https://github.com/tensorflow/models/tree/master/official/projects/movinet). This repository contains a collection of MoViNet models that TF Hub uses in the TensorFlow 2 SavedModel format.\n",
72+
"\n",
73+
"This transfer learning tutorial is the third part in a series of TensorFlow video tutorials. Here are the other three tutorials:\n",
74+
"\n",
75+
"- [Load video data](https://www.tensorflow.org/tutorials/load_data/video): This tutorial explains much of the code used in this document; in particular, how to preprocess and load data through the `FrameGenerator` class is explained in more detail.\n",
76+
"- [Build a 3D CNN model for video classification](https://www.tensorflow.org/tutorials/video/video_classification). Note that this tutorial uses a (2+1)D CNN that decomposes the spatial and temporal aspects of 3D data; if you are using volumetric data such as an MRI scan, consider using a 3D CNN instead of a (2+1)D CNN.\n",
77+
"- [MoViNet for streaming action recognition](https://www.tensorflow.org/hub/tutorials/movinet): Get familiar with the MoViNet models that are available on TF Hub."
7278
]
7379
},
7480
{
@@ -111,6 +117,7 @@
111117
"import cv2\n",
112118
"import numpy as np\n",
113119
"import remotezip as rz\n",
120+
"import seaborn as sns\n",
114121
"import matplotlib.pyplot as plt\n",
115122
"\n",
116123
"import keras\n",
@@ -132,7 +139,7 @@
132139
},
133140
"source": [
134141
"## Load data\n",
135-
"\n",
142+
" \n",
136143
"The hidden cell below defines helper functions to download a slice of data from the UCF-101 dataset, and load it into a `tf.data.Dataset`. The [Loading video data tutorial](https://www.tensorflow.org/tutorials/load_data/video) provides a detailed walkthrough of this code.\n",
137144
"\n",
138145
"The `FrameGenerator` class at the end of the hidden block is the most important utility here. It creates an iterable object that can feed data into the TensorFlow data pipeline. Specifically, this class contains a Python generator that loads the video frames along with its encoded label. The generator (`__call__`) function yields the frame array produced by `frames_from_video_file` and a one-hot encoded vector of the label associated with the set of frames.\n",
@@ -598,6 +605,111 @@
598605
" verbose=1)"
599606
]
600607
},
608+
{
609+
"cell_type": "markdown",
610+
"metadata": {
611+
"id": "KkLl2zF8G9W0"
612+
},
613+
"source": [
614+
"## Evaluate the model\n",
615+
"\n",
616+
"The model achieved high accuracy on the training dataset. Next, use Keras `Model.evaluate` to evaluate it on the test set."
617+
]
618+
},
619+
{
620+
"cell_type": "code",
621+
"execution_count": null,
622+
"metadata": {
623+
"id": "NqgbzOiKuxxT"
624+
},
625+
"outputs": [],
626+
"source": [
627+
"model.evaluate(test_ds, return_dict=True)"
628+
]
629+
},
630+
{
631+
"cell_type": "markdown",
632+
"metadata": {
633+
"id": "OkFst2gsHBwD"
634+
},
635+
"source": [
636+
"To visualize model performance further, use a [confusion matrix](https://www.tensorflow.org/api_docs/python/tf/math/confusion_matrix). The confusion matrix allows you to assess the performance of the classification model beyond accuracy. In order to build the confusion matrix for this multi-class classification problem, get the actual values in the test set and the predicted values."
637+
]
638+
},
639+
{
640+
"cell_type": "code",
641+
"execution_count": null,
642+
"metadata": {
643+
"id": "hssSdW9XHF_j"
644+
},
645+
"outputs": [],
646+
"source": [
647+
"def get_actual_predicted_labels(dataset):\n",
648+
" \"\"\"\n",
649+
" Create a list of actual ground truth values and the predictions from the model.\n",
650+
"\n",
651+
" Args:\n",
652+
" dataset: An iterable data structure, such as a TensorFlow Dataset, with features and labels.\n",
653+
"\n",
654+
" Return:\n",
655+
" Ground truth and predicted values for a particular dataset.\n",
656+
" \"\"\"\n",
657+
" actual = [labels for _, labels in dataset.unbatch()]\n",
658+
" predicted = model.predict(dataset)\n",
659+
"\n",
660+
" actual = tf.stack(actual, axis=0)\n",
661+
" predicted = tf.concat(predicted, axis=0)\n",
662+
" predicted = tf.argmax(predicted, axis=1)\n",
663+
"\n",
664+
" return actual, predicted"
665+
]
666+
},
667+
{
668+
"cell_type": "code",
669+
"execution_count": null,
670+
"metadata": {
671+
"id": "2TmTue6THGWO"
672+
},
673+
"outputs": [],
674+
"source": [
675+
"def plot_confusion_matrix(actual, predicted, labels, ds_type):\n",
676+
" cm = tf.math.confusion_matrix(actual, predicted)\n",
677+
" ax = sns.heatmap(cm, annot=True, fmt='g')\n",
678+
" sns.set(rc={'figure.figsize':(12, 12)})\n",
679+
" sns.set(font_scale=1.4)\n",
680+
" ax.set_title('Confusion matrix of action recognition for ' + ds_type)\n",
681+
" ax.set_xlabel('Predicted Action')\n",
682+
" ax.set_ylabel('Actual Action')\n",
683+
" plt.xticks(rotation=90)\n",
684+
" plt.yticks(rotation=0)\n",
685+
" ax.xaxis.set_ticklabels(labels)\n",
686+
" ax.yaxis.set_ticklabels(labels)"
687+
]
688+
},
689+
{
690+
"cell_type": "code",
691+
"execution_count": null,
692+
"metadata": {
693+
"id": "4RK1A1C1HH6V"
694+
},
695+
"outputs": [],
696+
"source": [
697+
"fg = FrameGenerator(subset_paths['train'], num_frames, training = True)\n",
698+
"label_names = list(fg.class_ids_for_name.keys())"
699+
]
700+
},
701+
{
702+
"cell_type": "code",
703+
"execution_count": null,
704+
"metadata": {
705+
"id": "r4AFi2e5HKEO"
706+
},
707+
"outputs": [],
708+
"source": [
709+
"actual, predicted = get_actual_predicted_labels(test_ds)\n",
710+
"plot_confusion_matrix(actual, predicted, label_names, 'test')"
711+
]
712+
},
601713
{
602714
"cell_type": "markdown",
603715
"metadata": {
@@ -611,10 +723,11 @@
611723
"\n",
612724
"In particular, using the `FrameGenerator` class used in this tutorial and the other video data and classification tutorials will help you load data into your models.\n",
613725
"\n",
614-
"To learn more about video data, check out:\n",
726+
"To learn more about working with video data in TensorFlow, check out the following tutorials:\n",
615727
"\n",
616-
"- [Load video data](https://www.tensorflow.org/tutorials/load_data/video): This tutorial explains much of the code used in this document.\n",
617-
"- [Build a 3D CNN model for video classification](https://www.tensorflow.org/tutorials/video/video_classification). Note that this tutorial uses a (2+1)D CNN that decomposes the spatial and temporal aspects of 3D data; if you are using volumetric data such as an MRI scan, consider using a 3D CNN instead of a (2+1)D CNN."
728+
"* [Load video data](https://www.tensorflow.org/tutorials/load_data/video)\n",
729+
"* [Build a 3D CNN model for video classification](https://www.tensorflow.org/tutorials/video/video_classification)\n",
730+
"* [MoViNet for streaming action recognition](https://www.tensorflow.org/hub/tutorials/movinet)"
618731
]
619732
}
620733
],

‎site/en/tutorials/video/video_classification.ipynb

Lines changed: 25 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,13 @@
6666
"* Build an input pipeline\n",
6767
"* Build a 3D convolutional neural network model with residual connections using Keras functional API\n",
6868
"* Train the model\n",
69-
"* Evaluate and test the model"
69+
"* Evaluate and test the model \n",
70+
"\n",
71+
"This video classification tutorial is the second part in a series of TensorFlow video tutorials. Here are the other three tutorials:\n",
72+
"\n",
73+
"- [Load video data](https://www.tensorflow.org/tutorials/load_data/video): This tutorial explains much of the code used in this document.\n",
74+
"- [MoViNet for streaming action recognition](https://www.tensorflow.org/hub/tutorials/movinet): Get familiar with the MoViNet models that are available on TF Hub.\n",
75+
"- [Transfer learning for video classification with MoViNet](https://www.tensorflow.org/tutorials/video/transfer_learning_with_movinet): This tutorial explains how to use a pre-trained video classification model trained on a different dataset with the UCF-101 dataset."
7076
]
7177
},
7278
{
@@ -835,7 +841,7 @@
835841
},
836842
"outputs": [],
837843
"source": [
838-
"model.evaluate(test_ds, return_dict = True)"
844+
"model.evaluate(test_ds, return_dict=True)"
839845
]
840846
},
841847
{
@@ -905,8 +911,8 @@
905911
},
906912
"outputs": [],
907913
"source": [
908-
"labels = ['ApplyEyeMakeup', 'ApplyLipstick', 'Archery', 'BabyCrawling', 'BalanceBeam',\n",
909-
" 'BandMarching', 'BaseballPitch', 'Basketball', 'BasketballDunk', 'BenchPress']"
914+
"fg = FrameGenerator(subset_paths['train'], num_frames, training = True)\n",
915+
"label_names = list(fg.class_ids_for_name.keys())"
910916
]
911917
},
912918
{
@@ -1013,6 +1019,21 @@
10131019
"source": [
10141020
"recall"
10151021
]
1022+
},
1023+
{
1024+
"cell_type": "markdown",
1025+
"metadata": {
1026+
"id": "d4WsP4Z2HZ6L"
1027+
},
1028+
"source": [
1029+
"## Next Steps\n",
1030+
"\n",
1031+
"To learn more about working with video data in TensorFlow, check out the following tutorials:\n",
1032+
"\n",
1033+
"* [Load video data](https://www.tensorflow.org/tutorials/load_data/video)\n",
1034+
"* [MoViNet for streaming action recognition](https://www.tensorflow.org/hub/tutorials/movinet)\n",
1035+
"* [Transfer learning for video classification with MoViNet](https://www.tensorflow.org/tutorials/video/transfer_learning_with_movinet)"
1036+
]
10161037
}
10171038
],
10181039
"metadata": {

0 commit comments

Comments
 (0)