Merge branch 'feature/pytorch-handwritting-recognition' into develop

Author: pythonlessons

CHANGELOG.md

@@ -1,3 +1,16 @@
+## [1.0.2] - 2022-03-... (unreleased)
+### Changed
+- changes `OnnxInferenceModel` in `mltu.torch.inferenceModels` to load custom metadata from saved ONNX model
+- improved `mltu.dataProvider` to remove bad samples from dataset on epoch end
+
+### Added:
+- added `mltu.torch.losses`, used to create PyTorch losses, that may be used in training and validation
+- added CTC loss to `mltu.torch.losses` that can be used for training CTC based models
+- added `Model2onnx` and `Tensorboard` callbacks to `mltu.torch.callbacks`, used to create PyTorch callbacks, that may be used in training and validation
+- added `CERMetric` and `WERMetric` to `mltu.torch.metrics`, used to create PyTorch metrics, that may be used in training and validation
+- created 08 pytorch tutorial, that shows how to use `mltu.torch` to train CTC based models
+
+
 ## [1.0.1] - 2022-03-06
 ### Changed
 - In all tutorials removed stow dependency and replaced with os package, to make it easier to use on Windows 11
@@ -25,7 +38,7 @@
 - 
 ### Added:
 - added 05_sound_to_text tutorial
-- added WavReader to mltu/preprocessors, used to read wav files and convert them to numpy arrays
+- added `WavReader` to `mltu/preprocessors`, used to read wav files and convert them to numpy arrays
 
 
 ## [0.1.7] - 2022-02-03
@@ -35,11 +48,11 @@
 
 ## [0.1.5] - 2022-01-10
 ### Changed
-- seperated CWERMetric to SER and WER Metrics in mltu.metrics, Character/word rate was calculatted in a wrong way
+- seperated `CWERMetric` to `CER` and `WER` Metrics in `mltu.metrics`, Character/word rate was calculatted in a wrong way
 - created @setter for augmentors and transformers in DataProvider, to properlly add augmentors and transformers to the pipeline
 - augmentors and transformers must inherit from `mltu.augmentors.base.Augmentor` and `mltu.transformers.base.Transformer` respectively
 - updated ImageShowCV2 transformer documentation
-- fixed OnnxInferenceModel in mltu.inferenceModels to use CPU even if GPU is available with force_cpu=True flag
+- fixed OnnxInferenceModel in `mltu.inferenceModels` to use CPU even if GPU is available with force_cpu=True flag
 
 ### Added:
 - added RandomSharpen to mltu.augmentors, used for simple image augmentation;

README.md

@@ -22,4 +22,5 @@ Each tutorial has its own requirements.txt file for a specific mltu version. As
 4. [Handwritten sentence recognition with TensorFlow](https://pylessons.com/handwritten-sentence-recognition), code in ```Tutorials\04_sentence_recognition``` folder;
 5. [Introduction to speech recognition with TensorFlow](https://pylessons.com/speech-recognition), code in ```Tutorials\05_speech_recognition``` folder;
 6. [Introduction to PyTorch in a practical way](https://pylessons.com/pytorch-introduction), code in ```Tutorials\06_pytorch_introduction``` folder;
-7. [Using custom wrapper to simplify PyTorch models training pipeline](https://pylessons.com/pytorch-introduction), code in ```Tutorials\07_pytorch_wrapper``` folder;
+7. [Using custom wrapper to simplify PyTorch models training pipeline](https://pylessons.com/pytorch-introduction), code in ```Tutorials\07_pytorch_wrapper``` folder;
+8. [Handwriting words recognition with PyTorch](https://pylessons.com/handwriting-recognition-pytorch), code in ```Tutorials\08_handwriting_recognition_torch``` folder;

Tutorials/02_captcha_to_text/train.py

@@ -34,10 +34,10 @@ def download_and_unzip(url, extract_to='Datasets'):
 captcha_path = os.path.join('Datasets', 'captcha_images_v2')
 for file in os.listdir(captcha_path):
     file_path = os.path.join(captcha_path, file)
-    file_name = os.path.splitext(file)[0]
-    dataset.append([file_path, file_name])
-    vocab.update(list(file_name))
-    max_len = max(max_len, len(file_name))
+    label = os.path.splitext(file)[0] # Get the file name without the extension
+    dataset.append([file_path, label])
+    vocab.update(list(label))
+    max_len = max(max_len, len(label))
 
 configs = ModelConfigs()
 

Tutorials/08_handwriting_recognition_torch/README.md

@@ -0,0 +1,9 @@
+# Using custom wrapper to simplify PyTorch models training pipeline
+### Construct an accurate handwriting recognition model with PyTorch! Understand how to use MLTU package, to simplify the PyTorch models training pipeline, and discover methods to enhance your model's accuracy!<br><br>
+
+# **Detailed tutorial**:
+### [Handwriting words recognition with PyTorch](https://pylessons.com/handwriting-recognition-pytorch)
+
+<p align="center">
+    <img src="https://pylessons.com/media/Tutorials/mltu/handwriting-recognition-pytorch/handwriting-recognition-pytorch.png">
+</p>

Tutorials/08_handwriting_recognition_torch/configs.py

@@ -0,0 +1,16 @@
+import os
+from datetime import datetime
+
+from mltu.configs import BaseModelConfigs
+
+class ModelConfigs(BaseModelConfigs):
+    def __init__(self):
+        super().__init__()
+        self.model_path = os.path.join('Models/08_handwriting_recognition_torch', datetime.strftime(datetime.now(), "%Y%m%d%H%M"))
+        self.vocab = ''
+        self.height = 32
+        self.width = 128
+        self.max_text_length = 0
+        self.batch_size = 64
+        self.learning_rate = 0.002
+        self.train_epochs = 1000

Tutorials/08_handwriting_recognition_torch/inferenceModel.py

@@ -0,0 +1,42 @@
+import cv2
+import typing
+import numpy as np
+
+from mltu.inferenceModel import OnnxInferenceModel
+from mltu.utils.text_utils import ctc_decoder, get_cer
+
+class ImageToWordModel(OnnxInferenceModel):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+
+    def predict(self, image: np.ndarray):
+        image = cv2.resize(image, self.input_shape[:2][::-1])
+
+        image_pred = np.expand_dims(image, axis=0).astype(np.float32)
+
+        preds = self.model.run(None, {self.input_name: image_pred})[0]
+
+        text = ctc_decoder(preds, self.vocab)[0]
+
+        return text
+
+if __name__ == "__main__":
+    import pandas as pd
+    from tqdm import tqdm
+
+    model = ImageToWordModel(model_path="Models/08_handwriting_recognition_torch/202303142139/model.onnx")
+
+    df = pd.read_csv("Models/08_handwriting_recognition_torch/202303142139/val.csv").values.tolist()
+
+    accum_cer = []
+    for image_path, label in tqdm(df):
+        image = cv2.imread(image_path)
+
+        prediction_text = model.predict(image)
+
+        cer = get_cer(prediction_text, label)
+        print(f"Image: {image_path}, Label: {label}, Prediction: {prediction_text}, CER: {cer}")
+
+        accum_cer.append(cer)
+
+    print(f"Average CER: {np.average(accum_cer)}")

Tutorials/08_handwriting_recognition_torch/model.py

@@ -0,0 +1,113 @@
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+def activation_layer(activation: str='relu', alpha: float=0.1, inplace: bool=True):
+    """ Activation layer wrapper for LeakyReLU and ReLU activation functions
+
+    Args:
+        activation: str, activation function name (default: 'relu')
+        alpha: float (LeakyReLU activation function parameter)
+
+    Returns:
+        torch.Tensor: activation layer
+    """
+    if activation == 'relu':
+        return nn.ReLU(inplace=inplace)
+    
+    elif activation == 'leaky_relu':
+        return nn.LeakyReLU(negative_slope=alpha, inplace=inplace)
+
+class ConvBlock(nn.Module):
+    """ Convolutional block with batch normalization
+    """
+    def __init__(self, in_channels: int, out_channels: int, kernel_size: int, stride: int, padding: int):
+        super(ConvBlock, self).__init__()
+        self.conv = nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size, stride=stride, padding=padding)
+        self.bn = nn.BatchNorm2d(out_channels)
+        
+    def forward(self, x: torch.Tensor):
+        return self.bn(self.conv(x))
+
+
+class ResidualBlock(nn.Module):
+    def __init__(self, in_channels, out_channels, skip_conv=True, stride=1, dropout=0.2, activation='leaky_relu'):
+        super(ResidualBlock, self).__init__()
+        self.convb1 = ConvBlock(in_channels, out_channels, kernel_size=3, stride=stride, padding=1)
+        self.act1 = activation_layer(activation)
+
+        self.convb2 = ConvBlock(out_channels, out_channels, kernel_size=3, stride=1, padding=1)
+
+        self.dropout = nn.Dropout(p=dropout)
+        
+        self.shortcut = None
+        if skip_conv:
+            if stride != 1 or in_channels != out_channels:
+                self.shortcut = nn.Conv2d(in_channels, out_channels, kernel_size=1, stride=stride)
+
+        self.act2 = activation_layer(activation)
+        
+    def forward(self, x):
+        skip = x
+        
+        out = self.act1(self.convb1(x))
+        out = self.convb2(out)
+
+        if self.shortcut is not None:
+            out += self.shortcut(skip)
+
+        out = self.act2(out)
+        out = self.dropout(out)
+        
+        return out
+
+class Network(nn.Module):
+    """ Handwriting recognition network for CTC loss"""
+    def __init__(self, num_chars: int, activation: str='leaky_relu', dropout: float=0.2):
+        super(Network, self).__init__()
+
+        self.rb1 = ResidualBlock(3, 16, skip_conv = True, stride=1, activation=activation, dropout=dropout)
+        self.rb2 = ResidualBlock(16, 16, skip_conv = True, stride=2, activation=activation, dropout=dropout)
+        self.rb3 = ResidualBlock(16, 16, skip_conv = False, stride=1, activation=activation, dropout=dropout)
+
+        self.rb4 = ResidualBlock(16, 32, skip_conv = True, stride=2, activation=activation, dropout=dropout)
+        self.rb5 = ResidualBlock(32, 32, skip_conv = False, stride=1, activation=activation, dropout=dropout)
+
+        self.rb6 = ResidualBlock(32, 64, skip_conv = True, stride=2, activation=activation, dropout=dropout)
+        self.rb7 = ResidualBlock(64, 64, skip_conv = True, stride=1, activation=activation, dropout=dropout)
+
+        self.rb8 = ResidualBlock(64, 64, skip_conv = False, stride=1, activation=activation, dropout=dropout)
+        self.rb9 = ResidualBlock(64, 64, skip_conv = False, stride=1, activation=activation, dropout=dropout)
+
+        self.lstm = nn.LSTM(64, 128, bidirectional=True, num_layers=1, batch_first=True)
+        self.lstm_dropout = nn.Dropout(p=dropout)
+
+        self.output = nn.Linear(256, num_chars + 1)
+
+    def forward(self, images: torch.Tensor) -> torch.Tensor:
+        # normalize images between 0 and 1
+        images_flaot = images / 255.0
+
+        # transpose image to channel first
+        images_flaot = images_flaot.permute(0, 3, 1, 2)
+
+        # apply convolutions
+        x = self.rb1(images_flaot)
+        x = self.rb2(x)
+        x = self.rb3(x)
+        x = self.rb4(x)
+        x = self.rb5(x)
+        x = self.rb6(x)
+        x = self.rb7(x)
+        x = self.rb8(x)
+        x = self.rb9(x)
+
+        x = x.reshape(x.size(0), -1, x.size(1))
+
+        x, _ = self.lstm(x)
+        x = self.lstm_dropout(x)
+
+        x = self.output(x)
+        x = F.log_softmax(x, 2)
+
+        return x

Tutorials/08_handwriting_recognition_torch/requirements.txt

@@ -0,0 +1,4 @@
+torch==1.13.1
+tensorboard==2.10.1
+onnx==1.12.0
+torchsummaryX

Tutorials/08_handwriting_recognition_torch/train_torch.py

@@ -0,0 +1,136 @@
+import os
+import tarfile
+from tqdm import tqdm
+from io import BytesIO
+from zipfile import ZipFile
+from urllib.request import urlopen
+
+import torch
+import torch.optim as optim
+from torchsummaryX import summary
+
+from mltu.torch.model import Model
+from mltu.torch.losses import CTCLoss
+from mltu.torch.dataProvider import DataProvider
+from mltu.torch.metrics import CERMetric, WERMetric
+from mltu.torch.callbacks import EarlyStopping, ModelCheckpoint, TensorBoard, Model2onnx, ReduceLROnPlateau
+
+from mltu.preprocessors import ImageReader
+from mltu.transformers import ImageResizer, LabelIndexer, LabelPadding, ImageShowCV2
+from mltu.augmentors import RandomBrightness, RandomRotate, RandomErodeDilate, RandomSharpen
+
+from model import Network
+from configs import ModelConfigs
+
+def download_and_unzip(url, extract_to='Datasets', chunk_size=1024*1024):
+    http_response = urlopen(url)
+
+    data = b''
+    iterations = http_response.length // chunk_size + 1
+    for _ in tqdm(range(iterations)):
+        data += http_response.read(chunk_size)
+
+    zipfile = ZipFile(BytesIO(data))
+    zipfile.extractall(path=extract_to)
+
+dataset_path = os.path.join('Datasets', 'IAM_Words')
+if not os.path.exists(dataset_path):
+    download_and_unzip('https://git.io/J0fjL', extract_to='Datasets')
+
+    file = tarfile.open(os.path.join(dataset_path, "words.tgz"))
+    file.extractall(os.path.join(dataset_path, "words"))
+
+dataset, vocab, max_len = [], set(), 0
+
+# Preprocess the dataset by the specific IAM_Words dataset file structure
+words = open(os.path.join(dataset_path, "words.txt"), "r").readlines()
+for line in tqdm(words):
+    if line.startswith("#"):
+        continue
+
+    line_split = line.split(" ")
+    if line_split[1] == "err":
+        continue
+
+    folder1 = line_split[0][:3]
+    folder2 = "-".join(line_split[0].split("-")[:2])
+    file_name = line_split[0] + ".png"
+    label = line_split[-1].rstrip('\n')
+
+    rel_path = os.path.join(dataset_path, "words", folder1, folder2, file_name)
+    if not os.path.exists(rel_path):
+        print(f"File not found: {rel_path}")
+        continue
+
+    dataset.append([rel_path, label])
+    vocab.update(list(label))
+    max_len = max(max_len, len(label))
+
+configs = ModelConfigs()
+
+# Save vocab and maximum text length to configs
+configs.vocab = "".join(sorted(vocab))
+configs.max_text_length = max_len
+configs.save()
+
+# Create a data provider for the dataset
+data_provider = DataProvider(
+    dataset=dataset,
+    skip_validation=True,
+    batch_size=configs.batch_size,
+    data_preprocessors=[ImageReader()],
+    transformers=[
+        # ImageShowCV2(), # uncomment to show images during training
+        ImageResizer(configs.width, configs.height, keep_aspect_ratio=False),
+        LabelIndexer(configs.vocab),
+        LabelPadding(max_word_length=configs.max_text_length, padding_value=len(configs.vocab))
+        ],
+    use_cache=True,
+)
+
+# Split the dataset into training and validation sets
+train_dataProvider, test_dataProvider = data_provider.split(split = 0.9)
+
+# Augment training data with random brightness, rotation and erode/dilate
+train_dataProvider.augmentors = [
+    RandomBrightness(), 
+    RandomErodeDilate(),
+    RandomSharpen(),
+    RandomRotate(angle=10), 
+    ]
+
+network = Network(len(configs.vocab), activation='leaky_relu', dropout=0.3)
+loss = CTCLoss(blank=len(configs.vocab))
+optimizer = optim.Adam(network.parameters(), lr=configs.learning_rate)
+
+# uncomment to print network summary, torchsummaryX package is required
+summary(network, torch.zeros((1, configs.height, configs.width, 3)))
+
+# put on cuda device if available
+if torch.cuda.is_available():
+    network = network.cuda()
+
+# create callbacks
+earlyStopping = EarlyStopping(monitor='val_CER', patience=20, mode="min", verbose=1)
+modelCheckpoint = ModelCheckpoint(configs.model_path + '/model.pt', monitor='val_CER', mode="min", save_best_only=True, verbose=1)
+tb_callback = TensorBoard(configs.model_path + '/logs')
+reduce_lr = ReduceLROnPlateau(monitor='val_CER', factor=0.9, patience=10, verbose=1, mode='min', min_lr=1e-6)
+model2onnx = Model2onnx(
+    saved_model_path=configs.model_path + '/model.pt', 
+    input_shape=(1, configs.height, configs.width, 3), 
+    verbose=1,
+    metadata={"vocab": configs.vocab}
+    )
+
+# create model object that will handle training and testing of the network
+model = Model(network, optimizer, loss, metrics=[CERMetric(configs.vocab), WERMetric(configs.vocab)])
+model.fit(
+    train_dataProvider, 
+    test_dataProvider, 
+    epochs=1000, 
+    callbacks=[earlyStopping, modelCheckpoint, tb_callback, reduce_lr, model2onnx]
+    )
+
+# Save training and validation datasets as csv files
+train_dataProvider.to_csv(os.path.join(configs.model_path, 'train.csv'))
+test_dataProvider.to_csv(os.path.join(configs.model_path, 'val.csv'))