Survey phase

[mkassm]

Arabic Manuscripts papers

ICFHR_2018_Competition_on_Recognition_of_.pdf

• Methodology
  • page segmentation then,
  • region classification (layout analysis) then,
  • text line detection then,
  • optical character recognition (OCR)
• Algorithms and technologies
  • Google cloud vision API
  • FCN (Fully Convolutioonal network) for page segmentation
  • RDI company method for text lines and OCR
  • Tesseract OCR
• Dataset
  • British library
  • Qatar digital library (ODL)
• Results/Accuracy
  • RDI was the best in OCR and text recognition

Arabic Word Spotting Based on Key-Points Features

• Word spotting is an efficient alternative to OCR for understanding historical manuscripts by matching features of words. Word spotting classified into two main categories
• learning based (which like OCR where models of keywords are trained using labeled data which are used to recognize queries in the target document)
  • The problem they suffer from the need of large labeled datasets
• template matching based (which matches directly the query image with word images without using labeled data)
• Methodology
  • Use key-points method which capture both topological and local characteristics,
  • feature extracted from key-points and distance for word matching to reduce space matching and connected components.
• Algorithms
  • Gaussian mixture model,
  • Dynamic time warping (DTW)
• Dataset
  • IBEN sina dataset
• Results

Handwritten Arabic Word Spotting Using Speeded Up Robust Features Algorithm

This paper present a method of exploration and research in content of Arabic manuscript images by characterizing segmented handwritten words with a set of points
- Methodology

- Algorithms
SURF invariant local detectors and descriptors

Arabic Word Recognition System for Historical Documents using Multiscale Representation Method

• Methodology
  • Morphological image processing
  • Segmentation
    • Line Segmentation
    • Word Segmentation
  • Multiscale Convexity Concavity (MCC) representation
  • Matching and dissimilarity
  
• Algorithms
  • novel feature extraction to define robust features from Arabic words to segment the words
  • features of each word/sub-word are defined based on multiscale convexity concavity (MCC) analysis of contour word shape.
  • feature matching, a circular shift method instead of traditional dynamic time warping (DTW) which exhibits high computational cost.
• Dataset
  • IBN Sina
• Results
  

Subword Recognition in Historical Arabic Documents using C-GRUs

• Methodology
  • CNN-GRU ⇒ CNN for robust feature representations, CRU for sequence modelling and generate the transcription of the text
  
• Algorithms
  • CNN followed by GRU (Gated Recurrent Units)
• Dataset
  • IBN SINA
  • VML-HD
• Results
  • IBN SINA ⇒ 96.10%
  • VML-HD ⇒ 98.60%

Word Spotting Using Convolutional Siamese Network

• Methodology
  • Use convolutional siamese neural network to learn word image descriptors and compare them for word spotting
  • Out of vocabulary (OOV) word spotting method based on Siamese newtork
• Algorithms
  • Siamese Networks
    
• Dataset
  • VML-HD
  • George Washington dataset
• Code
  • https://github.com/beratkurar/word-spotting-using-siamese-network

Enabling Indexing and Retrieval of Historical Arabic Manuscripts through Template Matching Based Word Spotting

• Methodology
  • Thresholding the image
  • Binarization
  • Noise removal
  • Word image matching using NCC algorithm
• Algorithms
  • Normalized Cross Correlation (NCC) algorithm which are searching algorithm to determine the position of a pattern in an image (similarity measure).
• Dataset
  • HADARA80P dataset
• Results
  • Tested into 4 words only and get 80% of the relevant word location

Historical Arabic Manuscripts Text Recognition Using Convolutional Neural Network

• Methodology
  • Preprocessing: The image is transformed to the binary representation
  • Line segmentation: The binary image is segmented to lines. A line is a set of Arabic words written in a single horizontal line
  • Character segmentation: Each line from step two is segmented to characters of Arabic letter
  • Building the dataset of Arabic characters
  • Recognizing the text (classification) using CNN
    
• Algorithms
  • Preprocessing: convert to binary image
  • Line segmentation: Horizontal Projection Profile (PP) method to convert the image form 2D to 1D
  • Character segmentation: Vertical PP
  • Dataset: dataset contain image for Arabic letters at different states (e.g. start, middle, end and standalone)
  • CNN Classification
    • Input Layer: image that represents a single character with size 30301
    • Convolutional Layers: nonlinear activation function ReLU, first layer has 8 neurons, second 16 neurons, third has 32 neurons
    • Max Pooling Layers: down-sampling operation to reduces the spatial resolution with size 2
    • Fully Connected Layer
    • Output Layer: Softmax for each input and assign the input to one of output classes
• Dataset
  • Three Custom collected dataset from Islamic University in Madinah in the period of 7-8 Hijri centuries
  • First Dataset: has all 28 Arabic letters with average of 40 images per letter
  • Second Dataset: The second dataset has partial letters from the first one. It includes 10 letters, there are 100 images per letter
  • Third Dataset: The third dataset has the same letters presented in the second dataset with average of 200 images per letter
• Results
  • Dataset1 ⇒ 74.29%
  • Dataset2 ⇒ 84.67%
  • Dataset3 ⇒ 88.20%

English Manuscripts papers

• https://github.com/ihdia
• https://paperswithcode.com/paper/digital-peter-dataset-competition-and

General Arabic handwritten papers

[gemydev]

• Arabic handwriting recognition system using CNN

We propose an automatic handwriting recognition model based on CNN. We train our model on Hijja, as well as the AHCD dataset. Results show that our model’s performance is promising, achieving accuracies of 97% and 88% on the AHCD dataset and the Hijja dataset, respectively, outperforming other models in the literature.

• Printed Arabic Script Recognition

  DOC to more details
  This paper has overviewed the main stages used in printed Arabic OCR. Its main aim is to reveal the current status of printed Arabic OCR

  • ARABIC SCRIPT CHARACTERISTICS AND CHALLENGES

    

    The characteristics of Arabic script that may complicate recognition will be discussed:

    • Shapes and Positions
    • Overlapping characters and Ligatures
    • Diacritics
    • Cursive
    • Presence of dots

  • GENERAL ARABIC OCR METHODOLOGY (MODEL)

    

    The process of implementing Arabic OCR consists of five phases:

    1. Preprocessing
    2. Segmentation
    3. Feature extraction
    4. Classification
    5. Post-processing

    Preprocessing Phase

    This is the first phase of OCR methodology which is responsible for enhancing the readability of the input image.
    Generally, several preprocessing operations are employed on the input image: binarization, layout analysis, thinning, smoothing and filtering, size and slant normalization, slant detection, skew detection, and baseline detection.

    Segmentation Phase

    During the segmentation phase, the text image is segmented into small components, with a page being segmented into lines, a line into words, and a word into letters.
    Segmenting a text image can be graded into two types: External segmentation and Internal segmentation
    
    Arabic OCR systems have been developed by two main paradigms:

    1. Holistic approaches (segmentation–free) which require a large lexicon of Arabic words
    2. Analytical approaches (segmentation based) where a word is segmented into units and each unit is recognized separately

    Feature Extraction Phase

    The process of obtaining distinguishing attributes of the segmented character to be utilized by the next phase which is classification.
    The authors point out that the selection of feature extraction methods depends on the output of the preprocessing stage.
    The set of features extracted must match the specification of the selected classifier.
    However, the selection of feature types is a major issue in OCR development.
    Such features can be categorized into three groups: Structural features, Statistical features, and Global transformation features
    In conclusion, the feature extraction stage plays a critical role in Arabic OCR development in which distinguishing attributes are extracted and it is clear that each Arabic OCR developer needs to apply different feature extraction approaches. Still, good features are required, which assist in distinguishing a character from other characters and maximize the accuracy performance simultaneously. Furthermore, these features must be selected specifically for a selected classifier. Some researchers apply different feature extraction methods in combination. However, this may cause extra complications for the implementation.

    Classification Phase

    The classification phase has the responsibility for assigning a pattern into a pre-classified class based on the features of the pattern which have been extracted in the previous phase.
    The pre-classified classes can be words, sub-words, characters, or strokes, based on the OCR approach used. There are a number of different classification approaches that have been applied for Arabic OCR, such as Hidden Markov Models (HMM), Support Vector Machines (SVM), K-nearest neighbor(KNN).

    Post-processing Phase

    Post-processing is the final stage of the development of Arabic OCR
    The objective of this step is to enhance the recognition accuracy by detecting and correcting linguistic misspellings in the produced OCR text without human intervention
    Generally, post-processing methods can be categorized into two main approaches: lexicon-based methods and context-based (statistical) methods

    The typical technique for correcting the mistakes of Arabic OCR outputs is the lexicon-based method which requires the utilization of an Arabic dictionary.
    This technique corrects errors without considering any contextual information in which the errors appear

  • PERFORMANCE EVALUATION

    OCR performance evaluation can be classified into two types:

    • black-box evaluation
    • white-box evaluation

    Performance evaluation of OCR systems is essential for:

    • Monitoring progress of OCR systems development.
    • Assessing the effectiveness of OCR algorithms.
    • Identifying open areas for further research.
    • Providing scientific justification for the performance of OCR systems.

    For Arabic OCR, conducting performance evaluation is challenging as no standard dataset is available.
    This accuracy metric is insufficient to assess how Arabic OCR systems are overcoming the challenges of Arabic text. However, a study suggests a new set of objective performance metrics for evaluation of Arabic OCR with respect to the challenges of Arabic script which are character accuracy based on character position, dot character accuracy, zigzag-shaped character accuracy, loop-shaped character accuracy, and diacritics accuracy.

• OCFormer: A transformer-based Model For Arabic Handwritten Text Recognition

  DOC to more details
  In this paper, we present an OCR approach that utilizes state-of-the-art Deep learning techniques for the Arabic language.
  We built a custom dataset of obfuscated and noisy images to imitate the noise in historical Arabic documents, with a collection of 30 million images paired with their ground truth.
  The model utilizes both page segmentation and line segmentation techniques to enhance the resultant transcription.
  The model is complex enough for transcribing handwritten manuscripts. In addition, the model can detect and transcribe documents that contain Arabic diacritics.

  The model attained a CER of 0.0727, a WER of 0.0829, and an SER of 0.10.

  • Related works
    The paper talked about four different scientific papers dealing with some methods for solving the problem of recognizing Arabic letters, words, data sets, and finally the results of these methods.

  • Dataset
    The constructed dataset consists of images containing Arabic text collected from the web along with their ground truth.

    A portion of the text includes

    • Arabic diacritics.
    • Multiple Arabic fonts that closely resemble the old fonts used in historical manuscripts (dating back to the 18th century) are used.

    There are four categories of images:

    • Full sequences (i.e., images with more than five words)
    • Short sequences (i.e., images that have five or fewer words), Full sequences
    • With diacritics (the images with more than five words with diacritics)
    • Short sequences with diacritics (images with five or fewer words, with diacritics).

    The handwritten manuscripts from the KHATT database are also included (KHATT contains unconstrained handwritten Arabic Texts written by 1000 different writers).

    The figure illustrates the various categories of the dataset.
    

  • Methodology

    • Image Preprocessing

      • Image augmentation
        Shearing and Changing the Luminosity are two commonly applied augmentation approaches when training large neural networks
      • Image Segmentation
        Two types of line segmentation are used. procedures are applied.
        • Layout segmentation is applied, where the model segments areas containing valid text from a page.
        • line segmentation, where the model segments the output from the layout segmentation (i.e., text areas) into segments each corresponding to sentence (i.e., line) detected within the image.
          

    • OCR Model
      we trained a CNN model (Resnet101) with 101 layers as a feature extractor, resulting in the feature vectors that are crucial for the subsequent Transformer Encoder phase.
      The proposed OCR Model utilizes CNNs together with a Transformer.
      Our proposed model is initialized with four encoders, four decoders, four attention heads, and 256 hidden dimensions.

  • Proposed Dataset
    The database is constructed using 13 Arabic fonts and multiple font styles. The dataset contains 30.5 million single-line images, including more than 270 million words and 1.6 billion characters. The ground truth, style, and font used to generate each image are available

  • Proposed Model

• Arabic Islamic Manuscripts Digitization based on Hybrid K-NN/ SVM Approach and Cloud Computing Technologies

  DOC to more details
  Our idea consists on:

  • On the first hand, to consider cloud computing as an infrastructure (IaaS) to deploy our combination of algorithms K-NN/SVM for Arabic Islamic Manuscripts Recognition System AIMRS.
  • On the second hand, to consider cloud Storage as a Service (SaaS) to store and retrieve large amounts of Arabic Islamic Manuscripts.

  Paper components:

  • K-NN
  • SVM
  • Cloud computing
  • Cloud Storage
  • THE PROPOSED APPROACH:
    Our main idea is attending three objectives:
    • increase the recognition rate,
    • speed up the pattern recognition system
    • decrease the cost of management, handling, and archive backup, processing data.
      This approach consists of using SVM as a decision classifier to exceed the limits of K-NN (sensible to different classes with similar attributes)
      
  • Dataset
    To evaluate the proposed approach we have used a corpus with 16000 pages (370 characters/page) randomly chosen from the Islamic Heritage Project (IHP) of, Harvard University.
  • Experimental environment
    This part, explains the hardware specifications and the cloud used.
  • Results
    After making two comparison experiments on both KNN (with K=15) and KNN/SVM:
    • Average of recognition rate:
      • K-NN 95.94%
      • K-NN/SVM 96.92%
    • Time Execution (H):
      • For one computer:
        • KNN >> 7 HOURS
        • KNN/SVM >> 8 HOURS
      • For 100 computers:
        • KNN >> 0.126 HOUSR
        • KNN/SVM >> 0.140 HOURS

• Experiment study on utilizing CNN to recognize historical Arabic handwritten text

  DOC to more details
  This scientific paper talks about the possibility of merging data sets from images of historical Arabic manuscripts and printed Arabic letters and Automatic synthesis of historical Arabic subwords, and building a CNN model and training it on the data.
  EXPERIMENTAL STUDY

  • Automatic synthesis of historical Arabic subwords
  • Generating printed forms of Arabic subwords
    
  • Datasets
    We created three datasets:
    • the historical handwritten text image dataset (HD).
    • the synthesized text image dataset (SD).
    • the printed text image dataset (PD).
      DETAILS OF THE DATASETS
      
  • The architecture
    
  • RESULTS
    

• Toward Classification of Arabic Manuscripts Words Based on the Deep CNN

  DOC to more details
  We propose two methods of word classification based on deep learning,

  • Simple Neural Network (DNN)
  • Convolutional Neural Network (DCNN)
    The idea is to segment words of Arabic manuscripts images and predict the class of each word.

  PROPOSED METHOD:
  Our method aims to classify images by words, in order to help search engines to access the content of these images and to facilitate the recognition of their texts afterward. We use In the first contribution a simple Neural Network (DNN) and in the second one a Convolutional Neural Network (DCNN).
  the sequence of steps:

  • Preprocessing
  • Word Segmentation
  • Neural Networks
  • Convolutional Neural Networks
  • Dataset
    The dataset we used is called HADARA80P that consists of 80 images of a fully handwritten book.
    
    Results:
    We could achieve an accuracy of around 98.42%.
    The model obtained with the DCNN is more efficient than that of the DNN.

[OlaAbdallahM]

• Arabic-manuscript

https://link.springer.com/article/10.1007/s10032-018-0312-3

• ICFHR_2018_Competition_on_Recognition_of_.pdf

• KeyWords

  • Otsu threshold method
  • k-Nearest Neighbor (k-NN)
  • performance evaluation
  • optical character recognition(OCR)

• Preprocessing on dataset
  Different image sizes are created by scaling the image to smaller sizes and no cropping takes place.

First, the dataset was preprocessed to be used with writing style-based features. Text area was segmented and binarized using Otsu threshold method . Afterward, features were extracted by run length, edge hinge and edge direction methods. We evaluated the performance of the features using k-Nearest Neighbor (k-NN) using writing style-based features on KERTAS dataset.

(IRHT) has an online dataset of over 76,000 manuscripts in multiple languages including but not limited to Latin, Hebrew, Greek and Arabic.

• Methodology

  • page segmentation
  • text line detection
  • optical character recognition (OCR)
  • layout analysis

• Dataset

  • KERTAS

Guide_to_OCR_for_Arabic_Scripts

• Pre-processing
  • (binarization,noise reduction, removing the slant angle, and smoothing )
• segmentation algorithm
  • Line Scanning→ The method utilizes a sliding window
  • Filtering
• recognition
• Features for Recognition algorithm.
  -Isolated Characters
  - Word Model Recognizer (WMR)
  - lexicon-based method for recognizing English words
  - Generalized Shape Context (GSC)
  - Dealing with Dots
  - Word Shape Matching
  - Similarity Measure
• Document Search
  • Word Spotting
  • Versatile Search
• Construction of a Handwritten Arabic Corpus
• Layout Analysis of Arabic Script Documents
  • Ridge-Based Text Line Detection Method
  • Anisotropic Gaussian Filter Bank Smoothing
  • Ridge Detection
  • Whitespace Analysis
  • Text Line Labeling
• Glyph Tracer
• PAW Segmenter → splits input word/phrase into PAWs
• HMM
• Features for Sliding-Window Systems

A_Deep_Learning_Based_Prediction_of_Arabic_.pdf

• To predect the Arabic handwriting style

• Keywords:

  • Deep Learning Model
  • Convolutional Neural Network
  • Handwriting Style Prediction

• Methodology

  • main tasks of deep learning as following: detection, classification, segmentation, prediction, and recommendation.
  • four different machine learning classifiers as following: gaussian naive bayes, decision tree, random forest, and the K-nearest neighbor.
  • CNN for automatic feature extraction.
  • a Support Vector Machine (SVM) classifier.
  • K-nearest neighbour algorithm to measure the similarity.

• Dataset

  • Corel dataset

• Font Classification

Methodology

• CNN is trained to classify small patches of text into predefined font classes.
• average the CNN predictions over densely extracted patches to classify page or line images.
• This method achieves state-of-the-art performance on a challenging dataset of 40 Arabic computer fonts with 98.8% line level accuracy.
• This same method also achieves the highest reported accuracy of 86.6% in predicting paleographic scribal script classes at the page level on medieval Latin manuscripts.
• Analyze what features are learned by the CNN on Latin manuscripts and find evidence that the CNN is learning both the defining morphological differences between scribal script classes as well as overfitting to class-correlated nuisance factors.
• propose a novel form of data augmentation that improves robustness to text darkness, further increasing classification performance.
• If text lines are labeled with a font class, then a specialist OCR system for that font can potentially achieve higher recognition rates than an OCR system trained on many fonts.
• At prediction time, we densely extract patches from the test image and average font predictions over individual patch predictions.
• demonstrate state-of-the-art performance with 86.6% accuracy on the Classification of Latin Medival Manuscripts (CLaMM) dataset, where the highest previously reported accuracy is 83.9%
• Though CNNs are black box models, we can gain an understanding of what features are used for classification by measuring output responses as we vary characteristics of the input images.
• Such an analysis can demonstrate whether the CNN is overfitting to nuisance factors of the collection of documents it was trained on.
• CNNs trained on CLaMM are sensitive to how dark the text is.
• Data augmentation improves performance on CLaMM.

Font classification

• Fractal Dimension features for Arabic font classification and resulted in 98% accuracy for 10 font classes.
• Log-Gabor filter features extracted at multiple scales and orientations to obtain 96.1% accuracy on 20 fonts in the large scale KAFD dataset.
• Classifying calligraphy classes is similar to classifying script types in CLaMM, though CLaMM uses Latin script classes with page-level ground truth.

Analysis of Learned Features

• There are a variety of techniques used to examine features learned by CNNs, developed primarily in the context of natural images.
• Canonical class images are obtained by gradient descent over input pixels to maximally excite output class neurons.
• Additionally, they visualize intermediate neurons by finding the top-N maximally exciting input patches for each neuron. For quantitative analysis, Zeiler and Fergus measured how CNN outputs change in response to certain transformations of the input image (e.g. rotation, translation) .
• We perform a similar analysis, specific to the domain of document images, to see how sensitive CNNs are to noise factors in CLaMM.

Methods

• Classify large document images into script or font classes using CNNs. Because inputting entire high resolution images to a CNN is computationally slow, requires large GPU memory, and requires more training data, we resort to a patch classification scheme.

• Train a CNN to classify individual 227x227 patches into font/script classes. To obtain a classification for a large test image, we densely extract overlapping 227x227 patches on a regular square grid with 100 pixels between patch centers.

• The CNN produces a probability distribution over the classes for each patch, which distributions are uniformly averaged to obtain the final classification.

• For training data, extract 256x256 patches at a stride of 42 pixels from the training images and label patches with the class of the image it was taken from.

• During training, a random 227x227 crops from these patches are inputted to the CNN.

• Some training images are set aside as a validation set, which is used to select the best performing model.

• Compare two CNN architectures :

• The first is the AlexNet architecture is composed of 5 convolution layers followed by 3 fully connected layers . Each layer takes as input the output of the previous layer:
  xt = Ft(xt - 1, 0t) → (1)

• The other architecture is the state-of-the-art ResNet-50 , which was used to win the 2015 ImageNet Large Scale Visual Recognition Challenge (ILSVRC). Layers in a ResNet learn a residual function, where Ft is added to the layer input:
  Xf = Fe(Xt- 1, 0t) + Xt → (2)

• Residual learning enables deeper networks to be trained with gradient descent optimization .

• For ResNet-50, Ft is composed of multiple convolutions, ReLU, BatchNorm , and sometimes pooling operations.

• Additionally, we train CNNs at 7 image scales from 30-100%, where 100% is original image size and 50% is down-sampled by a factor of 2.

• For CLaMM experiments, we ensemble networks by averaging predictions of two CNNs trained from different random initializations.

• For training details, all CNNs are trained with Stochastic Gradient De-scent (SGD) using 0.9 momentum and L2 weight decay of 0.0005. The number of iterations was 350K-650K based on architecture and dataset, with mini-batches of size 40-64.

Dataset

• CLaMM
• KAFD

Accuracy of CNNs on KAFD dataset

KAFD classes that are easily accounting for approximately 70% of misclassifications in all trained models.

Results

• KAFD

  • For all types of data, the more powerful Resnet architec-tures outperforms AlexNet, providing 19-58% error reduction. Training and testing on segmented line images leads to the best page level classification at 99.2% accuracy for ResNet. When training and testing on densely cropped patches, the best accuracy is 98.5%, showing that using segmented data leads to a 47% reduction in error over densely cropped patches. The best accuracy achieved over line images is 98.8%.

• CLaMM

• Performance on CLaMM for CNNs trained at difference scales on (a) whole image classification and (b) per-patch classification. -Pre indicates that CNNs were first pretrained on synthetic data (Section III-B). A pretrained ResNet trained at 50% image scale performs best and outperforms the previous best result on this task. Subfigure (c) shows the effect of patch extraction stride at test time. Smaller strides are more important for smaller image scales.

Analyze acquired features

analyze how sensitive a ResNet model is to noise factors present in the CLaMM dataset. We take two approaches.
modify the training data and evaluate the performance of the ResNet on the task images.
take the trained ResNet and measure how changing certain characteristics of input patches affects the classification decision for the patch.
We created three modified training sets with the annotations:

A. Modified Training Data

• CLaMM-Filtered (12 classes) - Non-textual patches are removed from the training set.
• CLaMM-Extended (15 classes) - Non-textual patches are reassigned to one of background, figure, annotation. At test time, predictions for these classes are not allowed
• CLaMM-Noise (13 classes) - All textual patches are removed from the 12 script classes (manually verified),
  leaving only non-textual patches as examples of the

B. Text Darkness

• We also examined learned features by varying input patches and measuring per-patch output accuracy.

• In preliminary anal-ysis, we found two nuisance factors that influence classifica-tion decisions:
  - text darkness and inter-line spacing.

• To test the effect of text darkness on patch-accuracy, we extracted 30 patches of text from each class and computed reasonable foreground masks using Otsu binarization .

• For each patch, we produced a series of 100 patches, where 50 have darker text and 50 have lighter text compared to the original patch.

• To make text darker, we subtracted constant values from all foreground pixels and clipped values at 0. The value of the constant was linearly varied such that the intensity of all foreground pixels in the darkest image are uniformly 0.

• To make the text lighter, we produced linear combinations of the original patch and an estimated background image con-structed by averaging together sampled background patches.

• We classified each modified patch and recorded the average accuracy for each

• To make CNNs robust to varying text darkness, we apply a novel form of data augmentation. each time an image is input to the CNN, we choose either foreground or background with equal probability.

• Then, we draw a random value from a Gaussian distribution with it = 0, a = 30 and add that value to the grayscale pixel values of the selected region.

• In particular, we reach a new record at 86.6% accuracy on CLaMM, which exceeds the previous state-of-the-art by an absolute 2.7%.

C. Line Spacing

• For each class, we manually extracted text lines and backgrounds from two images and were able to render those text lines at various spacings on the original backgrounds.
• We then measured patch-accuracy as a function of text line spacing for each image.
  accuracies stayed above 95% for line spacings greater than 20 pixels.

Conclusion

• We have presented a simple patch based classification framework for line image and page image font classification.

• We have shown that the ResNet architecture in our framework gives state-of-the-art performance by exceeding previously published results in Arabic font classification on the KAFD dataset and in Latin scribal script classification on the CLaMM dataset.

• We performed an analysis of the sensitivities of ResNet to nuisance factors in the CLaMM dataset, such as non-textual patches, text darkness, and line spacing.

• In the case of text darkness, we proposed novel data augmentation based on independently varying foreground and background intensities, which leads to improved model robustness and performance.

• Recognition of Off-line Printed Arabic Text Using Hidden Markov Models

• Different sizes of overlapping and non-overlapping hierarchical windows are used to generate 16 features from each vertical sliding strip.

• Arabic text is cursive, and each character may have up to 4 different shapes based on its location in a word.
• This research work considered each shape as a different class resulting in a total of 126 classes (compared to 28 Arabic letters).
• The achieved average recognition rates were between 98.08% and 99.89% for the eight experimental fonts.
• The main contributions of this work are the novel hierarchical sliding window technique using only 16 features for each sliding window, considering each shape of Arabic characters as a separate class, bypassing the need for segmenting Arabic text.

• Due to the advantages of Hidden Markov Models (HMM) many researchers have used them for Arabic text recognition. HMM offer several advantages.

• HMM was used for off-line Arabic handwritten digit recognition and character recognition.

• The techniques used are based on extracting different types of features of each digit as a whole, not on the sliding window principles used by the majority of researchers using HMM.

• For their technique to be applicable to Arabic text recognition, it has to be preceded by a segmentation step which is error-prone.

• Data preparation

  • each image in the 'tif file has been side-reversed through a mirroring tool to speed up the training and recognition testing processes. The same effect can be done by changing the index so that the window will slide from right to left.

• We prepared two dictionary books for each font. The first one represents the dictionary to be used in training and testing, where we coded each shape of a letter by its unique code. The second one includes coded Arabic characters, using English characters as the Hidden Markov Model Tool does not accept Arabic text as a dictionary.

• Feature Extraction

  • A technique based on the sliding window principle was implemented to extract text features. A window with variable width and height was used. Horizontal and vertical overlapping windows were experimented with.

• Hidden Markov Model (HMM)

  • HTK models the feature vector with mixture of Gaussians. It uses the Viterbi algorithm in the recognition phase which searches for the most likely sequence of a character given the input feature vector.

  • In this paper a left-to-right HMM for our Arabic handwritten text recognition is implemented.

  • This is in line with several research studies using HMM . This model allows relatively large variations in the horizontal position of the Arabic text. The sequence of state transition in the training and testing of the model is related to each text segment feature observations.

• Training and Recognition

  • Training
    • A large number of trials were conducted to find the most suitable combinations of the number of suitable states and codebook sizes.
  • Classification
    • In all cases there are improvements in both correctness and accuracy in combining the different shapes of the character after recognition into one code. The following two equations were used to calculate correctness and accuracy.

• Conclusions

  • This paper presents a technique for automatic recognition of off-line Arabic text recognition based on estimating simple and effective features that are suitable for use with the HMM (which is normally employed for speech recognition). We analyzed the performance of the HMM with different numbers of features, different sizes of sliding windows, different numbers of states and different dictionary sizes.

• ManuscriptsImage_Retrieval_Using_Deep_Learning_Incorporating_a_Variety_of_Fusion_Levels

• Index terms

  • Image retrieval
  • Fusion methods
  • Similarity measurement
  • Deep Learning (DL)
  • Convolutional Neural Networks (CNN)
  • Long Short-Term Memory (LSTM)

• Methodology

  • To accomplish our goal, the images are first classified according to their extracted deep learning visual features utilizing a pre-trained convolutional neural network.

  • Then, the texts written in the manuscripts’ images are extracted and pre-processed to classify the images according to their textual features using an optimized bidirectional LSTM deep learning model with attention and batch normalization layers.

  • Finally, both the visual and textual deep learning models are fused at three different fusion-levels named: decision-level, features-level, and score-level. Extensive experimentation and evaluation of the proposed fusion method on the collected ancient Arabic manuscripts dataset proved its robustness against other stateof-the-art methods recording 99% classification accuracy and 98% mean accuracy on the top-10 image retrieval.

  • Challenges associated with the historical Arabic text-based images:

    • There are many traditional methods for text-based image retrieval, such as word spotting, text queries, segmentation, and OCR.

• Research significance and contributions:

  • 1. Develop visual model by transfer learning from four pretrained deep CNNs.
  • 2. Develop textual model through optimizing the classical LSTM deep learning model.

• classification of image retrieval techniques

  • Markovian Semantic Indexing (MSI) method to do the annotation employed the weighted nearest-neighbor, as well as the multi-class classification techniques, to annotate medical images.

  • semantic-space method using linear algebra to do the annotation softly through training.

  • Methods in relevance feedback technique include delta mean algorithm, standard deviation, variance, query point movement, Bayesian framework, support vector machine, biased discriminant analysis, and kullback-leibler distance.

  • The techniques used for color features are conventional color histogram, fuzzy color histogram, and color correlogram.

  • The techniques used for the texture features are the steerable pyramid, contour let transform, and complex directional filter bank.

  • The techniques used for the shape features are Fourier descriptor, moment invariants, and directional histograms.

1) visual-based hand-crafted features

• Used Speeded-up Robust Feature (SURF) with the Sum of Squared Differences (SSD) measurement technique.

• Used Binary Robust Invariant Scalable Keypoints (BRISK) combined with the Hamming Distance (HD) measurement technique.

• BRISK because it reached 61% accuracy, while BRISK achieved only 37% accuracy.

• The authors used the “ETH” 3D images dataset to evaluate their proposed model.

2) visual-based deep learning features

• They used datasets that include 3D objects. Afterward, the Euclidean function was employed to compare their trained input images with the queried image.
• They extracted the features using a Sketch-Based Local Binary Pattern (SBLBP) algorithm. Then, the authors measured the similarity through computing the intersection between two viewpoints distances and divide the result by the images’ unions.
• The authors recorded 60% precision on the “NTU” dataset and 39% precision on the “3D sketched” images dataset.
• used the ResNet101 deep CNN to retrieve images successfully. Their proposed approach is based on the Multi-Scale Regional Maximum Activation of Convolutions (MS-RMAC) descriptor, which is utilizing the resulting fully connected CNN to extract images features. The authors measured the similarity using the K-NearestNeighbor (K-NN) algorithm to find-out the nearest four images to the original queried image. They evaluated their model by computing the mAP and accomplished 72.3% using the “Oxford5k” dataset, 87.1% using the “Paris6k” dataset, and 94% using the “INRIA Holidays” dataset.

3) textual-based hand-crafted features

• Two datasets were used, which are: WAHD and KHATT. For the preprocessing purpose, the authors cropped the background of the scanner from original images and then segmented the main text.
• collected hand-written Arabic texts in a user-friendly system to use them for testing their new segmentation-based approach for Arabic text recognition. The authors synthesized Arabic words vocabularies and called it "IESK-arDB SynWords". They preprocessed the collected dataset through segmenting its words into individual characters. Afterward, the authors classified the segmented words using both the Support Vector Machine (SVM) and the Active Shape Model (ASM) classifiers. In addition, the authors used the Levenstein distance metric for measuring the similarities. They recommended implementing the work using C/C++ instead of using Matlab. Moreover, the authors expect that implementing the system on the GPU or the FPGA would expedite the work.

4) textual-based deep learning features

• They concluded that the Siamese deep learning model outperformed the other two experimented models since it recorded the highest accuracy as 99.8% using both tested datasets.
• The final classification layer includes Softmax activation function. The authors tested their model on eight benchmark datasets and recorded 94% accuracy.

5) fusion-based image retrieval:

• the authors segmented and recognized the Arabic letters online using an integrated Genetic Algorithm (GA) with a Harmony Search (HS) algorithm.
• First, the dataset is segmented using a dominant point detection and afterward, the segmented text are recognized using the GA-HS fused model.
• The authors use the harmony search character algorithm to be able to measure the similarity among the searched Arabic characters and the characters stored in the dataset.
• Finally, the authors evaluate their proposed GA-HS fusion model for online Arabic characters recognition using two datasets. The model recorded 93.6% successful recognition rate when tested on the first dataset, which is collected manually. While, the second dataset was called ADAB, the model recorded 94.68%-96.33% successful recognition rate when tested on the ADAB dataset.

proposed method

• The proposed method consists of three main steps, which are: visual-based image retrieval, text-based image retrieval, and image retrieval using the fusion model. Considering that any image retrieval system consists of two main steps, which are: the classification and the similarity measurement. Thus, each one of the three main steps are divided further into classification and similarity measurement. The classification section explains classifying the images or text according to the manuscripts’ labels. While the similarity measurement section explains the method of matching the classified images or text to retrieve similar images to a user query image.
  A.visual-based image retrieval

- image classification

- similarity measurement

B. text-based image retrieval

• Binarization_Free_Layout_Analysis_for_Arabic_Historical_Documents_Using_Fully_Convolutional_Networks

• present a Fully Convolutional Network based method for layout analysis of non-binarized historical Arabic manuscripts. The document image is segmented into main text and side text regions by dense pixel prediction. Convolutional part of the network can learn useful features from the nonbinarized document images and is robust to degradation and uncontrained layouts. We have evaluated the proposed method on a private dataset containing challenging historical Arabic manuscripts to demonstrate it effectiveness.

• Page layout analysis is an important pre-processing step for the other document image processing algorithms. The analysis process consists of page segmentation and region classification. Page segmentation segments a document image into homogeneous regions. Region classification classifies the regions into predefined classes such as text, graphic and picture.

• Fully Convolutional Network(FCN) has been already used for page layout analysis on DIVA-HisDB dataset [2] and achieved state of the art results.

• DIVA-HisDB dataset does not cover the full range of difficulties present in the historical documents [3] in comparison to our dataset.

• Binarization algorithms tend to introduce artifacts for historical documents. Therefore, we present an approach that is not dependent on foreground and background pixels.

• Page segmentation algorithms work in either top-down or bottom-up manner. Top-down algorithms segment a whole page into regions. Bottom-up algorithms aggregate elements into regions.

• Primitive page layout analysis algorithms are based on a document structure assumption and applied to modern binary documents in a top-down manner.

• Usage Run Length Smearing Algorithm (RLSA) for page segmentation.

• RLSA links together the neighbouring black areas that are separated less than c pixels.

• RLSA is applied row by row as well as column by column to a document yielding two different bit maps.

• Methods

• Conventional methods for rectangular layout analysis are not proper for segmenting manuscripts with complex layout. Historical Arabic manuscripts usually contain skewed and curved side notes with non-uniform patterns. We used FCN for segmenting side text and main text in Arabic documents with complex layout. FCN has made great improvements in object segmentation field .

• It is an end to end segmentation framework that extracts the features and learns the classifier function simultaneously.

A. FCN architecture
B. Pre-processing

Dataset :

• dataset consists of 38 document images from 7 different historical Arabic books.

Training :

• We train by Stochastic Gradient Descent (SGD) with momentum equals to 0.9 and learning rate equals to 0.001.
• We initialize VGG with its publicly available pre-trained weights.
• We first trained on the randomly sampled dataset until overfitting.
• Then using the output weights we further trained on the dataset of patches with lower density of background labels.
• We continue training on train set until best F1 measure on validation set.
• All the experiments are conducted on Keras and run on a single Nvidia 1080GTX.

D. Results

• We first made an experiment with FCN32. Then we made an experiment with FCN8.

V. Conclusion

• This paper presents a method for page layout analysis of historical Arabic manuscripts. It segments main text and side text regions using FCN. Our method is aimed for historical document images since convolutional part of FCN is robust to degradation and complex layout. FCN8 with skip features from early layers of the network yields to finer segmentation. Number of training samples is an important factor in the prediction performance. Prediction performance on a test sample is related to the number of train samples from the same book of the test sample. Competitive performance on the private Arabic manuscript datasetis achieved which validates the proposed method.

[OlaAbdallahM]

Image_Retrieval_Using_Deep_Learning

Methodology:-

• Utilizing a pre-trained convolutional neural network.
• images are extracted and pre-processed to classify the images according to their textual features using an optimized bidirectional LSTM deep learning model with attention and batch normalization layers.
• both the visual and textual deep learning models are fused at three different fusion-levels named: decision-level, features-level, and score-level.
• Extensive experimentation and evaluation of the proposed fusion method on the collected ancient Arabic manuscripts dataset proved its robustness against other stateof-the-art methods recording 99% classification accuracy and 98% mean accuracy on the top-10 image retrieval.

Methods:-

• word spotting
• text queries
• segmentation
• OCR

Research significance and contributions:-

• Develop visual model by transfer learning from four pretrained deep CNNs.
• Develop textual model through optimizing the classical LSTM deep learning model.

classification of image retrieval techniques:-

• Markovian Semantic Indexing (MSI) method.

• semantic-space method using linear algebra.

• Methods in relevance feedback technique include delta mean algorithm, standard deviation, variance, query point movement, Bayesian framework, support vector machine, biased discriminant analysis, and kullback-leibler distance.

• The techniques used for the shape features are Fourier descriptor, moment invariants, and directional histograms.

• 1) visual-based hand-crafted features

  • Used Speeded-up Robust Feature (SURF) with the Sum of Squared Differences (SSD) measurement technique.
  • Used Binary Robust Invariant Scalable Keypoints (BRISK) combined with the Hamming Distance (HD) measurement technique.
  • SURF outperforms BRISK because it reached 61% accuracy, while BRISK achieved only 37% accuracy.
    The authors used the “ETH” 3D images dataset to evaluate their proposed model.

• 2) visual-based deep learning features

  • Extract the features using a Sketch-Based Local Binary Pattern (SBLBP) algorithm.
  • Measured the similarity
  • precision on the “NTU” dataset recorded 60% and 39% precision on the “3D sketched” images dataset.
    used the ResNet101 deep CNN to retrieve images successfully.
  • They evaluated their model by computing the mAP and accomplished 72.3% using the “Oxford5k” dataset, 87.1% using the “Paris6k” dataset, and 94% using the “INRIA Holidays” dataset.
    - 3) textual-based hand-crafted features
  • Two datasets were used : WAHD and KHATT.
  • called Arabic words vocabularies "IESK-arDB SynWords".Afterward, the authors classified the segmented words using both the Support Vector Machine (SVM) and the Active Shape Model (ASM) classifiers.
    - 4) textual-based deep learning features
  • The final classification layer includes Softmax activation function. The authors tested their model on eight benchmark datasets and recorded 94% accuracy.
    - 5) fusion-based image retrieval:
  • the authors segmented and recognized the Arabic letters online using an integrated Genetic Algorithm (GA) with a Harmony Search (HS) algorithm.
  • The authors use the harmony search character algorithm to be able to measure the similarity among the searched Arabic characters and the characters stored in the dataset.
  • Evaluate their proposed GA-HS fusion model for online Arabic characters recognition using two datasets. The model recorded 93.6% successful recognition rate when tested on the first dataset. While, the second dataset was called ADAB, the model recorded 94.68%-96.33% successful recognition rate when tested on the ADAB dataset.

proposed method:-
- visual-based image retrieval
- text-based image retrieval
- image retrieval using the fusion model.

Layout_Analysis

Methods:-

• Page layout analysis on DIVA-HisDB dataset by --> Fully Convolutional Network (FCN)
• page segmentation by --> Run Length Smearing Algorithm (RLSA)
  • RLSA links together the neighboring black areas, and applied row by row as well as column by column to a document yielding two different bit maps.
• region classification

Dataset :-

• dataset consists of 38 document images from 7 different historical Arabic books.

Training :-

• Train by Stochastic Gradient Descent (SGD) with momentum equals to 0.9 and learning rate equals to 0.001.
• Initialize VGG with its publicly available pre-trained weights.
• First trained on the randomly sampled dataset until overfitting.
• Using the output weights that trained on the dataset of patches with a lower density of background labels.
• Continue training on train set until best F1 measure on validation set.

Results:-

• All the experiments are conducted on Keras and run on a single Nvidia 1080GTX.
• We first made an experiment with FCN32, Then we made an experiment with FCN8.
• Prediction performance on a test sample is related to the number of train samples from the same book of the test sample.
• Competitive performance on the private Arabic manuscript datasetis achieved which validates the proposed method.
• DIVA-HisDB dataset does not cover the full range of difficulties present in the historical documents