A visual similarity-based method for children’s handwriting recognition

Abstract

Automatic transcription of words handwritten by children can support the process- ing of large scale literacy assessments. However, the transcription must preserve the exact character sequence written by the child, including spelling errors that provide information about literacy development. This dissertation investigates different Handwritten Text Recognition (HTR) architectures for recognizing handwriting produced by children. To conduct the experiments, a dataset of words written by Brazilian children during literacy assessments was constructed and annotated. The comparison of different architectures showed that some architectures are not well suited to the specific requirements of this task. The results indicated that the E-TrOCR architecture with a vocabulary defined at the character level is more suitable for preserving the original response. Another challenge is the visual ambiguity between letters in children’s handwriting. To address this problem, a method that modifies the E-TrOCR loss function was proposed. The new loss function teaches the model to consider visual similarities between characters. To represent these relationships, a visual similarity matrix was constructed. The values in this matrix were calculated using Histogram of Oriented Gradients (HOG) descriptors extracted from synthetic images rendered in handwriting-style fonts. The method was evaluated using an exclusive words protocol and increased accuracy from 72.57% to 78.06% and reduced the Character Error Rate (CER) from 7.61% to 5.78%, contributing to more faithful transcriptions. These results provide evidence that considering visual similarity between characters during training improves the transcription of words written by children without relying on dictionaries, linguistic rules, or text corpora.