Data augmentation of visual rhythms using symmetric extension for deep learning video based human action recognition

Abstract

Despite the significant progress of Deep Learning models on the image classification task, they still need enhancement for efficient Human Action Recognition. Such gain could be achieved through the augmentation of the existing datasets. With this goal, we propose the usage of multiple Visual Rhythm crops, symmetrically extended in time and separated by a fixed stride. The premise to augment the temporal dimension of the Visual Rhythms is that the direction of video execution does not discriminate several actions. Besides that, we propose to use the Weighted Visual Rhythm: its extraction method attempts to reinforce motion patterns by weighing the closest aspects of a specific video position in which the action typically occurs. Therefore, we replicate the Weighted Visual Rhythm by concatenating, along the temporal dimension, as many as necessary copies of it, having the even copies horizontally flipped. While providing the possibility of extracting crops matching the fixed input size of the Convolutional Neural Network employed, the symmetric extension preserves the video frame rate, which is crucial to not distort actions. In addition, multiple crops with stride ensure the coverage of the entire video. Therefore, the main contributions of this work are a new form of extracting the Visual Rhythm and a new method for performing the data augmentation of video samples. Aiming to evaluate our method, a multi-stream strategy combining RGB and Optical Flow information is modified to include two additional spatiotemporal streams: one operating on the horizontal Symmetrically Extended Visual Rhythm, and another operating on the vertical Symmetrically Extended Visual Rhythm. Accuracy rates close to the state of the art were obtained from the experiments with our method on the challenging UCF101 and HMDB51 datasets.