Uma arquitetura de redes neurais convolucionais para reconhecimento de ações humanas em vídeos utilizando ritmos de fluxo óptico

Abstract

The human action recognition problem based on video classification has significantly benefited from the introduction of deep learning models. However, this is still an open problem due to the inherent difficulty in developing a general and robust solution. In this domain, addressing the temporal aspect of the videos is crucial in order to build realistic models. A key step in this direction is the selection of features that characterize the complexity of human actions. With this goal, we propose a solution for the video classification problem based on a multi-stream deep convolutional neural network architecture. A multi-stream is an interesting way to aggregate information from different sources with a lower training cost if compared to other methods. A new stream called optical flow rhythm was incorporated to improve accuracy rates. Our experiments suggest that optical flow rhythm is complementary to other streams, such as RGB, optical flow and visual rhythm, boosting the results of the approach. To combine the various streams in our architecture, we introduce a new weighted average fusion scheme where the weights of the classifiers are defined by a simulated annealing metaheuristic. Furthermore, we propose a training and test protocol based on the extraction of multiple planes of the visual and optical flow rhythms. The results indicate a performance augmentation using this protocol. Experiments conducted on the challenging UCF101 and HMDB51 datasets demonstrate that our method is comparable to state-of-the-art approaches.