A framework for combined recognition of actions and objects

Abstract

This paper proposes a novel approach to recognize actions and objects within the context of each other. Assuming that the different actions involve different objects in image sequences and there is one-to-one relation between object and action type we present a Bayesian network based framework which combines motion patterns and object usage information to recognize actions/objects. More specifically our approach recognizes high-level actions and the related objects without any body-part segmentation hand tracking and temporal segmentation methods. Additionally we present a novel motion representation based on 3D Haar-like features which can be formed by depth color or both images. Our approach is also appropriate for object and action recognition where the involved object is partially or fully occluded. Finally experiments show that our approach improves the accuracy of both action and object recognition significantly.