Recognition of facial expressions using OpenCV and Haar classifiers

Recognition of facial expressions using OpenCV and Haar classifiers

Arteaga M. B., Buenaventura J., Chimbo J., Vaca P.

DECEM- Department of Energy and Mechanical Sciences, University of the Armed Forces ESPE

Sangolquí, Ecuador

mbarteaga1@espe.edu.ec

jrchimbo1@espe.edu.ec

gpvaca@espe.edu.ec

jmbuenaventura@espe.edu.ec

Abstract. - Recognizing human facial expression and emotion by computer is an interesting and challenging problem. Identification of facial feature points plays an important role in many facial image applications including human computer interaction, video surveillance, face detection, face recognition, facial expression classification, face modeling and face animation. In this paper we present a system for recognizing emotions through facial expressions displayed. The aim of this project is detection, analysis and recognition of facial features. This paper presents a method based of three main components, face detection, facial expresión feature extraction and facial expresión categorization. For the analysis during the process was used a Haar classifier. The system localizes characteristic points of analyzed face and, based on their displacements certain emotions can be automatically recognized.

Keywords—Emotion Recognition, Facial Expression Recognition (FER), Computer Vision, Haar Features, OpenCV.

1. INTRODUCTION

Effective computer image analysis was always a great challenge for many researchers. Tasks, usually quite simple for humans, such as object or emotion recognition proves to be very complicated in computer analysis. Among the main problems are susceptibility to varying lightning conditions, color changes and differences in transformation. Effective detection of human faces is one of the greatest problems in image analysis. Therefore, it is even more challenging to efficiently and effectively localize features of a face in analyzed image and to relate them to expression of emotions. Hereby work is actually an attempt to create a computer system able to automatically detect, localize and recognize facial features.  Sought features are, in this case, characteristic points placed in selected locations on human face model. The locations and distances between them change during facial expressions. There are many, more or less effective solutions capable of detecting or recognizing faces, however only a few comprehensive and effective solutions exist connecting all those features together and, at the same time, able to cooperate with an emotion recognition system. [1]

This work describes a real-time automatic facial expression recognition system using video or webcam input. Our work focuses on initially detecting the human face in the video stream, on classifying the human emotion from facial features and on visualizing the recognition results.

1. RELATED WORK

1. PROPOSED METHODOLOGY

The system consists of 2 main parts generalized: the detection of the face and classification of these.

[pic 1]

Fig.  1. Parts System [7]

1. Face Detection

Face detection is the first stage which is desired to be automated. In most of the research, face is already cropped and the system starts with tracking and feature extraction. In others, vision-based automated face detectors or pupil tracking with infrared (IR) cameras are used to localize the face. Alternatively, a face detector can be used to detect the faces in a scene automatically. [6]

Some free face detection softwares are available to researchers for usage and improvement. Most popular of these is the face detector of Open Source Computer Vision Library (OpenCV). This face detector depends on Haar-like wavelet-based object detection proposed by Viola and Jones and improved by Lienhart et al [7,8]. Their algorithm makes three main contributions:

• The use of integral images.
• A selection of features through a boosting algorithm

(Adaboost).

• A method to combine simple classifiers in a cascade

structure.

Each frame is processed firstly through Haar classifiers [9] trained for profile faces. To further improve frame rate and compensate for pose variation. We propose to use interleaved Haar Classifiers. Interleaving is done between front and profile classifiers.

[pic 2]

Fig.  2  Intervaled Classifiers [9]

1. Integral Images

Analyzing images is not an easy task. Using just the pixel information can be useful in some fields (i.e. movement detection) but is in general not enough to recognize a known object. In 1998, Papageorgiou et al [10] proposed a method to analyze image features

using a subgroup of Haar-like features, derived from

the Haar transforms. This subgroup was extended

later by Lienhart et al [11] to also detect small rotations

of the sought-after object. The basic classi-

fiers are decision-tree classifiers with at least 2 leaves.

Haar-like features are the input to the basic classifiers

and are calculated as described below. The algorithm

we are describing uses the Haar-like features shown

in figure 3.

[pic 3]

Fig.  3  Haar features [9]

The feature used in a particular classifier is specified by its shape (1a, 2b, etc), position within the region of interest and the scale (this scale is not the same as the scale used at the detection stage, though these two scales are multiplied). For example, in case of the third line feature (2c) the response is calculated as the difference between the sum of image pixel under the rectangle covering the whole feature (including the two white stripes and the black stripe in the middle) and the sum of the image pixels under the black stripe multiplied by 3 in order to compensate for the differences in the size of areas. Calculating sums of pixels over rectangular regions can be very expensive in computational terms, but this problem can be solved by using an intermediate representation of the images, namely integral images. [9]

...