HKUST ECE Ph.D. Student Didan Deng Creates Top Ranked Multi-Task Emotion Recognition System
The global market for automated emotion detection and recognition is projected to grow by a compound annual growth rate of 21%, up to over 50 billion US dollars by 2024, with the largest growth occurring in the Asia Pacific region. Emotion recognition systems are finding applications in a wide range of different areas, including retail, education and medicine.
One of the common ways humans estimate a person’s emotional state is through their facial expressions. Computers can do the same using pictures taken of a person’s face. The development of automated video-based systems for emotion recognition is an important research topic. However, there is no single universally accepted method for describing facial expressions. Each method or descriptor has its own advantages and disadvantages. This has led to a proliferation of many different systems each producing a different label. The development of a single system capable of simultaneously labelling a facial image with more than one emotion descriptor has been hindered, because most datasets contain facial images labelled with only one emotion descriptor.
HKUST ECE Ph.D. student Didan Deng has broken through this barrier with the development of a multi-task video-based emotion recognition system that can label a single facial image simultaneously with the three most common emotional descriptors in use today: facial action units, valence-arousal and discrete expressions. Her system is trained using a combination of three single-task datasets. In other words, even though her system sees images labelled with one descriptor at a time, it can combine this information to produce three simultaneous descriptors when presented with a new image. Didan’s system won First Place at the Multi-Task Challenge of the Affective Behavior Analysis in-the-Wild (ABAW) Competition held at the IEEE Conference on Computer Vision and Pattern Recognition held in New Orleans, LA, USA in June 2022, outperforming the next best system by nearly 20 percent. To achieve such remarkable performance, Didan, an engineering student, exploited prior knowledge about emotions that she had gleaned from pouring over the psychological literature on emotions. Prof. Bertram Shi, Didan’s thesis advisor, commented that “This is an excellent example of the importance of interdisciplinary training and research.”
1st Place Award Certificate of the 3rd Affective Behavior Analysis in-the-Wild (ABAW) Competition
Title and Part of the paper
