Decoding social cues and body language takes time to master. Mostly, we gain knowledge of body language through observation and mimicking behavior as children. Often, we are able to discern if someone is lying about how they feel based on body language. If a friend is acting unlike themselves and when prompted, they respond that they are fine, but their shoulders are slumped, and eyes are red, it’s safe to say that they aren’t okay. Humans are capable of deciphering nonverbal movements, but can a robot do the same? Even robots with advanced computer vision and facial recognition technology have trouble noticing small ticks or body movements, leading to a failure to read social cues.
A resolution to this problem is a body tracking system developed by researchers at Carnegie Mellon University. The system named OpenPose, can track body, hand, and face movement in real time. Utilizing computer vision and machine learning, the system processes video frames, and can even monitor multiple people concurrently. The innovative tech could help simplify human-robot interactions and open the door for more interactive virtual (VR) and augmented reality (AR).
A crucial component of the OpenPose system is its ability to track limbs, head, and torso in addition to individual fingers. To make the system effective, researchers used CMU’s Panoptic Studio, a dome lined with 500 cameras, as they captured body poses in multiple angles and turned those images into data. The images are then run through a keypoint detector that identifies and labels body parts.
In addition to identifying body parts, the software learns to associate specific body parts with certain individuals, making it possible to track multiple people at one time. Originally, the images were captured in 2D, but the researchers altered them in 3D to aid the body-tracking algorithms to discern the differences in each pose. Combining all the data allows the systems to determine how a body part appears when in a particular position.
With all the data plotted, the system can now be run on solely a camera and a laptop. The researchers released the code to the public so that others may begin experiments of their own. This technology has the potential for application in multiple interactions between humans and machine. Researchers believe it could play a drastic role in VR experiences, making way for more subtle detection of physical movement.
