Microsoft's latest patent generates hyperreal virtual avatars based on users' faces

Microsoft has a thing with virtual avatars: its recently-released Mesh platform lets Teams users create avatars to engage in so-called virtual immersive spaces, which are contexts where employees can hang out together, virtually, even though they are miles away apart.

Microsoft Mesh is indeed a fun platform, and the Redmond-based tech giant says it’s working: many Microsoft employees feel a sense of bonding while hanging out there. But at the same time it feels off: the cartoon-like avatars, while cute and everything, seem displaced.

However, it seems that the Redmond-based tech giant might already be working on an update. The company recently published a patent that describes a technology capable of generating hyperreal virtual avatars based on users’ faces.

The technology is called Multi-modal Three-Dimensional Face Modeling and Tracking for Generating Expressive Avatars and describes a computer system that creates expressive avatars using 3D face modeling and tracking.

The computer system has a processor and a storage system of its own, which processes and keeps data needed to model the hyperreal virtual avatar. It also keeps instructions on how to render them.

The processor then receives initialization data, which is the starting look of a face model, and additional multi-modal data signals, which can be audio data (such as the voice of the user), that will be used to create a hyperrealist model of the face.

Multimodal 3D face modeling and tracking techniques can utilize multiple different sensor devices, each providing one or more input signals and/or measurements for a user’s face to detect, model, track, and/or animate a three-dimensional face model graphically as an avatar.

Microsoft

The initialization and multi-modal data are then taken by the system and are processed together in a fitting process. This process adjusts the data to fit the model. Based on the fitting process, the system then determines a set of parameters, which are used to describe the hyperreal virtual avatar.

3D face modeling and tracking techniques createe 3D vertices based on a user’s face and apply transformations to the vertices from a neutral face to depict expressions on a digital face model (e.g., an avatar representation of the user’s face).

Microsoft

The system uses deep learning to come up with a detailed virtual avatar that looks, acts, and is capable of facial expressions similar or identical to the user’s face. Deep learning is a form of AI technology that seeks to emulate the human brain as much as possible, and Microsoft has been investing in it over the last few years.

The framework utilizes deep learning techniques and forward modeling to perform a parametric fitting process that translate the multi-modal data signals into a set of parameteres or expression code that can be used to generate an expressive 3D facial model.

Microsoft

Since this is a computer system, the paper mentions the ability to integrate it on a plethora of devices, from VR/AR/MR headsets to mobile phones, laptops, gaming consoles, tablet computers, and many more, meaning users could have a device ready to transport their virtual avatars into a Meta-like or Mesh-like virtual space.

Microsoft might seek to implement this new hyperreal virtual avatar system in existing platforms, such as Microsoft Teams, Microsoft Mesh, or even Windows.

Even though virtual spaces already exist, they are not that popular, for now, due to their limiting capabilities, but a system like this one can encourage more people to join virtual spaces.

The gaming industry could also benefit from this technology, allowing users to customize their characters based on their looks, providing a highly personalized gaming experience, as well.

The full paper can be read here.