Microsoft patent: Generate a virtual self based on your human body

Microsoft has recently filed a patent describing a technology that will allow people to generate their virtual selves based on their human bodies. The technology creates a visual representation of the human body that can be used further in other applications and virtual spaces.

By mapping the user’s real-world movements to their virtual articulated representations, the immersion of the human user and any other participants in the virtual experience may be improved.

For example, just recently, Microsoft announced that Mesh, an immersive virtual space allowing users to co-exist with others, will come to Teams in October.

Could this sort of technology be implemented in Teams? Undoubtedly so, and as the patent shows, it’s only a matter of time until users can create a hyper-realistic virtual self.

With the resurgence of AR/VR headsets with the announcement of the Apple Vision Pro, Microsoft seems interested in developing technologies for the medium. Microsoft Mesh is a testament to that.

Generating a virtual human body: How does it work?

1. The technology receives data about a human user’s position of different body parts. This data is collected from sensors.
2. It keeps some rules (mapping constraints) that connect a model representation to a target representation.
3. It estimates the pose of both the model and the target at the same time, using the position data and the mapping constraints.
4. The technology also uses a machine that has been trained to optimize poses. This machine was trained using positioning data with correct model representation labels.
5. Finally, it shows the target representation with its pose as a virtual representation of the human user.

Where can Microsoft use the virtual human body technology?

Aside from the previously mentioned applications, Microsoft could use this technology (Microsoft Mesh in Teams), the technology can be used for various other situations:

• Video Games: It will create more realistic and immersive experiences. The player’s movements can be tracked and replicated in the game in real-time.
• Fitness and Rehabilitation: The technology can track and analyze a user’s exercises or movements. This can help provide feedback on form, track progress, or guide rehabilitation exercises.
• Virtual Reality (VR) and Augmented Reality (AR): The technology can be used to map the user’s movements to a virtual avatar (more realistic Teams avatars), making the virtual experience more immersive and interactive.
• Animation and Film Industry: Animators can use this technology to capture human movements for more realistic animations or special effects.
• Teleconferencing: In virtual meetings, it can create a more realistic representation of participants, enhancing communication. Microsoft Teams meetings are the perfect opportunity for this technology.
• Robotics: It can be used in robotics to guide the movements of a robot based on human movements.

This technology can be a game changer in the industry, allowing people to create a hyper-personalized virtual experience. Is Microsoft going to release this technology? Very likely so.