Microsoft might add the UFO, a highly customizable AI assistant, to its next Windows

Microsoft recently released UFO, a highly customizable AI assistant, capable of fulfilling users’ requests tailored to different operating systems, including Windows.

The AI assistant is based on and uses the capabilities of GPT-vision to visualize and understand various visual elements, including graphical user interface (GUI) and control information of Windows applications, and it can provide Windows users with additional assistance without the need to have direct audio input.

In other words, UFO is a special kind of computer program that helps users interact with other programs on their Windows computers. It uses a clever system to understand what’s happening on their screen and can perform tasks for them, like clicking buttons or typing text. UFO can do all this automatically, without needing any human input.

The AI tool was developed by a team of researchers working for Microsoft Research, and the paper can be read in its entirety here.

The abstract reads:

We introduce UFO, an innovative UI-focused agent to fulfill user requests tailored to applications on Windows OS, harnessing the capabilities of GPT-Vision. UFO employs a dual-agent framework to meticulously observe and analyze the graphical user interface (GUI) and control information of Windows applications. This enables the agent to seamlessly navigate and operate within individual applications and across them to fulfill user requests, even when spanning multiple applications. The framework incorporates a control interaction module, facilitating action grounding without human intervention and enabling fully automated execution. Consequently, UFO transforms arduous and time-consuming processes into simple tasks achievable solely through natural language commands. We conducted testing of UFO across 9 popular Windows applications, encompassing a variety of scenarios reflective of users’ daily usage. The results, derived from both quantitative metrics and real-case studies, underscore the superior effectiveness of UFO in fulfilling user requests. To the best of our knowledge, UFO stands as the first UI agent specifically tailored for task completion within the Windows OS environment.

Microsoft Research

Not only Microsoft UFO doesn’t need human input to work, but it can also be customized to each user, meaning the assistant can be highly personalized to fit the needs of each Windows user, and it can be automatized to run certain tasks without having to explicitly let it know.

This goes hand in hand with the idea of Windows coming alive, something Microsoft might be interested in exploring in the next version of Windows, which is reportedly AI-based.

Microsoft has also made the open-source code for UFO available on GitHub, and you can find it here.

What do you think? Would you like to have UFO as your Windows assistant? Let us know in the comments section below.