Microsoft's new Copilot Labs will make the AI model see everything you're browsing and reason with you about it

Microsoft is rolling out a new program called Copilot Labs, which will allow Copilot Pro users to test upcoming features before they launch to the public.

Copilot Labs is a response to early feedback from Copilot Pro users who wanted to help shape the direction of Copilot and get early access to features. The program will allow users to try experimental features and share their thoughts with Microsoft.

In the future, Copilot Labs will feature more experimental tools, including Copilot Skills, which will allow users to teach Copilot new abilities. In addition to Copilot Labs, Microsoft is launching a new feature called Copilot Vision for Copilot Pro users in the U.S. that assists you in finding information you’re looking at online.

In a blog post, Microsoft says Copilot Vision is accessible by clicking a Copilot icon in the Microsoft Edge toolbar. When you click, you can highlight a part of a webpage, and Copilot will analyze the selection and provide information. For example, if you highlight a paragraph of text on a Wikipedia page, Copilot can summarize the content, answer questions about it, and more.

You can see a preview of it in the video below:

Copilot Vision can also answer questions about images. If you select a photo of a landmark, for example, Copilot can help identify it, provide a brief overview, and more. If you highlight text in multiple languages, Copilot can translate it on the fly.

Microsoft uses machine learning to avoid returning false or misleading information and designed Copilot Vision with privacy in mind. Copilot Vision doesn’t store content, audio, images, or conversations with Copilot, and it’s blocked on sites with paywalls and sensitive content. Still, if we’re thinking about Windows Recall, this might not be enough to encourage users to use capability.

The Redmond-based tech giant is starting with a few hundred sites in the U.S., including Wikipedia, Yelp, Reddit, and others, and will expand Copilot Vision to other countries in the coming months.

Per the announcement, Copilot Labs is now available in preview. Copilot Vision has started rolling out to a subset of users in the U.S. If you’re a Copilot Pro subscriber, you can expect to see Copilot Vision in the coming weeks.

Copilot Labs will also allow users to try the Think Deeper feature, which allows Copilot to reason through more complex problems. Using the latest reasoning models, Think Deeper can help with anything from solving challenging math problems to weighing up the costs of managing home projects.

Think Deeper will roll out to a limited number of Copilot Pro users in the Americas, Asia, Europe, and Oceania.

In other news, Microsoft Copilot has a new interface and new Voice Modes, which are gradually rolling out worldwide.

You can read more about Copilot Labs here.