Microsoft is working on a contextual note-taking app that automatically generates notes just by highlighting parts of the screen

Even though Microsoft has recently updated Sticky Notes to support screenshot captures and linking to sources, it seems that the Redmond-based tech giant has bigger plans for note-takers.

According to a patent published recently, Microsoft is working on a contextual note-taking application that automatically generates notes when users highlight data. Suggestively called Universal Highlighter for Contextual Notetaking, the app would be quite revolutionary in its own right.

The Universal Highlighter for Contextual Notetaking is capable of 4 main tasks:

• Interactive Highlighting
• Contextual Notetaking
• Notes Generation
• Easy Recall

Basically, this technology allows you to highlight a part of your screen. It could be a part of a window or an application. The system then automatically figures out the context of the highlighted content. It gathers relevant data from the content, the window, the application, and the user.

All this information is combined to create notes for you. These notes are stored in a database and can be accessed later using a notetaking application.

As we mentioned before, the contextual note-taking app could exist on its own, but also be part of another platform. These include Web Browser Apps, such as Microsoft Edge, for example, or native Windows apps, such as the Calendar App, but also Microsoft 365 apps, such as Word, Excel, or Outlook.

On this note, Microsoft Edge already has a performant PDF reader that translates text in real-time by just highlighting it, so the note-taking app described in the patent already has a basis.

This Microsoft contextual sort-of Sticky Notes app would make note-taking easier and more efficient. Firstly, it’s all automatic, and the only thing users would need to do is highlight the information, thus it would reduce manual work, and enhance productivity all around.

Secondly, the intriguing part is its contextual way of taking notes: it takes into account everything you have been looking for online when highlighting certain parts of your screen, and it provides notes that are enriched with relevant data.

Interestingly enough, Microsoft is developing this contextual note-taking app to take into consideration not only text content, but image, video, or audio-based content, as well. The app processes these forms of content and generates text-based notes about them.

When the highlighted content includes a dara type that is non-text (e.g., image data, video data, audio data, and the like), the present disclosure processes the highlighted content and generates textualized data. For example, when the highlighted content includes and image of a car, the textual data is “the car” based on an image recognizition. The image recognition may identify a feature of the image data in text. For example, the textual data may firther include “the car as a pictograph” or “the car model X manufactured by Y in year 2022.”

Thirdly, it automatically stores these notes for later use, so you won’t have to worry about losing them. According to the patent, the notes would be compatible with multiple note-taking apps, allowing you to open them with your preferred option.

We could potentially see this kind of update for Sticky Notes, on even Copilot. The latter would make sense, as this contextual note-taking would use AI capabilities to generate notes, and Copilot is already being integrated into many Microsoft products and services.

The full paper can be read here.