Microsoft's InstructDiffusion will edit your images on your instructions

Microsoft’s latest AI model, Instruct Diffusion, will radically transform your images, or any image that you can upload, according to your instructions. The model, developed by Microsoft Research Asia, is an interface that brings together AI and human instructions to generate and complete a variety of visual tasks.

In other words, you choose an image that you want to edit, change, or transform, and InstructDiffusion will bring about its computer vision to change the image based on your input.

Microsoft released the paper for the model a few days ago, and InstructDiffusion already has a demo playground, where you can try the model for yourself.

The key innovation in IntructDiffusion is that the model doesn’t need prior knowledge of the image, but instead, it uses a diffusion process to manipulate pixels. The model is capable of a lot of useful features such as segmentation, keypoint detection, and restoration. Practically, InstructDiffusion will use your instructions to change the image.

In one example, Microsoft Research Asia was able to remove the watermark on a photo, by simply instructing the model to do so.

Microsoft’s InstructDiffusion is able to distinguish the meaning behind your instructions

InstructDiffusion, like many other Microsoft AI models, is capable of innovative behavior when it comes to solving tasks. Microsoft Research Asia claims that InstructDiffusion implements understanding tasks and generative tasks.

The model will use understanding tasks, such as segmentation and keypoint detections to locate the area and pixels that you want it to edit.

For example, the model uses segmentation to successfully locate the area of your following instruction: paint the man at the right of the image red. For keypoint detections, an instruction would be: use yellow to encircle the knee of the man on the far left of the image.

The generative tasks are made up of editing and restorative tasks. Not only InstructDiffusion will edit your image, but the model will also generate new elements for the image, based on your instructions.

Microsoft InstructDiffusion’s most promising feature is its ability to successfully generalize all the instructions it receives to form a cohesive and deep understanding of the meaning behind them. In other words, the model will remember the instructions you gave to it, and it will successfully use them to train itself even further.

But the model will also learn to distinguish meanings behind your instructions, leading it to solve unseen tasks and come up with new ways to generate elements. This ability to understand semantic meanings places InstructDifussion a step further than the other similar models: it outperforms them.

However, InstructDiffusion is also a step further to reaching AGI: By deeply understanding the semantic meaning behind every instruction, and being capable of successfully generalizing computer visions, the model will greatly advance AI development.

Microsoft Research Asia allows you to try it in a demo playground, but you can also use its code to train your own AI model.

What are your opinions on this model? Will you try it?