Microsoft plans changes based on feedback for Bing Chat

It’s been a whirlwind week since Microsoft’s surprise announcement that it was incorporating a version of OpenAI’s ChatGPT into Bing and Edge. Since then, there’s been lots of interest, some well reported quirks and failings. Microsoft’s new “Copilot for the web” has shown that it’s definitely in its early days, but has also shown a lot of promise.

In a report on the Bing Blog after a week of limited testing in over 169 countries, Microsoft has both been expecting that it has work to do, but also pleasantly surprised an the levels of engagement. The company reports at 71% positive rating based on the thumbs up / thumbs down rankings built into the Chat, and have seen Bing Chat used not only for searching the web, but for “more general discovery of the world, and for social entertainment. This is a great example of where new technology is finding product-market-fit for something we didn’t fully envision.”

Sometimes those long engaging sessions lead to problems, though. Microsoft reports that sometimes “Bing can become repetitive or be prompted/provoked to give responses that are not necessarily helpful or in line with our designed tone.” Because of this, the company plans to make it easier to refresh the context or start from scratch. Microsoft also indirectly addresses some of the reported issues with Bing Chat sort of going off the rails, offering sometimes nonsensical, angst-ridden, or generally out of character answers:

The model at times tries to respond or reflect in the tone in which it is being asked to provide responses that can lead to a style we didn’t intend.This is a non-trivial scenario that requires a lot of prompting so most of you won’t run into it, but we are looking at how to give you more fine-tuned control.

Microsoft also has found that it faces challenges in providing both up to the minute and also direct factual answers:

You are giving good marks on the citations and references that underly the answers in Bing. It makes it easier to fact check and it provides a nice starting point to discover more. On the other hand, we are finding our share of challenges with answers that need very timely data like live sports scores. For queries where you are looking for a more direct and factual answers such as numbers from financial reports, we’re planning to 4x increase the grounding data we send to the model.  Lastly, we’re considering adding a toggle that gives you more control on the precision vs creativity of the answer to tailor to your query.

Microsoft fully acknowledges that this Copilot for the web is in its very early stages, and that there’s a lot of work yet to do:

The only way to improve a product like this, where the user experience is so much different than anything anyone has seen before, is to have people like you using the product and doing exactly what you all are doing. We know we must build this in the open with the community; this can’t be done solely in the lab. Your feedback about what you’re finding valuable and what you aren’t, and what your preferences are for how the product should behave, are so critical at this nascent stage of development.

We’ve been using Bing Chat here, not in full testing mode, but just as casual users like many of you will be. First, it is engaging! It’s fun and somewhat addicting to start a chat with Bing and go down some pretty fun and interesting rabbit holes. These chats aren’t meant to find “10 blue link” answers to begin with, but it’s fascinating to chat with a bot that is personable and fun. Next, the Bing Chat Sidebar for Edge (I’m using Edge Dev, it is apparently rolling out to some with production Edge if you’ve been accepted into the preview) is becoming a very useful tool. I just asked Bing Chat for the spelling of a word, and got a straight simple answer without leaving the editor and wading through links or navigating to a search page. I’ve done this for more complex searches too, and it’s a very useful way to gather information.

Bing Chat needs to get better at what it does, but at the same time it has the potential to be game changing. We’re very intrigued so far. Have you been interacting with Bing’s Copilot for the web? Let us know in the comments below.