New Bing Chat Capabilities: Powerful Picture Comprehension Now Available for Users

Bing Chat , ChatGPT , and other similar generative AI tools have mostly been focused on understanding text content and providing answers. However, it’s also incredibly helpful to understand pictures, and that’s what Microsoft is now rolling out to its Bing Chat AI.

In a recent blog post, Microsoft announced a new feature for Bing Chat, called Visual Search. With Visual Search, you can upload an image or select an existing image on the web, and Bing will try to understand it and use that context in responses. Microsoft’s demo video (embedded below) shows off someone uploading a hand-drawn mockup of a web form, and asking Bing to generate HTML and CSS code to make it functional.

Microsoft explained in the blog post, “Whether you’re traveling to a new city on vacation and asking about the architecture of a particular building or at home trying to come up with lunch ideas based on the contents of your fridge, upload the image into Bing Chat and use it to harness the web’s knowledge to get you answers.”

Software that can detect the content of images is nothing new. Google Lens has been able to identify people, animals, plants, landmarks, and other objects in photos since 2017, and its more limited predecessor Google Goggles dates back to 2010. Microsoft is using the image detection features of GPT-4 , which is the same language model used by the premium version of ChatGPT. However, image inputs are not available in ChatGPT yet, so this is the first time we’re seeing that feature widely available.

I tried Visual Search with a few photos, and the results were impressive. Asking Bing to describe images gives you much more detailed responses than you’d get from Google Lens. For example, uploading a photo of my dog responded with, “This is a photo of a black and tan dog sitting on a fluffy brown rug. The dog is wearing a red collar with a silver tag. The dog is looking up at the camera with its ears perked up. The background consists of a white couch with blue and white pillows. The photo is taken from a high angle.” It also correctly interpreted that the image was uploaded as sideways.

You can try out the feature by opening Bing Chat in Microsoft Edge (Microsoft still blocks it in other browsers) and clicking the camera outline icon in the text field.

Source: Microsoft