AI-Powered Search Assistant: Microsoft’s Copilot Vision

Decades ago, if you wanted information on anything, you would go to the library and open a book. That changed with the emergence of the web and search engines, where now all you have to do is type in a search query and get all the information you could ever want. As if that wasn’t easy enough, artificial intelligence (AI) is here to make information gathering even more hands-off.

Introducing Copilot Vision

Earlier this month, Microsoft launched Copilot Vision, an experience that allows Copilot to view and understand the context of what you’re doing online to provide verbal real-time assistance. The idea is that when you need feedback or advice while browsing, you can tap into a live assistant for help.

Hands-on Experience

I got early access to Copilot Vision and put it to the test. The experience lives in Microsoft Edge and is available in preview for a select group of Copilot Pro subscribers with the $20-per-month subscription through Copilot Labs and US Copilot Pro subscribers on Windows.

Use Cases

In theory, searching the web is pretty self-explanatory, so getting help with it may seem superfluous. However, when I went through the onboarding demo experience, I got pretty excited, as the applications seemed genuinely useful. Example use cases included having several pictures on my screen and asking Copilot Vision to help sort through them. In one instance, there were many pictures of various dog breeds, and I was prompted to ask Copilot Vision to tell me more about them verbally. The assistant reviewed each picture and told me more about each breed — despite there being no text on the screen.

Limitations

However, when I started using it on my own websites, I was slightly disappointed. Copilot Vision can access a limited number of sites, including Wikipedia, Tripadvisor, Amazon, Target, OpenTable, Wayfair, Food & Wine, Williams Sonoma, and GeoGuessr. The majority of these are shopping websites, and my experience was that there isn’t much meaningful assistance that can happen when doing online shopping.

Security Concerns

Naturally, with having an AI model looking at your screen comes concerns about it looking at your data. To address these, Microsoft has a robust Q&A that answers people’s most burning questions. According to the company, Copilot’s responses are only logged to monitor unsafe interactions, but user inputs, including text, images, and contexts, are never stored. Furthermore, a user’s input data is deleted once a session ends.

Is it Worth it?

Right now, getting Copilot Pro just for Copilot Vision would not be worth it, especially because access is not guaranteed. However, with the $20-per-month subscription, users also gain access to other perks, such as priority access to the latest models and Copilot in select Microsoft 365 apps, including Word, Excel, PowerPoint, OneNote, and Outlook. This makes it a fun bonus to tinker with if you’re already a Copilot power user and could benefit from these other perks.

Frequently Asked Questions

Q: What is Copilot Vision?
A: Copilot Vision is an AI-powered search assistant that views and understands the context of what you’re doing online to provide verbal real-time assistance.

Q: What are the limitations of Copilot Vision?
A: Copilot Vision can access a limited number of sites, including Wikipedia, Tripadvisor, Amazon, Target, OpenTable, Wayfair, Food & Wine, Williams Sonoma, and GeoGuessr. It may not be very helpful for online shopping, but shows promise for research and summarizing content.

Q: Is Copilot Vision secure?
A: Yes, Microsoft has a robust Q&A that addresses people’s most burning questions about security. Copilot’s responses are only logged to monitor unsafe interactions, and user inputs are never stored.