How to Use Ollama via a Firefox Extension so You Don’t Have to Use the Command Line

The idea of querying a remote LLM makes my spine tingle — and not in a good way. When I need to do a spot of research via AI, I opt for a local LLM, such as Ollama.

How to Install the Page Assist Extension in Firefox

To make this work, you’ll need Ollama installed and running, as well as the Firefox browser. That’s it. Let’s make some magic.

1. Open Firefox and point it to the Page Assist entry in the Add-Ons store and click "Add to Firefox." When prompted, click Add.
2. You’ll notice the Add-Ons store lists Page Assist as not actively monitored for security by Mozilla. Because of that, I tracked down the source for the extension, which is hosted on GitHub. Since the source is available to download and view, I didn’t hesitate to install it.

How to Use Page Assist

Before you actually use Page Assist, you need to ensure that Ollama is running. If you’ve already installed it, you can run a local LLM with a command like this:

ollama run llama3.2

If you see the >>> prompt, the LLM is running and ready to accept queries.

Using Page Assist

1. Pin the extension: Click the puzzle piece icon and then click the gear icon associated with Page Assist. From the drop-down, click "Pin to toolbar." You should now see the Page Assist icon in the Firefox toolbar (which looks like a tiny thought balloon).
2. Open the extension: Click the Page Assist icon, and a new tab will open with the Ollama UI.
3. Select your model: Click the "Select a Model" drop-down and choose the model you’ve installed (such as llama3.2:latest).
4. Type your query: You can now type your query in the bottom section labeled "Type a message." Hit Enter on your keyboard or click Submit, and Ollama will go to work.
5. Adding a new model: You can also add new models with Page Assist. To do that, click the gear icon in the upper right of the Page Assist window. Click Manage Models in the left sidebar, and then click Add New Model. In the pop-up window, type the name of the model you want to add and click Pull Model.

Conclusion

And that, my friends, is how you can more easily interact with Ollama, thanks to a simple-to-use Firefox extension.

FAQs

Q: Is the Page Assist extension safe to use?
A: While the extension is not actively monitored for security by Mozilla, the source code is available to download and view, and no one has complained of malicious activity from the extension.

Q: Can I use Page Assist on multiple platforms?
A: Yes, the Firefox extension works on all three platforms: MacOS, Linux, and Windows.

Q: Can I add multiple models to Page Assist?
A: Yes, you can add as many models as you like, but do take note of the file size of each, as some can get very large.

Employing Tricks During Inference to Improve AI Accuracy

One of the big trends in artificial intelligence in the past year has been the employment of various tricks during inference — the act of making predictions — to dramatically improve the accuracy of those predictions.

Chain-of-Thought and Breakthroughs in Accuracy

For example, chain-of-thought — having a large language model (LLM) spell out the logic of an answer in a series of statements — can lead to increased accuracy on benchmark tests. Such "thinking" has apparently led to breakthroughs in accuracy on abstract tests of problem-solving, such as OpenAI’s GPTo3’s high score last month on the ARC-AGI test.

LLMs Fall Short on Practical Tests

However, it turns out that LLMs still fall short on very practical tests, something as simple as planning a trip. Google DeepMind researchers, led by Kuang-Huei Lee, pointed out in a report last week that Google’s Gemini and OpenAI’s GPTo1, the companies’ best respective models, fail miserably when tested on TravelPlanner, a benchmark test introduced last year by scholars at Fudon University, Penn State, and Meta AI.

Introducing Mind Evolution

Given the weak results of top models, Lee and team propose an advance beyond chain-of-thought and similar approaches that they say is dramatically more accurate on tests such as TravelPlanner. Called "mind evolution," the new approach is a form of searching through possible answers — but with a twist.

How Mind Evolution Works

The authors adopt a genetically inspired algorithm that induces an LLM, such as Gemini 1.5 Flash, to generate multiple answers to a prompt, which are then evaluated for which is most "fit" to answer the question. In the real world, evolution happens via natural selection, where entities are evaluated for "fitness" in their environment. The most fit combine to produce offspring, and occasionally there are beneficial genetic mutations.

Evaluating AI Model’s Multiple Answers

The point of such an evolutionary approach is that it’s hard to find good solutions in one stroke, but it’s relatively easy to weed out the bad ones and try again. As they write, "This approach exploits the observation that it is often easier to evaluate the quality of a candidate solution than it is to generate good solutions for a given problem."

Author-Critic Dialogue

The key is how best to evaluate the AI model’s multiple answers. To do so, the authors fall back on a well-established prompting strategy. Instead of just chain-of-thought, they have the model conduct a dialogue of sorts. The LLM is prompted to portray two personas in dialogue, one of which is a critic, and the other, an author. The author proposes solutions, such as a travel plan, and the critic points out where there are flaws.

Results

The Gemini 1.5 Flash is tested on multiple planning benchmarks. On TravelPlanner, Gemini with the mind evolution approach soars above the typical 5.6% success rate to reach 95.2%, they relate. And, when they use the more powerful Gemini Pro model, it’s nearly perfect, 99.9%.

Conclusion

The results show "a clear advantage of an evolutionary strategy" combining both a search for possible solutions very broadly speaking, and also using the language model to refine those solutions with the author-critic roles.

FAQs

Q: What is mind evolution?
A: Mind evolution is a form of searching through possible answers using a genetically inspired algorithm that induces an LLM to generate multiple answers to a prompt, which are then evaluated for which is most "fit" to answer the question.

Q: How does mind evolution work?
A: Mind evolution works by inducing an LLM to generate multiple answers to a prompt, which are then evaluated for which is most "fit" to answer the question. The process is repeated, and the LLM is forced to modify its output to be better, a kind of recombination and mutation as seen in natural selection.

Q: What are the benefits of mind evolution?
A: The benefits of mind evolution include improved accuracy on practical tests, such as planning a trip, and the ability to evaluate the quality of a candidate solution more easily than generating good solutions for a given problem.

Q: What are the limitations of mind evolution?
A: The limitations of mind evolution include the need for more computing power and the potential for the approach to be computationally expensive.

Operator: The AI Agent that Simulates Keyboard and Mouse Clicks

OpenAI is introducing a research preview called Operator, an AI agent that simulates keyboard and mouse clicks in a browser, reading the screen, and performing actions. This technology has the potential to revolutionize the way we interact with websites and automate repetitive tasks.

How Operator Works

Operator uses a model called CUA (Computing-Using Agent), which dictates how the AI talks to websites. Unlike other AI models, Operator doesn’t use APIs or extract text from the DOM. Instead, it views an actual web page in a live browser running in the cloud, reading the context directly off the screen.

Partnerships and Limitations

Operator has partnered with several websites, including Instacart, DoorDash, Etsy, OpenTable, Tripadvisor, AP, Priceline, StubHub, Thumbtack, Target, Uber, and more. These partnerships are unclear, but they may be affiliate deals, agreements to notify Operator of website changes, or additional modeling for those sites. Until we have a better understanding of these partnerships, we won’t know the scope of what Operator can do.

Guardrails and Privacy

OpenAI has given serious consideration to issues of privacy and guardrails. Operator knows when to pause and ask for human intervention, and it can also be controlled by the human user. Additionally, the human user can opt out of allowing website interactions to be used as training data for the AI.

Site-Specific Custom Instructions

Operator allows users to create site-specific custom instructions on a site-by-site basis. This can be useful for tasks that require specific settings or preferences, such as booking a hotel room with a free breakfast.

Baby Steps

Operator feels like baby steps at this time. For example, I’d love to tell an AI to go through my inbox and find all the press releases and assign them to one label. This is both a complex task and one that’s got quite a long runtime. As such, it’s way beyond the scope of what Operator can do.

Conclusion

Operator is an interesting technology that has the potential to automate repetitive tasks and make our lives easier. However, it’s still in its early stages, and there are limitations to what it can do. As the technology develops, we can expect to see more advanced capabilities and a wider range of applications.

FAQs

Q: What is Operator?
A: Operator is an AI agent that simulates keyboard and mouse clicks in a browser, reading the screen, and performing actions.

Q: How does Operator work?
A: Operator uses a model called CUA (Computing-Using Agent), which dictates how the AI talks to websites. It views an actual web page in a live browser running in the cloud, reading the context directly off the screen.

Q: What are the limitations of Operator?
A: Operator is still in its early stages, and there are limitations to what it can do. It may not be able to perform complex tasks or handle long-running tasks.

Q: Is Operator secure?
A: OpenAI has given serious consideration to issues of privacy and guardrails. Operator knows when to pause and ask for human intervention, and it can also be controlled by the human user. Additionally, the human user can opt out of allowing website interactions to be used as training data for the AI.

Q: Can I use Operator for free?
A: No, Operator requires a Pro account, which costs $200 per month. However, users of the $20-per-month Plus plan will eventually be able to use Operator.

Imagine an AI bot that can fill out online forms, book airline flights, order groceries, and more. That’s the intent of OpenAI’s new Operator, an AI that acts as an independent agent to carry out your commands all on its own.

Released as a research preview on Thursday, Operator is able to interact directly with a web browser. That means it can navigate web pages by typing, scrolling, and clicking in all the right spots, just as you would yourself. The difference here is that Operator aims to do all that without any intervention on your part.

### How to try ChatGPT Pro

Beyond its initial research preview status, the tool is now accessible only with ChatGPT Pro subscriptions in the US, which cost $200 a month. As the AI evolves and learns from its mistakes, OpenAI plans to expand its reach to Plus, Team, and Enterprise users and eventually integrate its skills directly into ChatGPT.

ChatGPT Pro users who want to take Operator for a spin should browse to its dedicated web page. Make sure you’re signed in with your OpenAI account. From there, type a request at the prompt just as you normally would with ChatGPT. Only you’ll want to fashion that request as one that asks Operator to carry out tasks on the web independently.

### What to expect with Operator

As cool as all this may sound, there are certainly potential pitfalls and problems. Depending on the complexity of the task, Operator could get stuck or make a mistake along the way. In that case, it will try to self-correct. If that doesn’t work, the tool will hand control back to you to intervene.

Operator is also unable to handle confidential information, such as passwords, payment details, and CAPTCHA challenges. If it runs into a website requiring a login or payment card, it will ask you to take over.

### Conclusion

OpenAI’s new Operator is an AI that can interact with a web browser to carry out tasks independently. While it has the potential to be very useful, there are certainly potential pitfalls and problems that need to be addressed. With its ability to navigate web pages and perform tasks, Operator could save people time on everyday tasks and open up new engagement opportunities for businesses.

### FAQs

Q: What is Operator?
A: Operator is an AI that can interact with a web browser to carry out tasks independently.

Q: How do I try Operator?
A: ChatGPT Pro users can try Operator by browsing to its dedicated web page and typing a request at the prompt.

Q: Is Operator available to everyone?
A: No, Operator is currently only available to ChatGPT Pro subscribers in the US, and will be expanded to other users and integrated into ChatGPT in the future.

Q: Can Operator handle confidential information?
A: No, Operator is unable to handle confidential information, such as passwords, payment details, and CAPTCHA challenges.

Q: How does Operator prevent cybercriminals and hackers from exploiting and abusing it?
A: Operator is designed to refuse harmful requests, block prohibited content, detect and ignore prompt injections, and have a built-in monitor that looks out for suspicious behavior.