DeepSearcher: An Open-Source Agent for Research and Report Generation
In the previous post, "I Built a Deep Research with Open Source—and So Can You!", we explained some of the principles underlying research agents and constructed a simple prototype that generates detailed reports on a given topic or question. The article and corresponding notebook demonstrated the fundamental concepts of tool use, query decomposition, reasoning, and reflection. The example in our previous post, in contrast to OpenAI’s Deep Research, ran locally, using only open-source models and tools like Milvus and LangChain. (I encourage you to read the above article before continuing.)
DeepSearcher Architecture
The architecture of DeepSearcher follows our previous post by breaking the problem up into four steps – define/refine the question, research, analyze, synthesize – although this time with some overlap. We go through each step, highlighting DeepSearcher’s improvements.
Define and Refine the Question
Break down the original query into new sub-queries:
- "How has the cultural impact and societal relevance of The Simpsons evolved from its debut to the present?"
- "What changes in character development, humor, and storytelling styles have occurred across different seasons of The Simpsons?"
- "How has the animation style and production technology of The Simpsons changed over time?"
- "How have audience demographics, reception, and ratings of The Simpsons shifted throughout its run?"
Research and Analyze
Having broken down the query into sub-queries, the research portion of the agent begins. It has, roughly speaking, four steps: routing, search, reflection, and conditional repeat.
Routing
Our database contains multiple tables or collections from different sources. It would be more efficient if we could restrict our semantic search to only those sources that are relevant to the query at hand. A query router prompts an LLM to decide from which collections information should be retrieved.
Search
Having selected various database collections via the previous step, the search step performs a similarity search with Milvus. Much like the previous post, the source data has been specified in advance, chunked, embedded, and stored in the vector database. For DeepSearcher, the data sources, both local and online, must be manually specified. We leave online search for future work.
Reflection
Unlike the previous post, DeepSearcher illustrates a true form of agentic reflection, inputting the prior outputs as context into a prompt that "reflects" on whether the questions asked so far and the relevant retrieved chunks contain any informational gaps. This can be seen as an analysis step.
Conditional Repeat
Unlike our previous post, DeepSearcher illustrates conditional execution flow. After reflecting on whether the questions and answers so far are complete, if there are additional questions to be asked the agent repeats the above steps. Importantly, the execution flow (a while loop) is a function of the LLM output rather than being hard-coded. In this case, there is only a binary choice: repeat research or generate a report. In more complex agents, there may be several such as: follow hyperlink, retrieve chunks, store in memory, reflect, etc. In this way, the question continues to be refined as the agent sees fit until it decides to exit the loop and generate the report. In our Simpsons example, DeepSearcher performs two more rounds of filling the gaps with extra sub-queries.
Synthesize
Finally, the fully decomposed question and retrieved chunks are synthesized into a report with a single prompt.
Results
Here is a sample from the report generated by the query "How has The Simpsons changed over time?" with DeepSeek-R1 passing the Wikipedia page on The Simpsons as source material:
Report: The Evolution of The Simpsons (1989-2022)
- Cultural Impact and Societal Relevance
The Simpsons debuted as a subversive critique of American middle-class life, gaining notoriety for its bold satire in the 1990s. Initially a countercultural phenomenon, it challenged norms with episodes tackling religion, politics, and consumerism. Over time, its cultural dominance waned as competitors like South Park and Family Guy pushed boundaries further. By the 2010s, the show transitioned from trendsetter to nostalgic institution, balancing legacy appeal with attempts to address modern issues like climate change and LGBTQ+ rights, albeit with less societal resonance.
… Conclusion
The Simpsons evolved from a radical satire to a television institution, navigating shifts in technology, politics, and audience expectations. While its golden-age brilliance remains unmatched, its adaptability—through streaming, updated humor, and global outreach—secures its place as a cultural touchstone. The show’s longevity reflects both nostalgia and a pragmatic embrace of change, even as it grapples with the challenges of relevance in a fragmented media landscape.
Discussion
We presented DeepSearcher, an agent for performing research and writing reports. Our system is built upon the idea in our previous article, adding features like conditional execution flow, query routing, and an improved interface. We switched from local inference with a small 4-bit quantized reasoning model to an online inference service for the massive DeepSeek-R1 model, qualitatively improving our output report. DeepSearcher works with most inference services like OpenAI, Gemini, DeepSeek, and Grok 3 (coming soon!).
Conclusion
DeepSearcher is a significant step forward in the development of research agents, and we are excited to share it with the community. We believe that this project has the potential to democratize research and reporting, making it more accessible to a wider range of people. We will continue to iterate on this work in future posts, examining additional agentic concepts and the design space of research agents. In the meantime, we invite everyone to try out DeepSearcher, star us on GitHub, and share your feedback!
FAQs
Q: What is DeepSearcher?
A: DeepSearcher is an open-source agent for performing research and writing reports.
Q: What is the architecture of DeepSearcher?
A: The architecture of DeepSearcher follows a four-step process: define/refine the question, research, analyze, and synthesize.
Q: What are the improvements of DeepSearcher over the previous post?
A: DeepSearcher adds features like conditional execution flow, query routing, and an improved interface.
Q: How does DeepSearcher generate reports?
A: DeepSearcher generates reports by synthesizing the fully decomposed question and retrieved chunks into a report with a single prompt.
Q: What are the limitations of DeepSearcher?
A: DeepSearcher is limited by its reliance on LLMs and online inference services, which may not always be available or reliable.
