Solution Overview
Planview, a leading provider of connected work management solutions, embarked on an ambitious plan in 2023 to revolutionize how 3 million global users interact with their project management applications. To realize this vision, Planview developed an AI assistant called Planview Copilot, using a multi-agent system powered by Amazon Bedrock.
Technical Overview
Planview used key AWS services to build its multi-agent architecture. The central Copilot service, powered by Amazon Elastic Kubernetes Service (Amazon EKS), is responsible for coordinating activities among the various services. Its responsibilities include managing user session chat history using Amazon Relational Database Service (Amazon RDS), coordinating traffic between the router, application agents, and responder, and handling logging, monitoring, and collecting user-submitted feedback.
Router and Responder Sample Prompts
The router and responder components work together to process user queries and generate appropriate responses. The following prompts provide illustrative router and responder prompt templates. Additional prompt engineering would be required to improve reliability for a production implementation.
Model Evaluation and Selection
Evaluating and monitoring generative AI model performance is crucial in any AI system. Planview’s multi-agent architecture enables assessment at various component levels, providing comprehensive quality control despite the system’s complexity. Planview evaluates components at three levels: prompts, AI agents, and AI system.
Results and Impact
Over the past year, Planview Copilot’s performance has significantly improved through the implementation of a multi-agent architecture, development of a robust evaluation framework, and adoption of the latest FMs available through Amazon Bedrock. Planview saw the following results between the first generation of Planview Copilot developed mid-2023 and the latest version:
* Accuracy: Human-evaluated accuracy has improved from 50% answer acceptance to now exceeding 95%
* Response time: Average response times have been reduced from over 1 minute to 20 seconds
* Load testing: The AI assistant has successfully passed load tests, where 1,000 questions were submitted simultaneous with no noticeable impact on response time or quality
* Cost-efficiency: The cost per customer interaction has been slashed to one tenth of the initial expense
* Time-to-market: New agent development and deployment time has been reduced from months to weeks
Conclusion
In this post, we explored how Planview was able to develop a generative AI assistant to address complex work management process by adopting the following strategies:
* Modular development: Planview built a multi-agent architecture with a centralized orchestrator. The solution enables efficient task handling and system scalability, while allowing different product teams to rapidly develop and deploy new AI skills through specialized agents.
* Evaluation framework: Planview implemented a robust evaluation process at multiple levels, which was crucial for maintaining and improving performance.
* Amazon Bedrock integration: Planview used Amazon Bedrock to innovate faster with broad model choice and access to various FMs, allowing for flexible model selection based on specific task requirements.
About Authors
Sunil Ramachandra is a Senior Solutions Architect enabling hyper-growth Independent Software Vendors (ISVs) to innovate and accelerate on AWS. He partners with customers to build highly scalable and resilient cloud architectures. When not collaborating with customers, Sunil enjoys spending time with family, running, meditating, and watching movies on Prime Video.
Benedict Augustine is a thought leader in Generative AI and Machine Learning, serving as a Senior Specialist at AWS. He advises customer CxOs on AI strategy, to build long-term visions while delivering immediate ROI. As VP of Machine Learning, Benedict spent the last decade building seven AI-first SaaS products, now used by Fortune 100 companies, driving significant business impact. His work has earned him 5 patents.
Lee Rehwinkel is a Principal Data Scientist at Planview with 20 years of experience in incorporating AI & ML into Enterprise software. He holds advanced degrees from both Carnegie Mellon University and Columbia University. Lee spearheads Planview’s R&D efforts on AI capabilities within Planview Copilot. Outside of work, he enjoys rowing on Austin’s Lady Bird Lake.
FAQs
Q: What is Planview Copilot?
A: Planview Copilot is an AI assistant developed by Planview using a multi-agent system powered by Amazon Bedrock.
Q: What are the key components of Planview Copilot?
A: The key components of Planview Copilot include the router, responder, and application agents.
Q: How does Planview evaluate the performance of its AI models?
A: Planview evaluates its AI models at three levels: prompts, AI agents, and AI system.
Q: What are the benefits of using Amazon Bedrock for Planview Copilot?
A: Amazon Bedrock provides Planview with broad model choice and access to various FMs, allowing for flexible model selection based on specific task requirements.
Q: What are the results of Planview Copilot’s performance improvement?
A: Planview saw significant improvements in accuracy, response time, load testing, cost-efficiency, and time-to-market.
