Building a Slack Platform Expert AI: A Journey of Learning and Creation
Key System Components
The system is built using several key components, including:
- Streamlit for the web interface, providing a dynamic and user-friendly feel
- ChromaDB for semantic search
- SQLite3 with BM25 ranking algorithm for keyword searches
- OpenAI’s chat.completions API for generating responses
- GitHub APIs for publicly available quality content for Q&A
- Docker for ensuring a consistent environment and Fly.io for deployment and scale adjustments
- Sentry for monitoring and logging errors for continuous improvement
Each component was chosen to balance quality, performance, cost efficiency, and ease of development.
Keeping the AI Grounded
To reduce the risk of hallucinations, the system prompt explicitly instructs the AI to base its answers solely on the provided in-context documents and the session’s past messages. For example, the system prompt includes:
"Your answer must consider only the # Context section data and the past messages in the same session. Please be particularly precise when discussing code related to SDKs. You must not respond with any speculations like suggesting any classes and methods that are not mentioned in the # Context section."
Hybrid Retriever Architecture
The system uses a hybrid retriever architecture to retrieve the most relevant documents for each question. This involves combining two search layers and applying my own ranking logic:
- ChromaDB handles semantic search
- SQLite3 (using BM25 ranking algorithm) manages traditional keyword searches
The process of building the search index involves:
- Fetching publicly available documents, Q&A, code examples, and unit tests from GitHub
- Optimizing some of the data and storing it in a single JSON data file (source_data.json)
- Inserting all the data into ChromaDB and SQLite3 as part of the Docker image
Building Search Index within Docker Image
Both search systems run locally within the Docker container, ensuring low latency and cost efficiency. The databases are built when the Docker image is created.
Abuse Prevention & Cached Responses
To prevent abuse, the system implements a simple rate-limiting mechanism. This involves tracking request frequency using a local SQLite3 database and temporarily blocking users who send too many requests in a short time. This is achieved through a simplified code snippet:
def detect_frequent_requests(ip_address: str, block_again: bool = True):
# ...
if is_this_user_blocked(last_accessed, last_blocked):
if block_again is True:
save_as_blocked(ip_address)
st.error("Thanks for asking questions! but I'm unable to answer many questions at once. Please try again later.")
st.stop()Simple Abuse Prevention & Cached Responses
To reduce system load and minimize costs, a simple cache layer is added. This caching mechanism stores responses to frequently asked questions, reducing the need for AI retrieval and CPU time consumption.
Sign in with Google
The system allows logged-in users to perform resource-intensive generation tasks, helping to manage costs and prevent system abuse. The is_logged_in() function is used to check if a user is logged in and display the login/logout link on the right side of the header.
Conclusion
I’m happy with the system’s capabilities, but I’m aware it’s far from perfect. I plan to improve it by exploring other available solutions for better service quality and adding more documents to cover broader topics.
FAQs
Q: What is the purpose of the system?
A: The system is designed to provide expert-level answers to Slack platform-related questions.
Q: What are the key components of the system?
A: The system consists of Streamlit, ChromaDB, SQLite3, OpenAI’s chat.completions API, GitHub APIs, Docker, and Fly.io.
Q: How does the system prevent abuse?
A: The system implements a simple rate-limiting mechanism to prevent frequent requests from a single IP address.
Q: Can I use the system to develop my own Slack app?
A: Yes, the system can be used as a reference for building your own Slack app.
Q: What are the future plans for the system?
A: I plan to improve the system by exploring other available solutions for better service quality and adding more documents to cover broader topics.
