Neuroscience Knowledge Exploration Framework
The Indian Institute of Technology Madras – IIT Madras Brain Centre is using generative AI to build applications that can deliver life-saving impacts. Advancing neuroscience research, the IIT Madras Brain Centre is using AI to generate analyses of whole human brains at a cellular level across various demographics.
Neuroscience Knowledge Exploration Framework
The knowledge exploration framework leverages neuroscience publications to help researchers link brain imaging data with the latest neuroscience research. With this tool, researchers can explore recent advancements related to brain images and discoveries in particular brain regions, such as the causes of specific conditions seen in the imaging data. They can also track the current state of any neuroscience research area and find answers to related queries.
Processing Pipeline
The framework’s processing pipeline consists of two parts:
- Ingestion: Indexes the latest neuroscience publications into the knowledge base.
- Q&A: Enables users to interact with the knowledge base using queries.
Visual Question Answering and Multimodal Retrieval
Users can interact with the framework using images of brain regions and ask questions about the displayed image. The framework employs the latest VQA models for biomedical domains, such as Llava-Med, to provide answers. Additionally, the framework enables the retrieval of similar images based on a given image or text.
Using NVIDIA Technology to Overcome Research Challenges
The NVIDIA technology stack powers the processing pipeline of the knowledge base framework. Various NVIDIA tools and frameworks have been used to ensure the robustness and performance of this pipeline.
Improving Retrieval Accuracy
The framework includes a specialized knowledge base centered on neuroscience publications. Since generic embedding models weren’t originally trained on this kind of data, fine-tuning was needed to improve retrieval accuracy. Manually creating a fine-tuning dataset at scale is challenging and requires input from neuroscience experts, so a synthetic dataset was generated with a large language model (LLM). To support large-scale dataset development, fast LLM inference is essential; the Mixtral 8x 7B NVIDIA NIM microservice was used to boost inference speed. Fine-tuning the embedding model improved the retrieval accuracy of the top two results by 15.25%. Retrieval accuracy was further enhanced with NVIDIA NeMo Retriever, a set of NIM microservices for information retrieval.
User Input Filtering
To ensure only relevant content reaches users, researchers at IIT Madras used NVIDIA NeMo Guardrails for filtering. They implemented a user input guardrail using the Llama Guard 2 8B language model and developed a custom prompt tailored to neuroscience. This prompt was tested with a public toxic chat database to assess its ability to block irrelevant questions and evaluated with neuroscience-specific questions to confirm it accepted relevant ones. Results showed:
- 38% of toxic content was blocked by the default prompt
- 68% of toxic content was blocked by the custom prompt
- 98% of neuroscience-specific questions were accepted by the custom prompt (based on a custom dataset)
Visual Question Answering Examples
- Example 1: What is the brain region in the image? Answer: The brain region in the image is the cerebellum, which is a part of the brain that plays an important role in motor control, coordination, and balance.
[Image: cerebellum]
- Example 2: Does this image show the frontal cortex? Answer: Yes, the image shows the frontal cortex, which is a region of the brain.
[Image: frontal cortex]
Image-to-Image Retrieval Examples
- Example 1: Figure 5 shows two microscopic images of tissue sections, side-by-side. The left image shows an input tissue sample with a purple stain and some structural detail, while the right image shows a retrieved, similar-looking tissue sample with comparable staining and shape, demonstrating image-to-image retrieval.
[Image: tissue section]
- Example 2: Figure 6 shows two microscopic images of brain tissue sections, side-by-side. The left image shows an input sample stained in purple, with distinct areas of light and dark textures and several elongated, lighter streaks. The right image shows a retrieved tissue sample with a similar overall shape, staining, and texture patterns, used to demonstrate image-to-image retrieval.
[Image: brain tissue]
Summary
The IIT Madras Brain Centre and NVIDIA accelerated computing and AI technologies—including NVIDIA NeMo, NVIDIA NIM, NVIDIA AI Blueprints, and NVIDIA DGX—are helping to advance neuroscience research, opening new avenues for understanding brain structure and function, and accelerating research that could lead to life-saving discoveries.
FAQs
Q: What is the goal of the IIT Madras Brain Centre’s knowledge exploration framework?
A: The goal is to help researchers link brain imaging data with the latest neuroscience research and explore recent advancements related to brain images and discoveries in particular brain regions.
Q: What NVIDIA technologies are used in the framework?
A: Various NVIDIA tools and frameworks have been used, including NVIDIA NeMo, NVIDIA NIM, NVIDIA AI Blueprints, and NVIDIA DGX.
Q: How does the framework improve retrieval accuracy?
A: The framework includes a specialized knowledge base centered on neuroscience publications, and fine-tuning was needed to improve retrieval accuracy. Manually creating a fine-tuning dataset at scale is challenging and requires input from neuroscience experts, so a synthetic dataset was generated with a large language model (LLM).
