Business Challenge
The exploration and methodology described in this post addresses two key challenges: costs associated with generating a ground truth dataset for multiclass classification use cases can be prohibitive, and conventional approaches and synthetic dataset creation strategies for generating ground truth data are inadequate in generating balanced classes and meeting desired performance parameters for the real-world use cases.
Ground Truth Data Generation is Expensive and Time Consuming
Ground truth annotation needs to be accurate and consistent, often requiring massive time and expertise to ensure the dataset is balanced, diverse, and large enough for model training and testing. For a multiclass classification problem such as support case root cause categorization, this challenge compounds many fold.
Conventional Techniques to Get Balanced Classes or Synthetic Data Generation Have Shortfalls
A balanced labeled dataset is critical for a multiclass classification use case to mitigate bias and make sure the model learns to accurately classify all classes, rather than favoring the majority class. If the dataset is imbalanced, with one or more classes having significantly fewer instances than others, the model might struggle to learn the patterns and features associated with the minority classes, leading to poor performance and biased predictions.
Solution: Amazon Bedrock
To address this challenge, we can use Amazon Bedrock, a generative AI solution that can play an invaluable role during the model development phase by simplifying training and test data creation for multiclass classification supervised learning use cases. We can use XML tags to structure the prompt and guide Amazon Bedrock in generating a balanced label dataset with high accuracy.
Real-World Example: Predicting Root Cause Category for Support Cases
This use case, solvable through ML, can enable support teams to better understand customer needs and optimize response strategies. For instance, let’s say the task at hand is to predict the root cause categories (Customer Education, Feature Request, Software Defect, Documentation Improvement, Security Awareness, and Billing Inquiry) for customer support cases.
Classification Examples
Here are some classification examples:
- Software Defect: The support case is for ETL Pipeline Performance Degradation where the customer reports their nightly data transformation job takes 6 hours to complete instead of 2 hours before but no changes to configuration occurred. The agent mentions Engineering confirmed memory leak in version 5.1.2 and are deploying a Hotfix indicating this is a Software Defect.
- Customer Education: The agent tells the customer that the timeout is caused by misconfiguration and needs to be restricted using filters. The agent provides documentation explaining how to troubleshoot the issue.
- Documentation Improvement: The agent tells the user they have to change the action configuration and define an Action type attribute.
About the Authors
- Sumeet Kumar is a Sr. Enterprise Support Manager at AWS leading the technical and strategic advisory team of TAM builders for automotive and manufacturing customers.
- Andy Brand is a Principal Technical Account Manager at AWS, where he helps education customers develop secure, performant, and cost-effective cloud solutions.
- Tom Coombs is a Principal Technical Account Manager at AWS, based in Switzerland. In Tom’s role, he helps enterprise AWS customers operate effectively in the cloud.
- Ramu Ponugumati is a Sr. Technical Account Manager and a specialist in analytics and AI/ML at AWS.
