Generative AI for Customer Service – Lessons Learned

Generative AI (GenAI) has the potential to revolutionize customer service. According to McKinsey, GenAI can enhance productivity by up to 45% in customer engagement scenarios. However, most early GenAI projects in customer service have failed to meet operational cost reduction and customer experience goals. Gartner Research predicts that by 2025 (we are almost there), 100% of all virtual assistant projects (in customer service) lacking integration with modern knowledge management systems will fail to meet their objectives!

This white paper explores the lessons learnt from early GenAI projects in customer service automation, focusing on the foundational need for knowledge management. Drawing insights from recent industry research and practical experiences, this paper aims to provide actionable recommendations for leveraging GenAI effectively.

Working with clients, we see five common challenge patterns faced by early GenAI projects in customer service.

1. Multiple Silos of Knowledge and Content feed GenAI tools: The effectiveness of GenAI is contingent upon the quality and relevance of the knowledge content it is fed, especially in customer service where the answers to questions need to be specific to the business, their products and offerings. Silos result in multiple (and inconsistent) input without a common framework to verify and establish content credibility. As a result, GenAI output cannot be trusted.
2. Lack of Comprehensive Prompt Management Capability: Effective prompt management is essential for obtaining valuable output from GenAI. Best-practice prompts guide GenAI to generate responses that align with business needs. A robust prompt management service acts like a supervisor for a new hire, ensuring that GenAI receives clear, actionable instructions. A modern knowledge system includes such a capability with a library of best practice prompts that can be easily configured to address the specific and evolving needs of the business.
3. Rudimentary Content Compliance and User Experience Controls: GenAI tools must operate within defined and auditable business constraints. This involves setting up controls to prevent inappropriate use of knowledge, such as excluding compliance-heavy content from GenAI processing. Without fine-grained controls, it is impossible for GenAI to deliver trusted answers at scale – across different customer segments, product lines, and service channels. Furthermore, when controls do kick in during a customer interaction, we see the lack of seamless step-down capabilities (for example, how to carry the conversation forward when GenAI responses cannot be used in the dialog for compliance reasons) leads to awkward experiences that frustrate customers and, worse yet, fan social media flames!
4. Poor Quality Assurance of GenAI Output: GenAI can sometimes produce incorrect or irrelevant outputs, a phenomenon known as “hallucination.” Without configurable and reliable quality assurance pipelines to verify GenAI responses in real-time, maintaining accuracy and relevance has emerged as a common pitfall in early GenAI projects. A few wrong answers are too many in customer service interactions, especially when customers are acutely aware of the tool’s AI origin. Building this capability is hard and early GenAI projects under-invested in them, resulting in poor customer experience.
5. Gap in Closed-Loop Analytics: Continuous measurement and optimization of GenAI performance are crucial. Without the ability to track GenAI interactions, assess prompt effectiveness, and leverage explicit and inferred user feedback, several GenAI projects could not improve their user experience quickly enough to avoid getting labelled yet another trivial chatbot.

1. Invest in a Modern Knowledge Management System: Ensure that your knowledge management system is equipped to support GenAI with connectors and open APIs. In-built capabilities should include optimizing prompt management, curating relevant content, implementing business controls, and providing quality assurance.
2. Insist on Eliminating Content and Process Silos: Break down barriers between GenAI tools and existing knowledge sources. Integrate GenAI with enterprise content to ensure that it can access and utilize relevant information effectively.
3. Design your Knowledge Management Process with AI in the Middle and Expert in the Loop: Establish workflows where GenAI handles routine tasks while human experts provide oversight and intervention when necessary. Ideally, your knowledge platform should provide this capability. Building (and then maintaining and enhancing) all this capability with Copilot (or similar developer tools) is a lot of effort, time, and money.
4. Measure and Manage from Day Zero: Anything that cannot be measured cannot be improved. And many GenAI projects have failed because of lack of detailed visibility into their GenAI automated process. Ensure that your knowledge system has deep analytics that lets you iteratively improve the performance of your AI Knowledge system. In particular, ensure that you have metrics in place before you activate the AI tool, so you can track the before/after effect.

Successful customer service automation with GenAI requires a strong foundation of integrated knowledge management. Investing in a modern knowledge system to power GenAI projects will help you meet aggressive operational cost reduction and customer experience goals.

Contact us