Abstract
Various industries, including but not limited to financial services, healthcare, and telecommunications, are increasingly adopting Artificial Intelligence (AI) and chatbot technologies to manage customer interactions and enhance customer service. However, the effectiveness of chatbots in resolving customer queries remains a critical area of evaluation. Drawing from an extensive literature review, the study identifies a gap between chatbot capabilities and customer service expectations and highlights the challenges posed by limited contextual comprehension, constrained language processing, and the need for adaptive learning within customer support systems. This study investigates the effectiveness of a rule-based WhatsApp chatbot implemented by a South African mobile telecommunications service provider in resolving customer queries and examines the chatbot’s quality and responsiveness. The study integrates the Human-Information Interaction (HII) Theory and the Information Systems Success Model (ISSM) to provide a structured assessment framework for analysing chatbot functionality and performance across three critical dimensions: information retrieval accuracy, linguistic processing capability, and AI proficiency. The study adopts a mixed-method, multi-method approach within the interpretivism paradigm and involves a cross-sectional analysis of quantitative interaction data from chatbot reports, and qualitative insights gathered through thematic analysis of interviews with three employees and the platform evaluation. Findings reveal that while the chatbot effectively resolves simple and routine queries, it struggles with complex and context-sensitive queries. These challenges, identified through recurring resolution gaps in customer interactions, suggest that the chatbot requires enhanced natural language understanding, context retention, and personalised response generation to resolve queries better. Findings also highlight the importance of ongoing performance monitoring and iterative improvements in chatbot design to align with evolving customer expectations and industry benchmarks for service quality. Based on these insights, the study provides recommendations and actionable insights for optimising the chatbot’s effectiveness. The focus is on augmenting linguistic capabilities, refining AI-driven response mechanisms, and enhancing overall system performance. This study contributes to the growing discourse on AI-driven customer service solutions in South African mobile telecommunications and other industries by offering a comprehensive framework for evaluating and enhancing chatbot effectiveness. The
Page xii of 112
study provides actionable guidance for organisations that leverage chatbot technology to improve customer experience and operational efficiency within the unique context of South Africa.