Abstract
Conversational chatbots have become integral to automated query resolution across industries to enhance
customer service efficiency. However, their effectiveness in accurately addressing diverse customer queries remains a critical
area of evaluation. This study examines the performance of a rule-based WhatsApp chatbot deployed by a South African
telecommunications service provider, focusing on its ability to resolve customer queries effectively. This paper analysed
chatbot interaction reports and quantifies success, failure, and abandonment rates across various query types. This study
adopted a quantitative research approach with interpretivism as a philosophical paradigm. Furthermore, document analysis
was employed for the analysis of TELCO X WhatsApp Chat’s extracted reports to examine and evaluate human-chatbot
interaction. This method allowed the researcher to determine, from the interactions, resolved and unresolved queries. The
findings indicate that the chatbot achieves high success rates for routine and structured queries, such as Frequently Asked
Questions (FAQs) and retrieving account-related information. However, the chatbot’s performance declines significantly in
handling complex, multi-step, or context-dependent queries, including SIM swaps and product purchases. The chatbot
exhibits lower success rates, with a significant number of customer interactions resulting in unresolved queries or user
abandonment. The analysis highlights key performance limitations, including natural language understanding, contextual
retention, and the inability to process multi-step interactions effectively. Additionally, the absence of seamless escalation
mechanisms to human agents contributes to customer frustration when the chatbot fails to provide satisfactory resolutions.
The study provides recommendations for optimising chatbot performance, particularly in enhancing AI-driven response
mechanisms, refining intent recognition, and using advanced dialogue management techniques. Additionally, integrating
seamless escalation pathways to human agents is proposed to improve resolution rates for complex queries. In conclusion,
the study emphasizes the importance of continuous performance monitoring and iterative improvements. Identifying key
performance gaps and improvement areas provides valuable guidance for organisations looking to optimise chatbot
functionality.