Improving business processes using ML-Based chatbots
Support and service organizations often find themselves fielding the same set of questions over and over, whether it’s employees asking HR about benefits or customers asking how to log onto a network. Chatbots, automated engines that respond to typed user inputs, are a way to assist with these business processes and reduce levels of manual support.
To improve chatbot performance, we developed a process that reduces the up-front effort required to answer questions, while structuring the chatbot to ensure that users do not get caught in feedback loops.
A chatbot’s high-level requirements are that it is 1) automated and 2) effectively answers questions input by the user. The latter depends on the creation of a knowledge base from which the chatbot can pull answers, as well as the incorporation of a feedback loop so that the chatbot can improve based on user ratings.
In some cases, a knowledge base may not be readily available because previous user questions may be split between phone calls, screen share sessions, emails, and instant messages, making it difficult to sort through and consolidate this information into a unified question/answer database. We made it a requirement that our solution could get up and running with minimal data. Our goal wasn’t to answer every possible question, but to handle the most straightforward and common requests.
We also wanted to be able to update the chatbot’s answers as new information became available without requiring significant model retraining. This would allow the team implementing the chatbot to independently manage the knowledge base going forward, without the need for machine learning (ML) engineers to make modifications.
Our chatbot uses a semantic similarity model to match user questions to those in our knowledge base, an approach that balances our requirements. Semantic similarity methods use a siamese (dual) transformers ML model to compare the numerical representation (embeddings) of two sentences. By calculating cosine similarity between the embeddings of questions in our knowledge base and the question the user asked, we can provide accurate answers to the user based on human-curated information. For questions not answered in the knowledge base, a user will be directed to contact a representative. The chatbot functions through the following process:
The knowledge base from which questions and answers are derived is created by domain experts. In this example, we have a chatbot to help answer questions for the HR team. The HR team would create the dataset, which may already exist in the form of an FAQ. Examples below:
| Question | Answer |
|---|---|
| How do I request business cards? | Fill out the Business Card Order Form at www.exampleadress.com. An HR Admin will process your submission and you'll receive the cards in 1-2 weeks. |
| Where can I view my PTO Balance? | To view your PTO balance, please go to your timesheet and scroll down to the blue bar that reads "Timesheet Lines." To the right of "Timesheet Lines" is a link that reads "Leave." Click on that link to view your PTO balance. |
Once we developed a prototype that addressed the identified requirements, we built a basic application to demonstrate how users would interact with the chatbot. The main components of our application are user interaction and user feedback.
We needed to enable user interaction with the chatbot to compare user questions to those in our existing knowledge base. For our proof of concept, we created a web app using Dash, though it could be deployed as Slackbot or embedded in a website.
As with any application that incorporates ML, the model will not always provide the correct answer. To enable the model to improve on a continuous basis, we built in a user feedback loop with separate workflows for correct and incorrect / unhelpful responses.
If the chatbot answers the question correctly (per the user), we store the question/answer pair. As the chatbot answers questions, it is also building a question answering dataset with differently-worded questions. Once the dataset is large enough, we will be able to fine-tune this chatbot model to better understand company-specific acronyms and terminology, boosting the overall accuracy of the chatbot. Fine-tuning can improve the model when there are 5 variations of each question. Over time, additional fine-tuning can be done optionally to improve the model further as more data is collected and specialized terminology changes, such as names of benefits providers.
If the chatbot answers the question incorrectly, a workflow saves the questions to an Excel file. Domain experts (e.g. a HR rep) will periodically review the list and take one of 3 actions:
As discussed above, it is important to have a process for when the model fails to provide a good user experience. In this process, there is a simple and easy route to send it to a human, but failures are flagged for review.
By using this approach, we can set up effective chatbots quickly with built-in feedback-to-improvement loops. Implementing chatbot support in this manner has the ability to save time for business process use cases that generally require human intervention.