Chatbot Testing

A chatbot is a program for sending automatic responses to user messages. The answers are as close as possible to the real communication and correspond to the asked question. Such chatbots are used to provide feedback and perform various functions for the client on websites, social networks, messengers, call centers, etc. They are also personal assistants (for example, Siri, Cortana, and Alice).

Chatbots can be built on artificial intelligence using machine learning (improving your understanding of human language in the communication process) or on programmed scenarios with a choice of different answers or questions (algorithms).

How smart a chatbot will be depends on how it is designed. Machine-learning chatbots handle tasks better because they can understand human language. They learn to talk to the user, “gain experience” and can solve new situations like the previous ones. Unlike chatbots in programmed scenarios, here the user does not have to write certain exact phrases to get a relevant answer.

Chatbots are popular due to their cost-effectiveness, the ability to provide feedback to users 24/7, the effectiveness of communication with customers, in particular, due to the ability to provide quality service in several languages.

How Can You Most Effectively Test the Work of Chatbots?

1. Check the process of conversation with the chatbot. Involve the chatbot in the conversation and go through different communication scenarios, from potentially the most frequent or important phrases and cases to exceptional situations.

2. Use specific terms to verify that it provides reliable information about a product or service.

3. Involve developers into testing. They test the questions and answers of the chatbot during the next build. This will be considered verification and validation, whether the chatbot provides accurate and relevant answers to the questions of the hypothetical user.

4. Check situations where the user enters a meaningless sentence or expression that is not used very often. Of course, it is impossible to predict all the information that users can enter and that is not relevant. You cannot check all the exceptions that the chatbot will have to handle. And yet developers need to come up with “emergency answers” for those exceptions that can be predicted. At this stage you need to test chatbot responses to “exceptional” scenarios.

There is also a list of words that the chatbot must understand so that the user does not consider it stupid. For example, these are such basic words as hello (and its analogs), restart, start, from the beginning, menu; such specific words as buy, order, range, services, delivery, payment, conditions, etc.

5. Use tools for testing chatbots, for example, Chatbottest, Botanalytics, or Dimon.

Chatbottest is an open-source resource which offers 120 questions to evaluate the user experience.

Works on three levels:

• possible scenarios for chatbot testing;
• expected scenarios;
• almost impossible scenarios.

Provides 7 metrics for evaluating chatbot performance:

• Understanding: does the chatbot understand any messages from users –swear words, small talk, idioms, emojis?
• Answers: are the bot’s answers relevant and accurate enough?
• Navigation: is a conversation with a chatbot intuitive enough?
• Personality: does the tone of the chatbot’s responses match the target audience and the nature of the conversations?
• Start: is it clear to the user how the chatbot works when first used? Is interaction intuitive enough? Each response of the chatbot should have a call to the next step: the need to write something or a button to further develop the conversation.
• Intelligence: whether the chatbot remembers certain details, key information that the user provides throughout the conversation.
• Error management: how the chatbot handles errors and exclusions.

Botanalytics: a custom service that allows you to test every key aspect of a chatbot, from usability to conversation.

Dimon: a distinctive feature of this chatbot testing tool is that it easily integrates with major platforms such as Facebook Messenger, Slack, WeChat, Telegram. It can be used to detect any problems in the chatbot dialog.

6. Automate chatbot testing. You can use a chatbot that will interact with the test chatbot without interference from the tester. This way, you can start recording conversations automatically. The tester’s work will be reduced to one key task: evaluating the chatbot’s responses.

7. Make the chatbot as close as possible to the person. The chatbot can have everything to make communication more comfortable and real: name, photo, or logo.

8. Check the duration of pauses between messages. Users should have enough time to read the message before receiving the next one.

9. Check the operation of the buttons and quick answers. To facilitate user responses to messages, the chatbot can send quick response options. The buttons you need to navigate can be used to respond to the chatbot. Quick answers usually disappear when you click on them, and the button does not. Quick responses used for rejection or denial must be very polite and tactful so that users do not feel uncomfortable using them.

10. Check that after the conversation with the chatbot it is possible to return to it again. There must be an answer that will re-engage the chatbot in the conversation. There should also be a start message from the chatbot when it is opened so that the user can understand what to do next.

11. Check that there are no deadlocks in the conversation with the chatbot. The user must understand what each stage of communication means and what to do next.

Chatbot testing is an ongoing process. Any testing before its release is, in fact, beta testing, regardless of whether the tester tests it manually or chooses automated testing. Real feedback from users can be a real challenge. You need to prepare to constantly configure your chatbot according to the feedback that will come after its release.