After testing the Training data, you can now conduct full Functional testing of the bot. This phase will ensure if the bot is ready to handle different types of user queries, whether the bot is taking the user through the right conversational flow and if the bot is able to handle out-of-scope queries from the users.

The following points elaborate on different components which are required to be tested.

All functional flows verification (as per IVA design document):

Testing all the functional flows will ensure if the bot is giving correct responses to the user. It will also ensure if the users are going through the journey as designed during the Conversation Design phase.

• Test all the positive and negative flows as per the design.
• Identify and write different scenarios before test execution.
• Execution of scenarios, and issues fixes with retesting.
• Prepare a sheet for all the test cases you are testing. Find a sample sheet at the end of the document.

API testing and its alignment with requirements in IVA:

All the APIs should be tested to ensure users are getting responses and the APIs is giving timely responses.

• Verify the API responses are as per requirements.
• Verify API errors.
• Verify API response time (should be no more than 20 seconds).
• Verification of different APIs responses handling on the IVA.
• Verify the responses for different types of test data.

Small talk verification:

Small Talk is a module present for all the bots. Small Talk responds to random questions asked by users such as "Are you a robot?"

• IVA should reply correctly for small talk (if it is enabled).

• Verify the negative intents for Small talk as well.

  Example: You are useless.

• Ask the question such as ‘Who made you?’ It should not give Haptik branding. Small talk customized messages are recommended.

FAQs verification (if FAQs are present):

• All the FAQs should be answered correctly by IVA.
• Each step should have a specific intent, avoid similar intents on multiple steps.
• Ask questions with 3-4 variations for the same intent.

Error handling: API error, invalid inputs, Bot break, and reprompts:

As the bot may break on different scenarios, the users should receive a prompt reply from the bot regarding the error.

• Bot break message should be displayed for invalid user inputs.

• Out of scope queries should be handled by giving proper guidance through bot break or outro messages, so as to bring the user back in the bot journey.

• APIs error messages should be handled by giving an appropriate response to the user.

• The entity reprompt message should be relevant, along with hint text; so as to guide the user about what is expected

  Eg. Entity bot response: What's your phone number?

  Entity retry message: That seems incorrect. Please enter a valid 10-digit phone number.

  Entity bot response: What's your date of birth?

  Entity retry message: That doesn't seem like a date. Please enter a valid date between 18/5/1990 and 26/11/2019.

UI elements and Content Verification

• Verify the functioning of various UI elements such as Buttons, QRs, Carousel, Forms, Webviews, Location Picker, and so on.
• Hyperlinks should be clickable and should point to the correct endpoints.
• Verify if the bot persona is in line with the company logo and branding.
• Verify the tone used in the bot is as per design i.e causal, formal, friendly, funny.
• Check the grammar and spelling along with punctuations.
• Use of correct emojis at correct places and bullet points

Carousel	Text and Buttons	Location
Text and Quick Replies	Menu	Chatform

Outro flows/follow up messages:

The journey should never end abruptly at any juncture unless the user abandons the conversation midway. If any Bot Response does not contain QRs, buttons, or a follow-up message, it is considered abrupt.

• Check for follow-up messages based on the use case/journey.

  Example: Was I able to resolve your payment issue?

• Check if there is a generic follow-up message to re-engage the user.

  Example: Wait at the end of a journey (20-30 Seconds).

  A normal follow-up message should appear.

  Example: Still looking for chairs? Tell me how many you need.

  Support email or contact number should be present in the bot whenever the user asks for further assistance from the support team.

Negative User Messages handling:

Test variations with negative tone within the bot flow.

• The IVA should be able to handle negative cases and journeys that are not linear. The goal should be to channel the user back into the flow.

  Example: I don't remember account number I didn't receive OTP I did not get any email

For multilingual IVAs (if enabled):

• Functional flows should work same as it works for the primary language.
• Adding a different language should not impact English language flow.
• Bot break message language should be checked.

Chat agent functionality (if enabled):

• Check if the chat is transferred to the agent on bot break or upon user messages.
• Check if the chat gets assigned to an agent and if you are getting responses on the bot.
• Check if the agent is able to close the chat.

Preparing Test cases

Test cases preparation guidelines and sample template:

• All the positive functional flows of IVA
  • Verify all possible flows in IVA
  • Verify all the bot’s invocation points
  • Verify UI elements like carousels, buttons, quick replies, chat elements, dropdowns, location pickers, attachments, web views, forms, message copies, emojis(WhatsApp IVA)
  • Verify user authentication e.g: OTP, PAN, etc
• Negative cases and validations
  • Bot break scenarios (when out of scope query is asked or invalid inputs are given by the user)
  • Verify the outro messages are clear and meaningful to the user

The following is a file containing sample test cases, click here to download.