The Training Suggestions mail helps Bot Builders discover methods by which a bot can be improved and brought to a state where it is utilizing the capabilities of the ML Model to its maximum potential. It aims to give feedback to Bot Builders on how well the bot has been trained from the "User Messages" and "Steps" created by the Bot Builder. However, before we delve into the details of the Training Suggestions, it’s essential to understand the expectations of the ML Model from the Bot Builder, so that improvements can be made accordingly.
What does the ML Model expect to perform at its fullest?
- 10 variants on each Start Step.
- 10 variants on each Connection response including -
- Start Step to Dependent Step.
- Dependent Step to Dependent Step.
- OutputStep to Static Step.
- Output Step to Dependent Step.
- If more than 10 Connection responses are not present, the performance of the ML Model, even on Start Steps would be affected negatively.
- Similar or exact same “User Messages” should not be present in multiple steps.
-
Steps should not be too similar in the goal they aim to achieve, else the bot may not be able to differentiate, and decide the correct Step. The image below shows two similar steps, and due to the high similarity between these Steps, a minor variation in the User Message prevented a one-shot response from the bot and triggered a Disambiguation Message.
- The quality of the “User Messages” should be as per User Messages Guidelines.
The Training Suggestions reaches the Conversation Studio as part of the Training Completion Email. It is divided into certain scenarios, and each scenario has a list of issues with Training Data present in the bot. If either of the scenarios have more than 30 entries, the section would be attached as a CSV to the email.
The Bot Builder should go through each of the scenarios, and make sure they fix the issues if there are any so that the bot functions properly.
The Training Suggestions only considers Start Steps for evaluation and display purposes.
1. Standard Scenario
From: Niko@haptik.co
To: Bot training, User
Subject Line: <bot name> training is completed. Here are some suggestions
Body:
Hi <first name>,
Your <bot name> training is completed. Good job - before you take it live, we want to make sure it’s intelligent enough to woo your customers.
Here are some quick checks we ran, just to be double sure. If the below tables are empty, pat yourself on the back - your bot is perfectly trained!
Will your bot get confused between two similar user says statements added at multiple steps? | |||
User says statement |
Step containing user says |
Similar user says statement |
Step containing similar User Says |
How to fix it?
If your bot gets confused between two similar user says statements added at multiple steps, then you can fix it by,
- Testing out different variations of User Message statements for your bot, wherein you can frame the above-mentioned User Messages in different ways and ask your bot. If the bot gets confused in taking the next step or answers the query incorrectly, you can then add new statements variations in your preferred Step.
- Make sure that similar User Message statements are not present in multiple steps.
Will your bot fail to interpret a question if framed differently? | |
User says statement |
Step containing user says |
How to fix it?
If your bot fails to interpret a question, when framed differently, then you can fix it by,
- Testing out different variations of the above user message statements for your bot. wherein you can frame the above-mentioned user messages in different ways and ask your bot. If it gets confused in taking the next step or answers the query incorrectly, you can also add the new variations of statements in the same Step.
- Make sure there are 10 or more User Message statements added for every Step.
- Make sure there are 10 or more User Message statements added for every Connection.
Will your bot get confused between any two similar steps? | ||||||
Step 1 |
No. of user says |
User Says causing overlap in step |
Step 2 |
No. of user says |
User Says causing overlap in step |
Degree of similarity
|
How to fix it?
If your bot gets confused between two similar steps, then you can fix it by,
- Testing out mixed variations of the above User Message statements for your bot. You can also mix the words from variations of similar User Messages and then ask your bot. If it gets confused in taking the next step or answers the query incorrectly, you can add new statements variations in your preferred Step.
- Merging two Steps that are very similar, or cater to similar use-cases.
- Make sure there are 10 or more variations of User Messages added for every Step and Connection.
If you have any questions, please reach out to support@haptik.ai. After you’ve resolved all the above issues, your bot is good to go!
Cheers <cheers emoji>,
Niko, your Platform Guide
2. Scenario - Feedback is too large to display
From: Niko@haptik.co
To: Bot training, User
Subject Line: <bot name> training is completed. Here are some suggestions
Body:
Hi <first name>,
Your <bot name> training is completed. Good job - before you take it live, we wanted to make sure it’s intelligent enough to woo your customers.
Here are some quick checks we ran, just to be double sure. If the below tables are empty, pat yourself on the back - your bot is perfectly trained!
Will your bot get confused between two similar answers for any particular question? |
The report is attached as similar_user_says.csv |
How to fix it?
If your bot gets confused between two similar answers for any particular question, then you can fix it by,
- Testing out different variations of User Message statements for your bot, and then framing the above-mentioned User Messages in different ways, and asking your bot. If it gets confused in taking the next step or answers the query incorrectly, add the new statements variations in your preferred Step.
- Making sure similar User Messages are not present in multiple Steps.
Will your bot fail to interpret a question if framed differently? |
The report is attached as bot_has_not_learnt.csv No. of entries - 123 |
How to fix it?
If your bot fails to interpret a question if it has been framed differently, then you can fix this by,
- Testing out different variations of the above User Message statements for your bot, and then framing the above-mentioned User Messages in different ways and asking your bot. If it gets confused in taking the next step or answers the query incorrectly, add the new variations of statements in the same Step.
- Make sure there are 10 or more variations of User Messages added for every Step.
- Make sure there are 10 or more variations of User Messages added for every Connection because without this a Step might not get initiated.
Will your bot get confused between any two similar steps? |
The report is attached as similarity_between_intents.csv |
How to fix it?
If your bot gets confused between two similar Steps, then you can fix it by,
- Testing out mixed variations of the above User Message statements for your bot. Mixing the words from variations of similar User Messages and then asking your bot. If it gets confused in taking the next step or answers the query incorrectly, add the new statements variations in your preferred Step.
- Merging two Steps that are very similar or cater to similar use-cases.
- Make sure there are 10 or more variations of User Messages added for every Step and Connection.
If you have any questions, please reach out to support@haptik.ai. After you’ve resolved all the above issues, your bot is good to go!
Cheers <cheers emoji>,
Niko, your Platform Guide
3. Scenario - There is no feedback
From: Niko@haptik.co
To: Bot training, User
Subject Line: <bot name> training is completed. Here are some suggestions
Body:
Hi <first name>,
Your <bot name> training is completed. Good job - before you take it live, we wanted to make sure it’s intelligent enough to woo your customers.
Here are some quick checks we ran, just to be double sure, and looks like your bot passed all the tests!
Will your bot get confused between two similar answers for any particular question? |
No errors found! |
Will your bot fail to interpret a question if framed differently? |
No errors found! |
Will your bot get confused between any two similar steps? |
No errors found! |
Your bot is good to go!
Cheers <cheers emoji>,
Niko, your Platform Guide
FAQs
1. Is it compulsory to add 10 User Messages to all independent responses and Connection User Messages?
Yes, because if you add less than 10 User Messages, then the ML model becomes confident giving rise to a lot of issues in the Training email.
2. Can we initially add more than 10 user messages as independent responses before working on the issue of the Training Suggestions email?
Yes, because if you add less than 10 User Messages, then the ML model becomes underconfident, giving rise to a lot of issues in the Training email.
3. Is it advisable to add duplicate User Messages?
No, you can reshuffle the words.
4. Should we remove the User Messages in order to solve the issue of the Training email?
No, you can test the variation if it is going to the wrong Step, and then add that User Messages to the correct Step, and add more similar variations on the same Step.