Intelligent Virtual Assitant (IVA) helps businesses automate their support. User queries are answered by these virtual assistants powered by NLU and Machine Learning. By being always available and providing a faster resolution to the users, these virtual assistants drive value for our business.
But just like an agent, virtual assistants also need to keep learning and improving over time for being effective. But how do we do this? This could be achieved only when we analyse data regularly and pick up feedback points for these virtual assistants.
After years of reading & analyzing data, Haptik has come up with few best practices that could be followed to make our virtual assistants grow.
What are the core objectives?
Virtual assistants are used for both Support and Lead Generation usecases. There are 2 basic objectives for Support and Lead Generation virtual assistants respectively -
- Answer maximum queries correctly and instantly to divert call centre volume
- Increase leads for a business
How can you achieve these objectives?
These objectives can be met by tracking few important metrics related to the virtual assistant. Haptik's Intelligent Analytics will help us do that! We can either track these metrics real time on the tool or run detailed analysis on the downloaded reports.
Intelligent Analytics has all types of metrics available related to the virtual assistant's efficieny and performance. We will go over the ones which will help us improve our virtual assistant over time.
1. Automation %
If a user's message is understood by the virtual assistant, then it gives the designed response and if a user's message is not understood by the virtual assistant then it gives a Bot Break Message or gets an agent involved. Bot Automation % tells us about what percentage of conversations were handled correctly by the virtual assistant and did not have either a bot break or an agent involved.
You should keep monitoring the Automation % regularly and make sure it does not drop below the threshold set in the beginning of the project planning.
An efficient virtual assistant will have a HIGH Bot Automation %.
How to improve Automation %?
Monitor the bot 1 week post go live. The ultimate goal is to increase virtual assistant's efficiency by making it answer all questions users are coming and asking. For this, we would need to find out messages where the virtual assistant is not giving desired responses. AI Recommendations, Word Analysis and Message Analysis are different screens that can be used to achieve this.
AI Recommendations can be used to identify issues with the virtual assistant by finding out missing user intents and training deficit on existing intents. This information is provided by the algorithm by going through thousands of conversations. It suggests both utterances that existing use cases should be trained on as well as out of scope categories that are being asked.
Word Analysis can be used to check Bot Break Topics for prioritising over broken messages based on the volume. We can fetch exact user messages which are resulting in these bot breaks to add them under User Says section of the virtual assistant.
To know the conversations in more detail once prioritisation has been done, you can view these chats on Message Analysis section. You can achieve this by selecting the Broken Messages filter.
You can see the actual messages sent by the user in the Body column or you can view the entire conversation.
If a similar intent is present on the virtual assistant, you can add this message in the User Says section of the virtual assistant. If the intent does not exist, then you can create a new one on the virtual assistant and train the virtual assistant to understand such user queries. You should train the virtual assistant by adding maximum user variants to increase its efficiency. Once the virtual assistant is trained on the broken intents, the automation % will increase.
You can also download Message Analysis report to get the entire dump at Message or Conversation level. You can go through every single message received on the virtual assistant.
You can leverage this report to -
- Analyse the actual user messages.
- You can pivot the report to find major and common bot breaks.
The following is a snippet of Message Analysis report -
If after a point, further analysis shows that the automation % reduction is majorly due to out-of-scope queries, then this could mean either or both:
- Marketing efforts have been ineffective in defining the capabilities of the virtual assistant to the end users.
- Wrong use cases have been picked for the virtual assistant.
2. User Count and Conversation Load
Unique User section shows the number of unique users who have visited the virtual assistant in the given duration period. Conversations graph represents the conversations count. Unique users and conversations graph can reveal the discoverability and usage of the virtual assistant.
A highly used virtual assistant will have a HIGH Unique User and Conversation count.
How to improve User Count and Conversation Load?
To improve User & Conversation Count, you can take the below suggested steps -
If the number of users who are visiting the virtual assistant is low, you can add Bot Prompt Triggers to grab users' attention
You can deploy the virtual assistant on additional channels such as WhatsApp, Facebook, Google Business Messages and others
You can have a release plan to make users aware about the presence of virtual assistant and that they can seek help on it. Have few marketing initiatives started to spread awareness about the virtual assistant
You can use Deep links for the virtual assistant used in email signatures or email footers
Whatsapp Notifications(HSMs) can be sent too to keep users engaged with the brand
Conversation count can be increased by adding new queries on the virtual assistant which users can come and ask. It may be related to the new products or services your brand has released.
Story Analysis will give you a deeper understanding of which usecases are having more traffic than others. This will help you prioritize the usecase that should be improved first on the virtual assitant. You can compare these numbers week by week to understand the trend as well.
This following is a snippet of Story Analysis report -
Direct feedback given by the user is always a great indicator of the performance of the virtual assistant.
Make sure feedback collection is enabled on your virtual assitant. If not, you can enable it by following the steps here.
User Rating & CSAT sections help us in finding this out as they give the average of all feedback provided by the users on a scale of 1-5 at the end of a conversation.
The feedback collected from the users can aid in making the user journeys easier and improving the virtual assistants' performance.
An efficient virtual assistant will have a HIGH User Rating.
How to improve Feedback collection?
There are few DOs and DONTs related to Feedback. Let us have a quick check at that -
- Feedback collection should be done only when users have completed their journey and not in the middle of the conversation
- Negative feedback received should be analysed in detail to derive insights on bad user experience
- User feedback should be distinguisged into Bot vs Brand and Relevant vs Irrelevant and Bot vs Agent
- User Reviews section will help you analyse the reviews or comments left by the users. It might reveal why users are leaving a negative feedback on the virtual assistant
- Viewing the respective chats under User Reviews can tell you what was the user's query and why the virtual assitant was not able to resolve it. If the virtual assistant could not understand the query, then it can be added as a new intent to handle it in the future
Using the User Rating report, you can check the all the ratings and reviews which users have given to the virtual assistant or the agent. You can also check the chat links in detail to further understand the reason of a low rating or a bad comment.
The following is a snippet of User rating report -
4. Completion Rate of Funnels
Smart Funnels help us track the effectiveness of the virtual assistant by tracking the completion rates and identifying the drop off points of conversations started on the virtual assistant. This feature plays an important role to enhance user journeys and IVA content.
Various metrics like completion rate, drop-off rate, drop-off nodes, and funnel hits help us to make changes in the virtual assitant to ensure users are completing the conversation flow and getting a resolution.
An efficient virtual assistant will have a HIGH Completion Rate and LOW Drop-Off Rate.
How to increase completion rate and decrease drop-offs?
Even before coming to increasing completion rate of funnels, let us start with how to create these funnels? You can know more about it here.
Smart Funnels give the number of people initiaing a user journey, number of people completing the user journey and places of drop-offs in the user journey. Once we have the funnels ready, we can monitor these metrics regularly to find the problem areas in the journey and fix them.
To decrease the funnel drop off rate, we can perform the below actions on the drop off points -
- Go to Message Analysis to see the bot breaks happening once the user enters the flow
- Modify the user journey by removing the unnecessary steps or questions asked in the journey
- Modify the virtual assitants' responses
- Add Follow-up messages on the virtual assistant so that users are prompted to continue their conversations and complete their user journey
- Make sure that user journeys are not complicated and lengthy so that more and more users are likely to complete it
You can download the Smart Funnel report which will help you highlight the touch points in the user journey which are causing the maximum drop-offs along with the reasons for the drops. You can check the corresponding chat links to understand the pretext of the drop-off. There are various reasons for these drop offs and you can read more about it here. On the basis of this analysis, you can find where the virtual assitance needs more improvement in terms of training data or design considerations.
The following is a snippet of Smart Funnel report -
Additional points to improve your virtual assistant
Building a flow if the data suggests that users are not happy with an FAQ approach.
Highlight all API errors immediately and take corrective measures.
Set usage, automation percentage and API failure triggers to get notified in case of sudden drops in usage or bot performance. You can contact your CSM to enable the same on your virtual assistant.
Review chats where users showed dissatisfaction and create action items to improve user experience.
Arrange for UX retro with Conversation Design team if the bot has too many breaking flows or user journeys.
If you have doubts regarding any of these best practices or you want to know about any of these in more detail, feel free to reach out to your Customer Success Manager.