Improve Your Bot

Collecting Data: The Bot Snapshot Report

Einstein Bots collects data in four major categories: User Activity, Errors, Natural Language Processing (NLP), and KPI changes. Bot data is also segmented by event activity (data about the bot conversations) and session data (the conversations themselves). If you want to report on bot sessions, use the Conversation Definition Session object in Report Builder. To capture bot activity over time, we recommend that you create reports from the Conversation Definition Dialog Daily Metrics and Conversation Definition Hourly Dialog Metrics objects.

Use standard reports to identify key usage trends in your Einstein Bot. Learn more at Navigate Einstein Bot Standard Reports. You can export event data by exporting a standard bot report as a formatted report. Or you can export the detail rows from a standard bot report in Excel or CSV format.

You can also collect event data through the API (ConversationDefinitionEventLog object). To learn more about event types, visit the Bot Data Reference Guide. To gain a full picture of the bot, we recommend that you include data that lives outside of the bot: different objects for recording CSAT, or comparing bot trends to the company calendar.

In building your Snapshot Report, we recommend that you pick the reports that work best for your company’s bot goals. Not every bot has data in all four categories. For instance, bots that don’t use Intent Management don’t have NLP data, but they accept customer text. Also, every company collects data on customer interactions and KPIs in different ways. The success of your Bot Snapshot Report isn’t in how it looks but how it’s used to recommend areas of growth.

User Activity Data: This data includes data about the bot sessions.

• Conversation Definition Session:
  • Session Duration: The length of the conversation.
  • Session Transfer Target Type: Identifies whether a conversation ended in a transfer to a bot or a support rep.
  • Session Transfer Type: Identifies the reason for transfer (Bot Request, Implementation Error, Invocation Timeout, Insufficient Privileges).
  • Session Transfer Result: The result of the transfer (Transfer Successful, Transfer Failed, No Transfer Requested, No Agents Available).
• Conversation Definition Dialog Daily Metrics & Conversation Definition Hourly Dialog Metrics:
  • Article: The ID of the selected knowledge article.
  • Article name: The title of the selected knowledge article.
  • Goal name: The name of an admin-defined goal. To define a goal, the admin adds the Goal dialog step to a dialog.
  • Integration name: The name of the external connection to the bot.
  • Integration type: The type of external connection to the bot (API).
  • Language: The selected language for the bot. A bot with one language shows the language set on the Bot Overview page, and a bot with multiple languages starts with the primary language and then updates to the selected secondary language.
  • Menu choices: A count of the static choices selected by end users.
  • Number of dialogs started: A count of dialogs started, by bot dialog name.
  • Number of sessions per channel: The number of sessions grouped by channel.
  • Session count: The number of conversations automated by the bot.
  • Transfers by type: A count of transfers, grouped by destination.
• Conversation Definition Session Metrics:
  • Last dialog: The name of the last visited dialog when the bot session ended.
  • Number of articles selected: A count of articles the end user selected inside a session.
  • Number of choices selected: A count of the total number of choices the end user selected inside a session, including static choices in questions and choices from the Main Menu.
  • Number of dialogs started A count of dialogs started inside the session.
  • Number of input and output messages: A count of messages sent by the end user to the bot, and a count of messages sent from the bot to the end user.
  • Number of interrupted dialogs: A count of dialogs where the end user chose to exit the dialog (selecting from the Options Menu or entering text that matches an intent).
  • Number of Options Menu choices selected: A count of choices the end user selected inside a session from the Options Menu.
  • Transfer wait time: Duration in seconds between the user requesting a transfer and a support rep accepting the transfer.
• Event Logs (use these events to count the number of times each happens in a session):
  • Call dialog: The bot called a dialog.
  • Clear Variable: The bot used the Clear Variable Value dialog. The Variable Name is included in the event log.
  • End chat requested: The bot ended the chat.
  • End dialog: The end of a dialog.
  • Engaged session: Within a bot session, a user sends at least one message or clicks at least one menu option or choice. A session is categorized as engaged or not.
  • Entity extracted: The bot successfully extracted an entity.
  • Escalation requested: An escalation to a support rep was requested.
  • External entities extracted: The bot successfully extracted an entity from an external source.
  • Goal completed: A user reaches an admin-defined point in a dialog flow where a common customer goal is considered complete. To define a goal, the admin adds the Goal dialog step to a dialog.
  • Initializing context variable: The bot is accessing a context variable.
  • Invocation successful: The bot invoked a flow or Apex.
  • Language: The selected language for the bot. A bot with one language shows the language set on the Bot Overview page, and a bot with multiple languages starts with the primary language and then updates to the selected secondary language.
  • Message sent: A message sent to a customer.
  • Prompt sent: A question sent to a customer.
  • Redirect to dialog: The bot redirected to a dialog.
  • Rule condition evaluation: The bot evaluated the customer input based on the rule conditions.
  • Rule condition item result: False: The bot evaluated the customer input based on the rule conditions and found it to not meet those conditions.
  • Rule condition item result: True: The bot evaluated the customer input based on the rule conditions and found it to meet those conditions.
  • Search Successful: A successful attempt at Object Search.
  • Sensitive data not available: Shows when Store Einstein Bots conversation data isn’t checked.
  • Session ended: The bot session ended.
  • Set Variable The bot used the Set Variable dialog. The Variable Name is included in the event log.
  • Start dialog: The start of a new dialog.
  • Starting a new session The bot started a new session.
  • Transfer successful: The transfer was successful.
  • Transferred from bot: The conversation started with a bot transfer.
  • User response: The response from the customer.

Error Data: This data includes data about any errors that occur inside a conversation.

• Conversation Definition Dialog Daily Metrics and Conversation Definition Hourly Dialog Metrics:
  • Number of exceptions thrown by a dialog.
• Event Logs (use these events to count the number of times each happens in a session):
  • Error: Identifies when an error occurred inside a conversation.
  • Invocation failed - errors: Highlights when an error occurred in a flow or Apex. Admins can pair this data with event log listings of Invocation Successful to create a success rate.
  • Variable already filled: Identifies whether a skip in conversation occurred because a variable was already filled. If the skip is unintentional, the admin can include Clear Variable actions to ensure the bot doesn’t skip the variable.
  • Search failed - errors: Identifies the number of times the search encountered an error.
  • Transfer failed: Identifies the number of times a transfer attempt failed.

NLP Data: This data reflects the processing of free text and the performance of the Intent Model (if applicable).

• Conversation Definition Dialog Daily Metrics / Conversation Definition Hourly Dialog Metrics:
  • Confused: The number of times that a bot can't match a user input to an intent or entity.
• Conversation Definition Session Metrics:
  • NLP intent request count: The number of text inputs sent for intent matching.
  • NLP intent request hit count: The number of text inputs with a successful intent match.
• Event Logs (use these events to count the number of times each happens in a session):
  • Intent detected successfully: An event listing where the bot identified an intent.
  • Intent detection failed: An event listing where the bot was unable to identify an intent.
  • External intents detected: An event listing where the bot detected an intent from an outside source.
• Model Management tab:
  • F1 Score per Intent: The performance of each individual intent based on customer input. To learn more about interpreting F1 scores, visit Evaluate How Well Your Bots Understand Your Customers.
  • Intent Recommendations: Details on how to improve each intent to reduce confusion or increase accuracy.
  • Bot Training: While training your model, record the unclassified free text to identify the top things that your customers are asking your bot.

Note To visualize the accuracy of your intent model, you can use the Einstein Intent Assessor AppExchange tool.

KPI Data: This data is included in the bot data. These performance metrics are specific to your company. They can include, but aren’t limited to, the following:

• Case Deflection
• Average Handle Time compared to Bot Session Time
• CSAT (Customer Satisfaction score)
• Active Lead Qualification: Sessions that result in an instant transfer to a support rep
• Passive Lead Qualification: Number of lead records created
• Number of opportunities in the pipeline with Bot as the source

Prioritize Bot Growth

Now that you have these insights in one place, revisit your bot goals to create a prioritized list of improvements. Many bot goals fall into these areas:

Service Goals: The most common way that a bot adds value to a company is in cost savings. By automating low-effort tasks, your support reps can handle more complex issues and can easily manage traffic spikes. Bots that are focused on cost savings can prioritize goals like increasing the number of functions. Bots can also expand into new digital channels such as SMS or Facebook Messenger to meet customers where they are.

To increase the number of functions, check for popular dialogs or menu selections to see what interests your customers the most. In addition, check Bot Training to see what your customers are asking your bot.

Sales Goals: Another common goal for bots is in sales lead qualification. Many end users are more comfortable answering questions in a conversation rather than filling out a form. Including contextual data such as the channel or source page helps match sales leads to the best sales representative for the job.

To increase lead qualification, add your bot to more pages on your website or expand into different channels.

Customer Satisfaction Goals: A great bot experience encourages repeat visits, so it’s important to prioritize customer satisfaction. Bot goals focused on customer satisfaction include training a robust NLP model, introducing new languages, and decreasing error or confusion rates.

Common tasks to increase customer satisfaction include generating new skills to increase functionality or fixing bot dialogs to reduce error rates. You can clone a bot to create the same bot in a different language or for a new audience. Training the NLP model and rewriting dialogs to match your company’s voice and tone helps the bot understand your customers and inject personality into the experience.