Test Multiple Intent Predictions

Use bulk testing to:

• Evaluate up to 1000 utterances in a single request
• Compare expected and predicted intents
• Review confidence scores for each prediction
• Analyze overall performance by using summary metrics

You can perform bulk testing in two ways:

• Test Custom Utterances: Provide a dataset of utterances and expected intents to evaluate specific scenarios.
• Test Using Existing Bot Data: Evaluate all utterances already configured in the bot to generate a complete performance summary.

These two approaches correspond to different API endpoints.

Before you begin, turn on Einstein Bots and Einstein for Intents, and build your model. Turn on the Batch Input Tester pilot permission, and turn on the cross-lingual intent model (Intent V3) on the Bot Overview page.

Note Contact Salesforce Customer Support to turn on the Batch Input Tester pilot permission. If you turn off the cross-lingual intent model, Salesforce builds your model using Intent Standard V2 (for single languages) or Intent Multi-Language 2.5 (for multiple languages).

How Bulk Testing Works

Bulk testing evaluates utterances against your bot's intent model and returns:

• You initiate an evaluation request with a set of utterances or a selected bot version.
• The intent model processes each utterance and predicts the most relevant intent.
• The system compares the predicted intent with the expected intent, when provided.
• The system assigns a confidence score and rating to each prediction.
• The system generates a prediction summary with overall performance metrics.
• The system returns consolidated results for all evaluated utterances.

Understand Prediction Results

To understand how accurately the bot classifies utterances and to identify areas for improvement, review the prediction results.

Per-Utterance Results. For each utterance, the system returns:

• Predicted intent: The intent identified by the model
• Prediction outcome: Indicates whether the predicted intent matches the expected intent (pass, fail, or unprocessed)
• Confidence score: Indicates how likely the prediction is correct

Summary Metrics. The system provides aggregated metrics to help assess overall performance:

• Success rate: Percentage of utterances predicted correctly
• Fail count: Number of utterances predicted incorrectly
• Intent coverage rate: Percentage of intents correctly predicted for at least one utterance
• Error count: Number of utterances that the bot couldn't process

Considerations

Keep these considerations in mind when you use bulk testing:

• You can evaluate a maximum of 1000 utterances at a time.
• Results depend on the quality of intent training data.
• Review failed predictions to improve intent accuracy.
• Errors for individual utterances don't stop the overall evaluation.

For API details, see Bot Utterance Intent Prediction Resources in the Connect REST API Developer Guide.