Test Preparation
Comprehensive testing begins with clearly defining what success looks like for your agent. Preparation also involves breaking down the agent into components, like subagents and actions, and identifying the most relevant data for evaluation. After you have this information, you can move on to designing effective structured test scenarios that reflect real-world interactions.
Required Editions
| Available in: Lightning Experience |
| Available in: Enterprise, Performance, Unlimited, and Developer Editions. Required add-on licenses vary by agent type. |
Define Objectives and Strategy
Before testing begins, clearly define what constitutes success for your AI agent.
- Set goals linked to business value: Establish specific, measurable, attainable, and relevant goals for your agent's performance, directly linked to the business objectives you identified for your agent.
- Deconstruct the agent: Break down the agent into components, such as subagents and actions. Testing these parts individually helps to pinpoint inefficiencies at their source.
After you’ve outlined your objectives, the next step is to create a testing strategy. A solid strategy includes:
- Identifying test scenarios and setting evaluation parameters.
- Running tests and validating the results.
- Making evidence-based adjustments and repeating tests until your agent is working as expected.
- Test on a regular cadence to make sure that your agent is still performing the way you intend.
Create Test Scenarios and Identify Data Requirements
A testing scenario evaluates how well an AI agent performs in a realistic situation. It simulates a real-world interaction and helps you test whether the agent understands the user input correctly, selects the right subagent, runs appropriate actions, and responds accurately, completely, and in the right tone. A strong testing scenario includes:
- Scenario description: A short narrative of the situation or context, such as a customer who just received their device and is confused by an error code.
- Test utterance: The actual phrase or question the user enters.
- Expected subagent: The subagent that the agent is expected to select based on either its configuration or deterministic routing transitions.
- Expected action: The specific action that the agent is expected to take in response, like “lookup information about products and policies” or “escalate the conversation to a service rep.”
- Expected response: A brand-appropriate reply based on validated information. The agent’s wording can vary, but verify that the tone feels right and the response is accurate.
- Utterance variations: The different ways that customers are likely to phrase a question or request. Include utterances that vary based on language differences and different customer emotional states as well as utterances that change the conversation topic.
- Multi-turn: A conversation that extends beyond one question and response. It’s important to test how the agent handles requests in a continued conversation.
