You are here:
Optimize and Test a Semantic Model
Building, enriching, and testing a semantic model is an iterative process rather than a linear one. As you define and refine the semantic model, you can use various feedback mechanisms available in the Semantic Model Builder to make incremental adjustments. As you fine-tune your approach, each improvement can lead to better operational outcomes.
Required Editions
| View supported editions. |
| User Permissions Needed | |
|---|---|
| To use Semantic AI Model Optimization and Q&A Calibration: | Tableau Unmetered Admin or Tableau Next Admin OR Tableau Unmetered Platform Analyst or Tableau Next Platform Analyst |
- Optimize
When defining the semantic model, use optimization tools in the Semantic Model Builder to directly embed your organization's unique intelligence into the model.
- Test
When getting Tableau Agent ready for use by your organization, you will want to ensure the quality and accuracy of the agent by testing, analyzing, refining, and retesting expected questions using Q&A Calibration, or through a manual, iterative testing process (or both).
Optimize the semantic model for AI
Use Semantic Model AI Optimization (Beta) to refine the model and help deepen Tableau Agent's understanding of your business. The contextual inputs you provide act as essential guardrails for the agent’s reasoning.
Semantic AI Model Optimization runs a series of automated checks against your semantic model and calculates an overall model health for AI-readiness.
Test the semantic model for accuracy and understanding
Testing is an integral part of preparing a semantic model for use with Tableau Agent. You have two options for testing questions.
- Use Q&A Calibration to save Verified questions (test, classify, and calibrate)
- Manually test large sets of questions
Use Q&A Calibration to save Verified questions
Q&A Calibration (Beta) is a good starting point for testing basic questions in Tableau Agent. Use Q&A Calibration to test, classify, and calibrate questions. The list of verified questions that you compile and save improves agent accuracy for similar questions.
- Collect sample questions
Gather questions from business users, use your own experience, or have an LLM generate a list of likely queries.
- Test and analyze
As you test questions using Tableau Agent in Q&A Calibration, analyze the agent's performance including its query logic and the accuracy of responses for your organization's needs. Evaluate whether the responses meet your user's needs and match how they talk about the business and data in your organization.
- Classify
Classify responses as Verified or Inaccurate. When a response is inaccurate, investigate why. What query did it use? What parts of the query are wrong?
- Calibrate
Make incremental changes in the semantic model. Calibrate the semantic model by updating business preferences, fields, and calculations.
- Retest and refine
Ask the question again in the agent. Observe the impact of these incremental changes on the accuracy of the responses in the agent.
Use manual testing for large sets of questions
When you have a large number of questions to be tested, test the questions directly in Tableau Agent in a supported Tableau Next surface. Use a spreadsheet to record each question and the response results. Improve accuracy through a continuous cycle of testing and refinement.
- Collect sample questions
What do your business users want to ask? Gather questions from your users, use your own experience, or use an LLM to generate a list of likely queries.
- Test and analyze
Use those initial questions as a guide in developing your semantic model. Record the questions, the responses, inaccuracies, and ideas for improvements in a spreadsheet.
Test your sample questions in Tableau Agent against a semantic model in Tableau Next. Observe where the agent response is incorrect, and then try to figure out where the agent has gaps and where you may need to enhance the model.
Analyze if the model is complete. Is it generating the queries you are expecting? Open the Troubleshooting Information details in Tableau Agent to review the query logic. What parts of the query are incorrect?
- Diagnose
Hypothesize why the response or logic might be inaccurate. What changes could you make to the semantic model to improve the response? Identify missing or incorrect data, missing or incorrect fields, missing calculations and dimensions, and incorrect terms during this phase. Could adding business preferences or descriptions improve the response?
- Refine and retest
Continuously refine the model based on testing insights before rolling it out to a pilot user group for ongoing feedback. Make the changes you think will help and retest the same questions. Record responses and determine if accuracy improved or more refinement is needed. Continue this process until the response is accurate for your list of questions.
As you discover commonly asked questions with accurate responses, consider add these questions in Q&A Calibration and mark them as Verified. This helps the agent learn and respond to similar questions more accurately, in language that is tailored to your organization.
