Retrieve Data

Data preparation for retrieval from a search index involves loading, chunking, vectorizing, and storing content in a search-optimized way. Data 360 uses a vector data store for unstructured data, such as documents and conversation histories. A search index stores chunked and vectorized data that can be searched and retrieved from other applications. You can create and manage search indexes in Data 360. They’re defined in a data space and associated with a Data Model Object (DMO).

With advanced document processing and indexing capabilities in Data 360, retrievers can retrieve and ground agent responses by processing both text and images. Enable image processing for the unstructured data model object search indexes to process documents with images and other visuals such as tables or flow charts. For more information on advanced set up for search indexes, see Use Search for AI, Automation, and Analytics.

Search indexes are often designed for general usage, such as indexing an entire knowledge base. You use retrievers to conduct specialized searches on search indexes and provide relevant results to agents, prompt templates, flows, and other features. Individual retrievers for search indexes apply the multi-purpose search index to a particular use case. For example, retrievers can return content that’s filtered based on a category of articles or the most recent information (such as the last 90 days), a geographical region.

You build retrievers in AI Models and then activate it to make it available to Prompt Builder users. In Prompt Builder, you ground a prompt template with knowledge by inserting and configuring a retriever. At run time, the retriever provides the LLM with accurate, current, and pertinent information to improve the relevance, accuracy, and value of LLM responses.

Retrievers adhere to the data governance policies set up for your data. Row-level access checks make sure that users have access to the data and knowledge returned by the retriever.

When you create a retriever in a Data 360 home org, you can use it in the companion org within Prompt Builder and Flow. For more information see, Data Cloud One Companion Connections.

Individual Retrievers

Use individual retrievers to retrieve data from from a search index. To create an individual retriever in AI Models, you select its search index, define filters, and specify what information it returns. Filters narrow the search focus to more relevant data. The search index defines the fields you can use for filtering. You can also configure the data returned to the prompt, including the fields of data to return and the number of results. Each time you edit and save an individual retriever, AI Models creates another version. Only one version of a retriever can be active.

Note Data 360 no longer creates a default Retrievers automatically when you create a search index. Previously, when you created a search index in Data 360, a default retriever was created automatically. You can use existing default retrievers in applications, such as prompt builder or flow.

• Get Trusted AI Responses with Retriever Citations
  Citations link AI-generated responses to the source content relevant to each response. Citations allow users to compare the LLM response with the source data to verify the validity of the response and identify any potential inaccuracies or hallucinations. Enable citations in AI Models (formerly Einstein Studio) at the individual retriever level to give access to citations to solution components that call the retriever, such as a prompt template.
• Manage Retrievers
  Create and customize retrievers for Retrieval Augmented Generation (RAG) in Data 360. To see the list of retrievers, navigate to the Retrievers tab in AI Models (formerly Einstein Studio). To learn about a retriever, select View Details from the dropdown menu. Each time you edit a custom retriever and save the changes, a new version is created. You can create multiple versions, but you can have only one version active at a time.
• Customize, Test, and Validate Retrievers with Retriever Playground
  Fine-tune and validate retrieval logic for retrieval-augmented generation (RAG) systems by using retriever playground, reducing deployment risks, and verifying accurate AI outputs. Use AI Models (formerly Einstein Studio) retriever playground to configure pre-filters, select test data, and adjust result parameters to customize retrieval logic for domain-specific use cases, such as retrieving complaince documents or knowledge articles.
• Retriever Filters Reference
  Configure filter conditions for retrieval accuracy when you create or test retrievers in AI Models (formerly Einstein Studio). Use filter conditions to filter records retrieved for search queries. You can configure up to ten filter conditions for an individual retriever.