You are here:
Billing Considerations for Unstructured Data and Search Index
When you use unstructured data and search index configurations, your data is stored and processed in Data 360. Use of Data 360 features for unstructured data has billing implications. Use of Data 360 services impacts the consumption of credits used for billing. There are four billing components for unstructured data in Data 360: data ingestion, data storage, data processing, and data queries. Each component has a distinct applicable usage type. Note that there are two types of unstructured data connectors: connectors that reference data that resides on an external data source, and connectors that ingest data from an external data source into Data 360. Billing works differently for the two connector types.
This feature has access to Digital Wallet, a free account management tool that offers near real-time consumption data for enabled products across your active contracts. Access Digital Wallet and start tracking your org's usage. To learn more, see About Digital Wallet.
| Digital Wallet Card | Usage Type | Usage Type Description | Notes |
|---|---|---|---|
| Data Services | Batch Data Pipeline (External Data Pipeline) | Usage is calculated based on the number of rows batch data processed by Data 360 data streams across all connectors, with the exception of structured data ingested via the Internal Data Pipeline. | Usage is calculated based on the number of records that are ingested either in batch or streaming mode by data streams. Unstructured data may be ingested through any Data 360 connector. One of these usage types is used depending on the ingestion pattern used (batch or streaming). For unstructured data files that are only referenced - and not ingested - from an external blob store, such as Amazon S3, there is no ingestion cost. |
| Data Services | Streaming Data Pipeline (External Data Pipeline) | Usage is calculated based on the number of rows of streaming data processed by Data 360 across all data streams with stream processing, with the exception of structured data ingested via the Internal Data Pipeline. Data streams that report usage with this usage type include streams created by the Website and Mobile App connector and streaming ingestion API. |
Usage is calculated based on the number of records that are ingested either in batch or streaming mode by data streams. Unstructured data may be ingested through any Data 360 connector. One of these usage types is used depending on the ingestion pattern used (batch or streaming). For unstructured data files that are only referenced - and not ingested - from an external blob store, such as Amazon S3, there is no ingestion cost. |
| Data Services | Unstructured Data Processed | Usage is calculated based on the amount of unstructured data that is processed. For example, if the search index processes 100 PDF documents that are 1 MB each, usage is calculated as 100 MB. If the search index processes five audio/video files that are on average 100MB each, usage is calculated as 500MB. In Data 360, unstructured data may be chunked and vectorized using an embedding model. Usage is computed only once across both these activities. For example, if one 100 MB PDF document is chunked and vectorized, usage is computed as 100 MB, not as 200MB. |
In Data 360, unstructured data may be chunked and vectorized using an embedding model. For audio and video files, a text transcript is created before the files are chunked. Usage is computed only once across all these activities. For example, if one 100 MB PDF document is chunked and vectorized, usage is computed as 100 MB, not as 200MB. If a video file of 1GB is transcribed, chunked, and vectorized, usage is computed as 1GB. A DMO and all its file attachments are treated as a single unit for processing purposes. For example, if Data 360 processes incremental changes either to the fields on a source DMO or to a file attachment on that DMO, all file attachments are reindexed. The cost of creating a search index remains the same for vector search and hybrid search. |
| Data Services | Intelligent Processing | Usage is calculated based on the amount of unstructured data that is processed using AI-assisted features such as LLM-based parsing, LLM-based visual data preprocessing, image processing, and Intelligent Context. | When LLM-based parsing is used, entire documents are sent to the LLM for processing, and size of all documents is reported against "Intelligent Processing” usage type. When LLM-based visual data preprocessing is used, only content that contains visual elements or tables is sent to LLM for processing. Size of all documents from which any content is sent to the LLM for processing is reported against "Intelligent Processing” usage type. The sizes of all other documents from which no content is sent to the LLM for processing are reported against “Unstructured Data Processed” usage type. The same guideline applies for documents that are uploaded and indexed in Intelligent Context. |
| Data Services | Accelerated Data Queries | As of August 16, 2024, usage is no longer billed to this category. | |
| Data Services | Data Queries | Usage is calculated based on the number of records processed. The count of records processed depends on the structure of a query as well as other related factors such as the total number of records in the objects being queried. |
For vector search queries against unstructured data, the number of vectors in the search index are counted. For hybrid search queries against unstructured data, the number of vectors and keyword records in the search index are counted. In a typical search index, the number of keyword records is the same as the number of vectors. |
| Data Storage | Storage Beyond Allocation | Usage is calculated based on the amount of storage used above the amount allocated. | Every file ingested and table created, including unstructured data lake objects (UDLO), unstructured data model objects (UDMO), CDMO, or index data model objects (index DMO), count toward Data 360 data storage, including the following.
|
| Einstein Requests | Standard Prompts Basic Prompts Advanced Prompts |
Usage is calculated based on two factors: the number of direct requests to the LLM via the LLM gateway, and whether the gateway uses a Salesforce managed large language model. The specific category depends on the model that is used. See Large Language Model Support to find out which usage types apply. All Standard, Basic, and Advanced prompts process up to 2,000 tokens per prompt. Token usage is rounded up in 2,000-token increments. All Standard, Basic, and Advanced prompts that exceed this limit will be metered as multiple prompts, with each additional 2,000-token chunk counting as a new prompt. For example, a prompt with a total of 6,500 input and output tokens will be metered as 4 prompts. Tokens are units of data processed by the AI models. |
For Document AI processing, LLM gateway calls are counted as Standard Einstein Request prompts. |
| Flex Credits | Standard Prompts Basic Prompts Advanced Prompts |
Usage is calculated based on two factors: the number of direct requests to the LLM via the LLM gateway, and whether the gateway uses a Salesforce managed large language model. The specific category depends on the model that is used. See Large Language Model Support to find out which usage types apply. All Standard, Basic, and Advanced prompts process up to 2,000 tokens per prompt. Token usage is rounded up in 2,000-token increments. All Standard, Basic, and Advanced prompts that exceed this limit will be metered as multiple prompts, with each additional 2,000-token chunk counting as a new prompt. For example, a prompt with a total of 6,500 input and output tokens will be metered as 4 prompts. Tokens are units of data processed by the AI models. |
For search indexes that use enriched indexing, calls to the LLM to generate enriched chunks are counted as Standard Prompts. For more information see, Flex Credits Billable Usage Types. |
Unstructured data and search index are not available for orgs operating Data 360 under the Customer Data Platform (CDP) license. For more information on how Data Cloud usage is billed, refer to your contract or contact your account executive.
