You are here:
Billing Considerations for Data Federation
Using data federation impacts the consumption of credits used for billing for orgs operating Data 360 under a Data Cloud license.
This feature has access to Digital Wallet, a free account management tool that offers near real-time consumption data for enabled products across your active contracts. Access Digital Wallet and start tracking your org's usage. To learn more, see About Digital Wallet.
| Digital Wallet Card | Usage Type | Usage Type Description | Notes |
|---|---|---|---|
| Data Services | Batch Data Pipeline (External Data Pipeline) | Usage is calculated based on the number of rows batch data processed by Data 360 data streams across all connectors, with the exception of structured data ingested via the Internal Data Pipeline. | If you enable acceleration in Query Federation and turn on caching, data is ingested into Data 360 via the batch data pipeline. |
| Data Services | Data Federation or Sharing Rows Accessed | For data federation, usage is calculated based on the number of records retrieved from the source. For a data share, usage is calculated based on the number of rows returned to fulfill an external data lake’s request. For data sharing between Data 360 orgs, this usage type applies to the target org for all queries. For external data shares, this usage type applies only to cross-cloud and cross-region queries. There’s no credit consumption if the query originates from the same region and the same cloud. |
|
| Data Services | Data Queries | Usage is calculated based on the number of records processed. The count of records processed depends on the structure of a query as well as other related factors such as the total number of records in the objects being queried. |
Data queries are consumed when querying the accelerated cache. |
| Data Storage | Storage Beyond Allocation | Usage is calculated based on the amount of storage used above the amount allocated. | If you enable acceleration in Query Federation and turn on caching, data is stored in Data 360. Storing data can impact your usage of data storage. |
For more information on how Data 360 usage is billed, refer to your contract or contact your account executive.
Reduce Data Federation Costs
You can lower consumption rates for data federation by changing the selecting the right mode and using Acceleration.
- Select the right data federation mode--Determine whether query federation or file federation best suits your use case.
- Turn On Acceleration--If your use case allowed for it, consider enabling
acceleration. Using acceleration allows Data 360 processes to query a local metadata cache
instead of repeatedly querying the external data source. This reduces the cost for data
federation. Using Acceleration also improves performance because queries to the
accelerated data cache provide faster responses. To learn about the differences in
different data federation methods, see Compare Data Federation Methods.
- Review cache refresh frequency--When you enable acceleration, select a cache refresh frequency that matches the frequency that source data changes. Refreshing your cache more frequently consumes credits without providing access to updated records.
- Configure icremental refresh--When you enable acceleration, use the
Incremental column to identify what field in your source data indicates a record has
been updated since the last cache refresh. Using incremental refresh reduces your
credit usage by only refreshing records that have changed.
Incremental refreshes don’t remove deleted records. When using incremental refresh, we recommend periodically performing manual full refreshes to ensure your cache is in sync with the source.
Live Query and Accelerated Query Consumption
Depending on how you’re using data federation, either live queries or accelerated queries can be more cost effective for your situation.
Data federation with acceleration is usually the better choice when your data is accessed frequently but the volume and frequency of changes are low.
For example, suppose you’re building a dashboard that shows sales trends for product categories over the last year for every store in your network. The data is updated daily and the dashboard is viewed an average of 20 times per day. The dashboard is built from query results each time it loads.
Suppose that your dashboard is built with data from 1,000,000 records that are stored in an external data lake. Around 10,000 records are added or updated each day.
In this scenario, your costs are lower when you turn on acceleration. Let's work through the math below.
- Credits Consumed by Query Federation via Live Queries
- Because your dashboard is loaded about 20 times per day, you query your data using live queries about
20 times per day. Each query accesses about 100,000 records. Assume you’re charged at a rate of 70 credits
consumed for each 1 million rows accessed.
- 100,000 records accessed per query x 20 daily queries = 2,000,000 rows accessed daily
- 2,000,000 rows accessed daily x 30 days = 60,000,000 rows accessed over one month
- 60 million x 70 credits per million = 4,200 credits consumed per month
- Credits Consumed by Query Federation with Acceleration
- Suppose you enable acceleration on your data federation queries and populate a local
data cache incrementally. Only new or changed records are added to the cache. Assume
you consume credits at a rate of 2,000 credits per 1 million new or changed rows in
your source data.
During initial setup, you populate the cache with all records in your external data lake object.
- 1 million records x 2,000 credits per million = 2,000 credits for cache initialization
You’ll also use 2,000 credits each time you perform a full refresh.
Assume your cache is updated daily with 10,000 new and changed records per day.
- 10,000 rows updated daily x 30 days = 300,000 rows returned over one month
- 0.3 million x 2000 credits per million = 600 credits per month
On the month you initialize your cache, and for any month you perform one full refresh, you use 2,000 credits. On months after the cache is established and you don’t perform a full refresh, you use 600 credits.
In this scenario, you save between 1,600 and 3,600 credits per month by using acceleration.
Data federation via live queries is best if your data changes frequently and you need the most up-to-date data to work with.
For example, suppose you’re publishing segments 20 times per day. You build your segment from about 1,000,000 records that are stored in an external data lake. Around 10,000 records are added or updated every 30 minutes.
In this scenario, your costs are lower when you use live queries. Let’s work through the math below.
- Credits Consumed by Query Federation via Live Queries
Because you publish segments about 20 times per day, you query your data using live queries about 20 times per day. Each query accesses about 100,000 records. Assume you’re charged at a rate of 70 credits consumed for each 1 million rows accessed.
- 100,000 records accessed per query x 20 daily queries = 2,000,000 rows accessed daily
- 2,000,000 rows accessed daily x 30 days = 60,000,000 rows accessed over one month
- 60 millions x 70 credits per million = 4,200 credits consumed per month
- Credits Consumed by Query Federation with Acceleration
Suppose you turn on acceleration on your data federation queries and populate a local data cache incrementally. You need fresh data, so you update your cache every 30 minutes. Around 10,000 new or changed records are added to the cache with each update. Assume that you consume credits at a rate of 2000 credits per 1 million new or changed rows in your source data.
During initial setup, you populate the cache with all records in your external data lake object.
- 1 million records x 2,000 credits per million = 2,000 credits for cache initialization
You also use 2,000 credits each time you perform a full refresh.
Your cache is updated every 30 minutes, or 48 times per day, with 10,000 new and changed records per day.
- 10,000 rows updated 48 times daily x 30 days = 14,400,000 rows returned over one month
- 14.4 million x 2000 credits per million = 28,800 credits per month
On the month you initialize your cache, and for any month you perform one full refresh, you use 30,800 credits. On months after the cache is established and you don’t perform a full refresh, you use 28,800 credits.
In this scenario, you save between 24,600 and 26,600 credits per month by using live queries.
