You are here:
Pull Data Directly into Recipes
Direct data for recipes in Salesforce Data Pipelines lets your recipes access data in Salesforce objects that you don’t sync. When you use direct data, the latest data is pulled directly from the Salesforce object with each recipe run, bypassing the data sync step. Direct data is designed to access a small number of objects instead of optimizing for performance as is the case with data sync connected objects. After you decide to use direct data, you can use it in recipes, add filtering, and switch the data to be connected instead.
Pull Data Directly into Recipes
Direct data for recipes in Salesforce Data Pipelines lets your recipes access data in Salesforce objects that you don’t sync. When you use direct data, the latest data is pulled directly from the Salesforce object with each recipe run, bypassing the data sync step. Direct data is designed to access a small number of objects instead of optimizing for performance as is the case with data sync connected objects. After you decide to use direct data, you can use it in recipes, add filtering, and switch the data to be connected instead.
Decide Whether to Use Direct Data or Connected Object Data
There are distinct benefits and limitations to consider when you build recipes for direct data and data sync-connected objects. To achieve faster recipe runtimes, use connected objects instead of direct data when multiple recipes access the same object or when a single recipe accesses multiple objects. Recipes with connected objects run faster because data is synced in advance. They also run faster because multithreading is enabled for connected objects where multiple, simultaneous Bulk API calls are made to process data in parallel. Unlike connected objects, direct data uses single threading, where Data Prep extracts data from one source object at a time. For example, if you have 4 recipes, each with 5 source objects, Data Prep pulls data from one object in one recipe while the other 19 source objects wait in the queue.
| Factor | Direct Data | Connected Object |
|---|---|---|
| Data volume: How many rows of the object’s data are used? | Low: Fewer than 200,000 rows. Supports filtering to reduce pulled rows. Direct data pulls data with each recipe run, with larger data volumes adding to the run time. | High: Greater than 200,000 rows. Supports filtering to reduce the number of rows synced. By separating the sync from the recipe run, higher data volumes can be used without slowing down either step. |
| Access frequency: How often is this data pulled? | Low: One recipe uses this data one time a week | High: Multiple recipes use this data, or it’s used multiple times per day |
| Incremental support: Can only changed and new records be pulled? | None | Most objects (see the Verify the Incremental Sync Settings for Salesforce Data for list) |
| Rate of change: How often do the object records change? | High frequency (recipe can be scheduled up to every 15 minutes, with no reliance on data sync’s schedule) | Low frequency (data sync can be scheduled up to every 15 minutes, followed by the recipe run) |
| Preview size: How many rows do you want to see in data preview? | Max 2,000 rows | Max 10,000 rows |
Use Direct Data in Recipes
When you add an input node to a recipe, you can select direct data in addition to
datasets and connected objects. 
In the Location column, Salesforce direct data is distinguished by a gray Salesforce cloud icon. Salesforce connected objects are distinguished by a blue Salesforce cloud icon.
Filter Data Pulled into Recipes
Filter the data pulled into the recipe by adding a Filter node next to the direct data input node. Use this push-down filter to limit the volume or type of data that is pulled with each recipe run. If you later add this object to data sync, the direct data filter is converted automatically to a standard filter node.
Switch Object Between Direct Data and Connected
A recipe’s object data can be connected data, with data pulled by data sync. Or it can be direct data that’s pulled with each recipe run. To convert an object from direct to connected, add the connection to data sync by following the directions in step 2 of Add and Remove Local Salesforce Objects and Fields from Data Sync. If you had a filter on the direct data input node, it’s automatically converted to a standalone filter node.
To switch an object from connected to direct, remove the object from data sync by following step 4 of Add and Remove Local Salesforce Objects and Fields from Data Sync.
Disable Direct Data for Recipes
- From Setup, in the Quick Find box, enter Analytics, and then select Settings.
- Deselect Allow recipes to use direct data.
- Save your changes.
