You are here:
Data Processing Engine Limits and Considerations
When you create and run Data Processing Engine definitions, stay within the limits for your runtime and consider the applicable restrictions. Some limits apply to all runtimes, while others are specific to CRM Analytics or Data 360.
Required Editions
| View supported editions. |
Runtime Applicability
This table shows which sections in this topic apply to each runtime.
| Section | CRM Analytics Runtime | Data 360 Runtime |
|---|---|---|
| Data Processing Engine Definition Limits | Yes | Yes |
| Flow Limits | Yes | Yes |
| Data Pipelines Limits | Yes | No |
| Notification Alerts | Yes | No |
| Data 360 Limits | No | Yes |
| Writeback Limits | Yes | Yes |
| Failed Records File Limit | Yes | Yes |
Data Processing Engine Definition Limits
When you design a definition, stay within these limits. Contact your Salesforce representative to increase any of the Data Processing Engine definition limits.
- A Salesforce org can have up to 50 active definitions.
- A definition can have 50 unique data source objects across definitions.
- A definition can have up to 500 nodes.
- A node in a definition can have up to 500 fields.
- A definition can have up to 50 Data Source nodes.
- A Filter node can have up to 50 filter conditions.
- A Group and Aggregate node can group fields by up to ten fields.
- Case statement conditions can’t exceed 1,600 characters cumulatively across all nodes in a definition.
Flow Limits
Review Flow limits and considerations before you create a flow to run a definition. For more information, see Flow Limits and Considerations.
Data Pipelines Limits
Monitor your org's daily and monthly Data Pipelines usage limits when you run Data Processing Engine definitions with the CRM Analytics runtime. Definitions don't run if your org exceeds these usage limits. Contact your Salesforce representative to increase the Data Pipelines limits.
| Limit | Value | Notes |
|---|---|---|
| Maximum number of writeback nodes per Data Processing Engine definition | 20 | N/A |
| Maximum number of users assigned to the Data Pipelines Base User permission set | 3 | N/A |
| Maximum number of rows in all datasets | 250 million | N/A |
| Maximum number of rows processed | 10 million per month | N/A |
| Maximum processing time per month | 30 hours | The cumulative minutes taken by the recipes of all Data Processing Engine definition runs to transform data. The recipe job runtime doesn't consider the time taken to sync data for recipe transformation and export results for writeback. |
| Maximum number of concurrent recipes | 1 | Data Processing Engine executes only one recipe at a time per license even if multiple jobs are triggered at the same time. Purchase additional licenses if you must process more than one recipe at a time. |
| Maximum amount of data written back in a rolling 24-hour period to Salesforce | 1 GB | Data Processing Engine assesses this limit when a job starts. If the limit is exceeded, the engine doesn’t start the job. If the limit isn’t exceeded, the engine starts and finishes the job even if the job writes data that results in the limit being exceeded. When the limit is exceeded, jobs fail until the amount of data written in the preceding 24 hours is below the limit. The amount of data that's written back by Data Processing Engine runs counts against the Data Pipelines Output to CSV limit, not the Data Pipelines Output to Salesforce limit. If your org reaches the 1 GB Output to CSV limit, work with your Salesforce account executive to purchase add-ons. Each add-on adds 1 GB to your org's Output to CSV limit. If your org requires 5 GB each day, you must get 4 add-ons. |
| Maximum number of recipes | 1000 | N/A |
| Maximum number of objects enabled for data sync | 50 | N/A |
| Maximum number of concurrent data sync runs | 2 | N/A |
| Maximum average number of values per multivalue field | 100 | N/A |
Notification Alerts
These limits apply to CRM Analytics runtime only and tie to Data Pipelines license entitlements. Admins receive notification tray alerts when Data Pipelines limits exceed 80% of capacity.
- Rows in All Datasets
- Recipe Job Runtime
- Output to CSV
- Recipe Rows Out
If you receive a notification, review your Data Processing Engine definitions and reduce data volume or processing time to stay within the limits.
Data 360 Limits
Track your org's daily and monthly usage limits for Data 360 when you run Data Processing Engine definitions using the Data 360 runtime. Definitions don't run if your org exceeds these usage limits. Contact your Salesforce representative to increase the Data 360 limits.
| Limit | Value |
|---|---|
| Maximum number of CSV data source nodes in a Data Processing Engine definition | 5 |
| Maximum number of writeback nodes in a Data Processing Engine definition | 5 |
| Maximum size of CSV file that can be uploaded | 75 MB |
Writeback Limits
Data Processing Engine writes results back to your Salesforce object and fields by using APIs. Writebacks are counted against your org's Bulk API 2.0 limits.
