You are here:
Data Refresh Process for Extraction
Data refreshes have two phases: full refresh, which replaces all destination data, and incremental refresh, which updates data between full refresh runs. You can perform both full and incremental refresh on demand. Learn about the different refresh scenarios, and the special behavior of ContentVersion and Knowledge_kav objects.
Required Editions
| Available in: All Editions supported by Data 360. See Data 360 edition availability. |
Data Refresh Phases
There are two phases for data refresh (or data sync):
- Full refresh: The entire source data replaces the entire destination data.
- Incremental refresh: Updates are made in between full refresh cycles. Each incremental update extracts data from the start time of the prior extraction to the current time.
All data refresh phases have two stages:
- Extraction of data from the source org.
- Ingestion of data to the Data 360 target org.
Extraction has to succeed before ingestion begins. The data stream Historypanel shows extraction and ingestion errors or successful ingestions. Data 360 considers ingestion as successful when the data lake object (DLO) contains the extracted data. The history panel doesn’t show a successful extraction, as the subsequent ingestion flow encapsulates it into the success message.
Full or Incremental Refresh On Demand
To minimize unnecessary refreshes and data transfers for objects that don’t require regular refresh, you can perform a full or incremental refresh on demand. To refresh a specific data stream on demand, click the Refresh Now button for that stream on the Data Streams tab. This table describes the refresh process for different on-demand request scenarios.
| on-demand refresh request scenario / refresh history setting | batch | streaming | none |
|---|---|---|---|
The data stream is incrementally ingesting since its last periodic full refresh, see Configure Periodic Full Refresh Schedule. The user initiates an on-demand incremental refresh. |
Data 360 ignores the on-demand request. | Data 360 ignores the on- demand request. | Data 360 initiates a new incremental refresh. |
The data stream is incrementally ingesting since its last periodic full refresh, see Configure Periodic Full Refresh Schedule. The user initiates an on-demand full refresh. |
If there's an existing request, Data 360 cancels it and initiates a new full refresh. |
If there's an existing request, Data 360 ignores the full refresh. | If there's an existing request, Data 360 cancels it and initiates a new full refresh. |
The data stream stopped ingesting in the past due to an error, which is now resolved. Ingestion hasn't run for a duration greater than the full refresh interval days, see Configure Periodic Full Refresh Interval. The user initiates an on-demand incremental refresh. |
Data 360 triggers a full refresh instead of an incremental refresh. | Data 360 triggers a full refresh instead of an incremental refresh. | Data 360 triggers a full refresh instead of an incremental refresh. |
The data stream is healthy but hasn't received a request for ingestion for a duration greater than the full refresh interval days, see Configure Periodic Full Refresh Interval. The user initiates an on-demand incremental refresh. |
Data 360 triggers a full refresh instead of an incremental refresh. |
Knowledge Articles and Content Version Objects Unique Refresh Behavior
Content version streaming through ContentVersion objects and knowledge article streaming through Knowledge_Kav objects follow unique refreshing patterns.
Content Version Special Day-Zero Behavior: Unlike the initial data syncing behavior for other objects, Data 360 doesn’t bring all the ContentVersion object data from the CRM org on the initial sync. Instead, it brings the data that gets created or updated after the time the data stream is created, as part of a full refresh.
Knowledge Article Refreshing Behavior:
| SYNC type | behavior and scope |
|---|---|
| Day-Zero | On the initial sync when the data stream is created, Data 360 syncs the latest versions of articles. |
| Incremental Sync |
|
| Full Refresh (Subsequent) | Data 360 syncs Published articles only. Archived articles aren't synced. |
- Configure Periodic Full Refresh Schedule
The periodic full refresh schedule for a Salesforce CRM connector data stream is determined by a number of variables, including a full refresh interval that you can configure to determine when the full refresh is triggered. - Configure Periodic Full Refresh Interval
The periodic full refresh schedule for a Salesforce CRM connector data stream is determined by a number of variables, including a full refresh interval that you can configure to determine when the full refresh is triggered. By default, the full refresh configurable interval is disabled (set as None). You can configure its value during a bundle or multi-object data stream deployment or for an existing data stream.
