You are here:
Create a Data Pipeline
Build a data pipeline using a block-based builder. You extract data from sources, shape it using component blocks, and format it into an event log using a process log block.
Required Editions
| Available in: All Editions of Salesforce supported by Apromore. |
| User Permissions Needed | |
|---|---|
| To create a data pipeline: | Edit data pipelines permission |
| To connect Data 360: | Data Cloud Architect permission set |
If your instance doesn't have Agentforce Operations or Salesforce Clouds connected, those specific pipeline options aren't available.
To learn how each merge type combines data and how rules apply conditions and transformations, see Merge and Transform Operations.
You build a pipeline in the Pipeline creation window.
Extract Data
Extract one or more datasets to use in your pipeline.
- In the Apromore Portal, click Pipelines.
- Click Add pipeline, and select From scratch from the dropdown.
-
In the Create pipeline from scratch window, enter a pipeline name
and an optional description.
A new tab opens with an empty canvas.
-
In the pipeline data left panel, click the Add + icon, and add a
dataset.
- To add a local file, click Upload file and select a CSV file.
- To add a dataset from Data 360, click Data Cloud, select the files, and click Select.
-
Preview the data to verify the column headers, and then click Upload
all.
If you have access to Data 360, you can use it to connect and extract data from many other external sources. You can then extract that data into Apromore using the Data 360 connector.
Source file names support only alphanumeric characters and underscores. Apromore converts spaces to underscores and ignores other characters.
Add a Component Block
Add a component block to clean, reshape, and combine your extracted datasets into the process log format that Apromore uses.
Extract at least one dataset before you add a component block. A component block bridges your extracted datasets with the final process log. You can chain multiple component blocks in a single pipeline to build complex data flows.
- In the pipeline canvas, click Add block, and select Component.
-
Hover over the component block to access its actions.
- To rename the component, click the Rename icon.
- To delete it, click the Delete icon.
- To open the component editor, click the Edit icon.
-
In the component editor, click the + icon or inside the
editor, and select your dataset objects.
You can add more components or data pipelines and merge the data.
- To apply rules and map columns, click the Transform tab.
Merge Data
Combine two or more datasets to consolidate your data sources. If you only need to clean or reshape a single dataset, skip this step and go to Transform Data.
You merge when you build an event log from more than one source. You transform to map your source columns to event log fields and to clean or reshape the data. For the available merge types, conditions, and transformations, see Merge and Transform Operations.
-
Open a component block.
Note You need at least two datasets in the merge canvas to perform a merge. You can't connect the same dataset multiple times in one component block. If you only need to clean or reshape a single dataset, skip to the Transform step. -
Add your first dataset to the canvas.
- Click the + icon on the right side of the dataset in the left panel.
- Drag and drop the dataset from the left panel onto the canvas.
- Right-click the empty canvas and select a dataset.
-
Add your second dataset to the canvas.
- To merge manually, draw a line from one dataset to the other to trigger the merge operator window.
- To merge automatically, hover over the first dataset on the canvas, click the + icon on its right side, and select the second dataset from the list. The datasets connect automatically, and the configure merge operator window opens.
-
In the configure merge operator window, select an operator from the
Operator dropdown list.
For the available merge types, conditions, and transformations, see Merge and Transform Operations.
- Review the merge operation data in the Preview merge operation section.
- Save your changes.
Transform Data
Apply operations to map, clean, and reshape your data inside a component block. You can transform a single dataset or the output of datasets you have already merged.
- In the component editor, click Transform.
-
Select the columns to include in your output.
- To add a specific column, click the + icon next to it.
Note Clicking the + icon repeatedly adds the same column multiple times and creates duplicate columns. - To add all columns from a dataset, click the + icon next to the dataset name.
- To edit a column name, click the Pencil icon or double-click the column header.
- To remove a column, click the X icon.
- To reset the table, click Clear all.
- To add a new column, click Add new, enter a name, and then click Create.
- To add a specific column, click the + icon next to it.
- To add a rule to a column, click the Function icon in the column row.
-
Select Direct to apply an operation directly, or select
Conditional to build an IF/ELSE rule.
For the available operations, see Merge and Transform Operations.
-
Click Apply on the main canvas to save all your component
changes.
Note If your data contains null values in the Case ID or Activity columns, the pipeline doesn't fail. Instead, Apromore skips those rows as corrupted data and displays a warning report in the portal.
Add a Process Log Block
Connect a process log block to the final component in your pipeline and tag the required semantic columns so Apromore can format your event log.
- Hover over the final component block, click the + icon, and select Process Log.
-
Click the process log block to edit it.
Note If a process log block isn't connected to a component, the tagging option is unavailable. -
Tag the required columns.
Set Case ID to Case, Activity to Activity, and End_Time to End Timestamp. End Timestamp is mandatory.Column names in the source data must be unique. If two columns share the same name, rename one before tagging.
- Optionally, tag additional event or case attributes, such as Loan_Amount or Country.
- Click Apply to save your changes.
-
Run or schedule the pipeline.
- To run the pipeline immediately, enter a name in the Log name field and click Upload log.
- To schedule the pipeline to run automatically, click Schedule pipeline in the top right toolbar.
When the pipeline runs, a confirmation message appears. If the run fails, an error message appears with an error ID for customer support. If the run imports but some rows fail, you download an error report that lists the failing rows and the reason for each failure.
