Best Practices for Building Batch Data Transforms

To increase processing performance and the speed that data appears in Preview, keep these considerations in mind.

• Don’t use more than 50 join nodes at a time.
• Avoid exploding joins—when a join produces many more records than the right or left source.
• Select only the required columns on an input node. You can add more columns as needed.
• Break up batch transforms that contain many nodes into smaller data transforms. The output of the previous data transform becomes the input to the next transform. You run them in sequence.
• Don’t include more than 250 nodes. A batch data transform that has more than 150 nodes takes longer to preview and run.

To increase speed on the canvas:

• Use the cleanup button to reduce the size of your data transform on the canvas. The cleanup button organizes branches horizontally with inputs on the left and outputs on the right. It minimizes overlapping branches and shortens connections as much as possible.
• Use keyboard shortcuts instead of a mouse to move between elements and add them to the canvas.
• To locate a node, use search or zoom instead of scrolling.