Streaming Data Transforms

A streaming data transform reads one record in a source object, reshapes the record data, and writes one or more records to a target object. The source and target objects must be different objects. A streaming data transform runs continuously as a streaming process, picking up new or changed data.

Use a streaming data transform to clean ingested data. Then, map the improved data to the data model.

• Streaming Data Transform Use Cases
  These use cases show how you can use streaming data transforms to normalize your data, filter it, and better align it with the Data Cloud data model.
• Set Up a Streaming Data Transform
  When you set up a streaming data transform, it reads and processes all records in the source and writes the data to the target object. The source and target objects must be different objects.
• Join Data in a Streaming Transform
  Joins enable data augmentation by combining data from two separate streams based on a primary key. In streaming data transformations, `INNER JOIN` and `LEFT JOIN` clauses in SQL statements are used to combine data from two distinct streams. A join expression consists of two objects: one identifies the object that triggers the streaming transform on update, and the other specifies the object used to evaluate the join.
• Create a Target DLO from the Data Lake Objects Tab
  One way to create a new Target Data Lake Object (DLO) is via the Data Lake Objects tab.
• Check the Status of a Streaming Data Transform
  To monitor the status of a streaming data transform, review its processing metrics.
• Filter Streaming Data Transform Records
  Use the WHERE clause in the streaming data transform SQL statement to limit which source object records the streaming data transform reads and processes.
• Rebuild Streaming Transform Target Objects
  Rebuild a streaming transform target object to ensure data consistency.
• Considerations for Working with Streaming Data Transforms
  Considerations for working with streaming data transforms can significantly affect the performance, reliability, and success of your data processing pipeline.
• Considerations for Deleting Data
  Keep the following in mind when you delete data in Data 360.
• Customer Data Subject Rights in Data Transforms
  A customer’s Data Subject Rights relate to the storage and use of their personal data. Data 360 offers tools to support Data Subject Rights in alignment with various data protection and privacy regulations.
• Streaming Data Transform Functions and Operators
  You can use these functions and operators in a streaming data transform SQL statement.