At Ascend we see all sorts of different pipelines, from data lake ingestion & data replication, to ETL on API data, to complex unifications of batch & streaming. One pattern we see quite often is that of change data capture (“CDC”) from databases and data warehouses, followed by data set reconstitution. Doing this data set reconstitution usually requires a full reduction — a transform in which you iterate over all records to find those representative of the latest state. This can become inefficient over time, however, as greater and greater percentages of any given data set become “stale”.
Creating recursive transforms is as easy as checking a box. Simply create a new Transform as always, and check the box “Is This Transform Recursive?” and away you go.
The most common need we’ve seen for recursive transforms is in recreating the state of a database table from a CDC stream or individual record change events. That said, recursive transforms also work great for incremental aggregation and sliding windows of data.
Ready to start with recursive transforms? Sign up for a free trial and start building! Want to learn more about this feature and the other parts of the Ascend Unified Data Engineering platform? Schedule a live demo with one of our data engineers. Have a favorite function you’d like to see us add support for? Join us in Slack (#feature-requests) and let us know — we’d love to hear from you!