At Ascend we see all sorts of different pipelines, from data lake ingestion & data replication, to ETL on API data, to complex unifications of batch & streaming. One pattern we see quite often is that of change data capture (“CDC”) from databases and data warehouses, followed by data set reconstitution. Doing this data set reconstitution usually requires a full reduction — a transform in which you iterate over all records to find those representative of the latest state. This can become inefficient over time, however, as greater and greater percentages of any given data set become “stale”.
“With Ascend’s new recursive transformations, we were able to more easily take the latest state of our records and reduce processing costs by 70%.“
To that end, we’re excited to announce Recursive Transforms! A recursive transform is a transform that uses the output of its previous run as an input into the next run. This pattern is often used to incrementally aggregate data. In cases where historical data is substantially larger than the aggregated data, this pattern can result in significant reduction in processing time and compute resources.
Creating recursive transforms is as easy as checking a box. Simply create a new Transform as always, and check the box “Is This Transform Recursive?” and away you go.
The most common need we’ve seen for recursive transforms is in recreating the state of a database table from a CDC stream or individual record change events. That said, recursive transforms also work great for incremental aggregation and sliding windows of data.
Ready to start with recursive transforms? Sign up for a free trial and start building! Want to learn more about this feature and the other parts of the Ascend Unified Data Engineering platform? Schedule a live demo with one of our data engineers. Have a favorite function you’d like to see us add support for? Join us in Slack (#feature-requests) and let us know — we’d love to hear from you!