The status quo for building data pipelines is painful and cumbersome. Sure, building the first couple isn’t too bad…it’s the many that follow. And how they impact each other. And how they become harder and harder to maintain as the dependencies grow and the codebase becomes increasingly unruly. For us, this was a terminal path for our happiness as engineers.

To solve this, there are two options: Frameworks and Control Systems. Most other companies went the framework (or imperative)route. They are great at making it easier to build at a static point-in-time. But they lack a feedback loop, which is critical for dynamic systems.

Dynamic systems, like data pipelines, are where control planes (or the declarative approach) shine. Instead of the “when X happens, do Y” framework mentality, a control plane takes the approach of “no matter what happens, make the system look like Z.” As engineers, we wanted the latter. So we built the Dataflow Control Plane.