The current state of the art in orchestration are workflow automation systems like Airflow. Imperative by design, you are responsible for designing the tasks, connecting them together via dependency relations, and passing them to a compute and infrastructure team/layer that manages the execution. This model puts the manual burden on you, the developer, to manage the innumerable details within (and between) each task and ensure data is correct, often resulting in brittle pipelines that are expensive to run and maintain.

There is an alternative: dataflow automation systems that are Declarative in design. These systems leverage a sophisticated Control Plane that is non-existent in Imperative systems to translate high-level data specs into the myriad of tasks required to achieve the specification, and then schedule, size, execute, and manage those tasks on behalf of the developer. You’re only responsible for creating and curating the high-level spec, not HOW the dataflow automation system manages the legion of tasks required to implement your spec.