Built for Agents: The Case for Spark Declarative Pipelines

Databricks has published an argument that Spark Declarative Pipelines, or SDP, are well matched to a workflow in which AI agents author much of the production data pipeline code. The company says agents now write more than 70% of SDP pipelines.
The claim reframes who the primary author of pipeline logic is. In a declarative model, the engineer or agent describes the intended output and transformations, and the engine handles ordering, dependency resolution and execution. That contrasts with imperative pipeline code, where sequencing and orchestration are spelled out by hand. The stated figure is the concrete anchor here: a majority share of SDP pipelines is agent-written.
This suggests the practical appeal is reviewability. Declarative definitions give human reviewers a compact specification to check against agent output, rather than tracing imperative control flow. A likely trade-off is that declarative abstractions can constrain highly custom logic.
Why it matters: teams adopting agents for pipeline authoring will increasingly need conventions for reviewing and validating generated definitions, and platform choices that make agent output easy to audit will shape that work. The 70% figure is Databricks' own, so treat it as a directional signal rather than an independent benchmark.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
AI agents now write a growing share of production data pipelines (over 70% of SDP...