DataFlow-Harness: Stop Letting Agents Write Throwaway Scripts
Ask a coding agent to build a data pipeline and you get a Python script. It runs, it works, and then it is a dead artifact nobody can edit, monitor or reuse. Peking University's DataFlow-Harness names this the NL2Pipeline gap and fixes it by making the agent build an actual editable DAG in a real platform instead of emitting code into a file.
The mechanism is a Request-Validate-Commit protocol over MCP tools. The agent does not generate a pipeline, it proposes operators against live platform state, the backend validates schema compatibility between them, and only then does the change commit. There is a synchronized web UI where conversational editing and visual editing look at the same object, plus a DataFlow-Skills layer encoding domain construction patterns the agent would otherwise have to guess at.
The numbers are the argument. 93.3% end-to-end pass rate on a twelve-task benchmark, within 0.9 points of just letting the model write scripts, but at 72.5% lower cost and 49.9% lower latency. That combination is unusual and worth staring at. Grounding the agent in validated platform state is cheaper than code generation, not more expensive, because the agent stops burning tokens re-deriving structure it can query and stops retrying pipelines that were never going to typecheck. The ablation says procedural guidance, the skills layer, contributes the biggest single chunk, four points, and it shows up exactly on tasks needing implicit domain knowledge.
The general lesson goes well beyond data pipelines. The default agent output is code, because code is what the training data rewards, but code is the worst possible handoff format when a human has to own the result afterward. Any domain with a real platform underneath, BI dashboards, ETL, infra, CI config, has this same choice: let the agent write a script that bypasses the platform, or give it MCP tools that make it a first-class user of the platform. The second one is more work to build and it is the one that survives contact with an ops team.
https://huggingface.co/papers/2607.16617
← Back to all articles
The mechanism is a Request-Validate-Commit protocol over MCP tools. The agent does not generate a pipeline, it proposes operators against live platform state, the backend validates schema compatibility between them, and only then does the change commit. There is a synchronized web UI where conversational editing and visual editing look at the same object, plus a DataFlow-Skills layer encoding domain construction patterns the agent would otherwise have to guess at.
The numbers are the argument. 93.3% end-to-end pass rate on a twelve-task benchmark, within 0.9 points of just letting the model write scripts, but at 72.5% lower cost and 49.9% lower latency. That combination is unusual and worth staring at. Grounding the agent in validated platform state is cheaper than code generation, not more expensive, because the agent stops burning tokens re-deriving structure it can query and stops retrying pipelines that were never going to typecheck. The ablation says procedural guidance, the skills layer, contributes the biggest single chunk, four points, and it shows up exactly on tasks needing implicit domain knowledge.
The general lesson goes well beyond data pipelines. The default agent output is code, because code is what the training data rewards, but code is the worst possible handoff format when a human has to own the result afterward. Any domain with a real platform underneath, BI dashboards, ETL, infra, CI config, has this same choice: let the agent write a script that bypasses the platform, or give it MCP tools that make it a first-class user of the platform. The second one is more work to build and it is the one that survives contact with an ops team.
https://huggingface.co/papers/2607.16617
Comments