CausalDS: Benchmarking Causal Reasoning in Data-Science Agents
Mean CausalDSScore versus tokens per task (log scale) on the realistic and hard exams.Abstract
Large language models (LLMs) increasingly act as data-science agents, combining abstract reasoning with advanced tool use. Yet the relevant benchmark landscape largely divides into symbolic causal reasoning benchmarks without realistic data analysis or data analysis benchmarks without causal data-generating structure or a range of causal data science tasks. Existing causal evaluation datasets are often restricted to curated examples from existing sources, with diversity coming from limited templatized variations rather than from systematic generation of novel synthetic causal structures. We introduce CausalDS, a benchmark for evaluating causal reasoning in agentic data-science workflows. Each benchmark instance is a scene consisting of a sampled structural causal model (SCM) with generated observational data and an accompanying synthetic natural-language story grounded in a real domain. We ground the composition of the benchmark components in empirical distributions obtained from real-world datasets, thus retaining empirical structure while reducing the memorization risk through completely synthetic generation. From each scene, we then derive tasks spanning all three of Pearl’s rungs. Most tasks include a data science coding component, where the model typically needs to use several tools to arrive at the final answer due to imperfect observations. Abstention is tested directly: for targets the graph does not identify, the graded answer is to abstain. The benchmark thus jointly evaluates symbolic causal reasoning, data science, uncertainty quantification, abstention, and tool use/coding. Because the generator is fully synthetic, the exam is not a fixed dataset: composition, difficulty, and size are parameters, and fresh exams can be drawn at will.
The paper introduces a benchmark generator for causal reasoning in agentic data-science workflows: every benchmark instance is synthetically generated rather than curated. In it, we:
- Generate fully synthetic scenes pairing a sampled causal graph and structural causal model with tabular data and natural-language story;
- Add an optional observation layer that varies the data-analysis difficulty without changing the causal side;
- Derive tasks spanning all three rungs of Pearl’s hierarchy which include non-answerable questions (i.e., testing abstention);
- Make the benchmark fully parameterizable, so that it can be tailored to specific evaluation goals or grounded in empirical distributions obtained from real-world corpora.
We thus test the agents along five separate axes of agentic causal data science: symbolic causal reasoning, data science, uncertainty quantification, abstention, and tool use/coding.
We evaluate twelve contemporary agents — GPT-6 Astra, Claude Fable 5.1, Claude Opus 4.8, Gemini 3.1 Pro, and GPT-5.5, along with the open-weight GLM 5.3, DeepSeek V4.1 Flash, Kimi K3, K2-MoVA 36B-A4B, Qwen 3.6 35B, Kimi K2.6, and Gemma 4 26B — on a 100-task exam with realistically grounded composition and a 200-task hard exam, with three complete runs per model on each. GPT-6 Astra leads the realistic exam and GLM 5.3 leads the hard exam by CausalDSScore, and the evaluated capabilities dissociate: graph recovery is largely mastered across the field, while abstention, uncertainty quantification, and coding/tool-use/reasoning efficiency separate the models.
The paper was featured in Hugging Face Daily Papers on July 10, 2026.
