CausalDS: Benchmarking Causal Reasoning in Data-Science Agents

Thu, 09 Jul 2026 00:00:00 +0000

The paper introduces a benchmark generator for causal reasoning in agentic data-science workflows: every benchmark instance is synthetically generated rather than curated. In it, we:

Generate fully synthetic scenes pairing a sampled causal graph and structural causal model with tabular data and natural-language story;

Add an optional observation layer that varies the data-analysis difficulty without changing the causal side;

Derive tasks spanning all three rungs of Pearl’s hierarchy which include non-answerable questions (i.e., testing absention);

Make the benchmark fully parameterizable, so that it can be tailored to specific evaluation goals or grounded in empirical distributions obtained from real-world corpora.

We thus test the agents along five separate axes of agentic causal data science: symbolic causal reasoning, data science, uncertainty quantification, abstention, and tool use/coding.

We evaluate six contemporary agents — Claude Opus 4.8, Gemini 3.1 Pro, and GPT-5.5, along with the open-weight Qwen 3.6 35B, Kimi K2.6, and Gemma 4 26B — on a 100-scene exam with realistically grounded composition. Claude Opus 4.8 leads overall, and the evaluated capabilities dissociate: symbolic causal reasoning is largely mastered across the field, while abstention, uncertainty quantification, and coding/tool-use/reasoning efficiency separate the models.

New preprint: CausalDS

Thu, 09 Jul 2026 00:00:00 +0000

We recently submitted a new preprint titled CausalDS: Benchmarking Causal Reasoning in Data-Science Agents to ArXiv.

CausalDS is a benchmark generator for causal reasoning in agentic data-science workflows: every benchmark instance is fully synthetically generated, and tests the agents along five separate axes of agentic causal data science.

Joint work with Yuekai Sun.

Agents | Andrej Leban

CausalDS: Benchmarking Causal Reasoning in Data-Science Agents

New preprint: CausalDS