CausalDS: Benchmarking Causal Reasoning in Data-Science Agents

Thu, 09 Jul 2026 00:00:00 +0000

The paper introduces a benchmark generator for causal reasoning in agentic data-science workflows: every benchmark instance is synthetically generated rather than curated. In it, we:

Generate fully synthetic scenes pairing a sampled causal graph and structural causal model with tabular data and natural-language story;

Add an optional observation layer that varies the data-analysis difficulty without changing the causal side;

Derive tasks spanning all three rungs of Pearl’s hierarchy which include non-answerable questions (i.e., testing abstention);

Make the benchmark fully parameterizable, so that it can be tailored to specific evaluation goals or grounded in empirical distributions obtained from real-world corpora.

We thus test the agents along five separate axes of agentic causal data science: symbolic causal reasoning, data science, uncertainty quantification, abstention, and tool use/coding.

We evaluate six contemporary agents — Claude Opus 4.8, Gemini 3.1 Pro, and GPT-5.5, along with the open-weight Qwen 3.6 35B, Kimi K2.6, and Gemma 4 26B — on a 100-task exam with realistically grounded composition. Claude Opus 4.8 leads overall, and the evaluated capabilities dissociate: symbolic causal reasoning is largely mastered across the field, while abstention, uncertainty quantification, and coding/tool-use/reasoning efficiency separate the models.

The paper was featured in Hugging Face Daily Papers on July 10, 2026.

A Bayesian approach to translators' reliability assessment

Tue, 12 Apr 2022 00:00:00 +0000

This grew out of the work I did as a Machine Learning intern at Translated in Rome during the summer of 2021.

Using sparse translation-quality annotations, we develop a Bayesian model that separates translation difficulty, translator skill, and reviewer behavior, including reviewer strictness and consistency. We specifically use fat-tailed zero-inflated distributions to model the phenomenon where reviewers are likely to skim translations and assign zero errors; if they, however, “decide to look”, the error distribution becomes fat-tailed.

Some of this work anticipates the more recent LLM eval literature, which has expanded dramatically since.

Preprint | Andrej Leban

CausalDS: Benchmarking Causal Reasoning in Data-Science Agents

A Bayesian approach to translators' reliability assessment