Preprint

CausalDS is a benchmark generator for agentic causal data science: every problem is generated fresh in its entirety, with tasks spanning all three rungs of Pearl’s hierarchy that involve significant tool use. Exam composition is a free parameter: it can be tailored to a specific goal or grounded in real-world corpora. The benchmark jointly tests symbolic causal reasoning, data-science execution, uncertainty quantification, epistemic abstention, and coding/tool use.

Modeling the translation and review processes as zero-inflated fat-tailed distributions, we show how to extract useful information on translators’ reliability with as little as one review per translation.