I am currently focusing primarily on improving diffusion-based language modeling and have worked on discrete diffusion and flow matching previously.1
Among published work, some of the more prominent research directions are:
Scoring rules measure the quality of probabilistic forecasts, such as predictions of tomorrow’s chance of rain; the standard reference is Gneiting and Raftery (2007). They are increasingly used to train generative models, my research examines their fundamental properties:
a nonlinear-PCA analogue
For the Distributional Principal Autoencoder (Shen & Meinshausen ‘24), I prove:
Neither of these generally hold for other nonlinear methods.
A general form of Tweedie’s identity
The Energy-Tweedie identity generalizes the Gaussian Tweedie identity into a distributional correspondence between the denoising posterior and a scoring rule induced by the noise distribution. Among other applications, this provides the score-based perspective on recent “distributional” diffusion models, permitting sampling along user-chosen paths in noise parameter space.
A fully customizable causal benchmark generator
CausalDS is an end-to-end synthetic benchmark generator that jointly tests along five axes: symbolic causal reasoning, data-science, uncertainty quantification, epistemic abstention, and coding/tool use skills. The fully synthetic structure is crucial for distinguishing “real causal reasoning” from repeating associations present in existing data (the “causal parrot” failure mode).
Through Project CETI, which aims to further our understanding of sperm whale communication, I have worked on:
Using generative models to identify potential carriers of meaning
We introduce a method to interpret representations learned by GANs, which helps us identify several potential carriers of meaning in sperm whale communication, including specific acoustic properties; acoustic properties had previously not been considered meaningful.
AI-suggested hypothesis verified
Following the clues given by Approaching …, we identify spectral patterns in sperm whale codas that are reminiscent of components of human speech. The patterns recur across whales, and appear to be actively controlled; this makes them promising candidates for carriers of meaning in sperm whale communication.
The work has received substantial media coverage.