Energy-Tweedie: Score meets Score, Energy meets Energy

Mon, 29 Dec 2025 00:00:00 +0000

Classical Tweedie’s formula links Gaussian corruption, squared-error denoising, the posterior mean, and the score of the noisy data. We present the Energy–Tweedie identity, which generalizes this correspondence from a mean-based relation to a distributional one and holds for any Gibbs (“energy-based”) noise distribution. Each noise distribution induces a kernel scoring rule, whose path derivative evaluated at the denoising posterior gives the noisy-data score. In the Gaussian noise case, this reduces exactly to classical Tweedie’s formula.

The identity has three main consequences:

The score can be estimated from samples from a denoising posterior model (i.e., a conditional generative model).
All the parameters of the noise distribution (within the Gibbs family) can be recovered from corrupted data in a principled fashion.
It provides a score-based perspective on diffusion approaches based on scoring rules; among other consequences, existing score-based samplers can thus be used to generate from such models, with the path through the (multidimensional) noise-parameter space a free design choice at sampling time - illustrated by the MNIST samples below.

MNIST samples generated using various sampling paths through noise-parameter space with the same model.

Distributional Autoencoders Know the Score

Mon, 17 Feb 2025 00:00:00 +0000

The Distributional Principal Autoencoder (DPA) (Shen and Meinshausen, 2024) is a recently introduced class of autoencoders that combines a deterministic encoder with a stochastic decoder trained to reconstruct the full conditional distribution associated with each encoding. Jointly optimizing successive encoding widths produces a principal- component-like ordering of the latent coordinates. Our paper establishes the theoretical structure underlying this method:

the encoder level sets align exactly with the data score in the normal directions, and
extra latent dimensions beyond the data manifold become completely uninformative, revealing the intrinsic dimension.

These hold simultaneously and circumvent the usual reconstruction/disentanglement trade-off in unsupervised learning. Thus, we extend the analogy to PCA made in the original DPA work by proving that, instead of finding principal linear subspaces, DPA learns nonlinear manifolds shaped locally by the data density, with a clear, testable dimensionality criterion — conditional independence.

The first result also leads to strong performance on molecular simulation data: when the data follow a Boltzmann distribution, the learned encoding aligns with the (unknown) force field. We demonstrate that this allows the method to recover an approximation of the minimum free-energy path for the Müller–Brown potential (a common benchmark) in a single fit, with the potential to speed up chemical simulations.

Conference paper | Andrej Leban

Energy-Tweedie: Score meets Score, Energy meets Energy

Distributional Autoencoders Know the Score