Comparison of the denoising field early in the diffusion process for Gaussian vs. fat-tailed noise. Denoising posterior in green, true distribution in red, noisy samples in blue.Abstract
Denoising and score estimation have long been known to be linked via the classical Tweedie’s formula. In this work, we first extend the latter to a wider range of distributions often called “energy models” and denoted elliptical distributions in this work.
Next, we examine an alternative view: we consider the denoising posterior $P(X|Y)$ as the optimizer of the energy score (a scoring rule) and derive a fundamental identity that connects the (path-) derivative of a (possibly) non-Euclidean energy score to the score of the noisy marginal.
This identity can be seen as an analog of Tweedie’s identity for the energy score, and allows for several interesting applications; for example, score estimation, noise distribution parameter estimation, as well as using energy score models in the context of “traditional” diffusion model samplers with a wider array of noising distributions.
Classical Tweedie’s formula links Gaussian corruption, squared-error denoising, the posterior mean, and the score of the noisy data. We present the Energy–Tweedie identity, which generalizes this correspondence from a mean-based relation to a distributional one and holds for a wider family of noise distributions. It connects the noisy-data score to the path derivative of a non-Euclidean energy score (a scoring rule). In the Gaussian noise case, this reduces exactly to classical Tweedie’s formula.
The identity has three main consequences:
- The score can be estimated from samples from a denoising posterior.
- The identity can be used to recover the parameters of the noise distribution.
- It provides a score-based perspective on diffusion approaches based on scoring rules; among other consequences, existing score-based samplers can thus be used to generate from such models.
