Exploding variance of means of exponentials: least-squares to the rescue
In short
### Briefing #### Exploding Variance of Means of Exponentials: Least-Squares to the Rescue A common task in machine learning involves estimating or optimizing "log-sum-exp" functions with many terms, such as \(\log \Big( \int_{\mathcal{X}} e^{v(x)} dq(x) \Big)\), where \(v: \mathcal{X} \to \mathbb{R}\) is a potential function and \(q\) is a probability distribution on \(\mathcal{X}\). This technique is widely used in various applications, including normalization, smooth approximations, and regularization in mode…
Key points
- ### Briefing #### Exploding Variance of Means of Exponentials: Least-Squares to the Resc…: ### Briefing #### Exploding Variance of Means of Exponentials: Least-Squares to the Rescue A common task in machine learning involves estimating or optimizing "log-sum-exp" functions with many terms, such as \(\log \Big( \int_{\mathcal{X}
- This technique is widely used in various applications, including normalization, smooth ap…: This technique is widely used in various applications, including normalization, smooth approximations, and regularization in models like transformers and reinforcement learning.
- For instance, in the simplest example with normally distributed random variables \(z_1, \…: For instance, in the simplest example with normally distributed random variables \(z_1, \dots, z_n \in \mathbb{R}\) with mean \(\mu\) and variance \(\sigma^2\), the relative squared error for estimating \(\mathbb{E}[e^z]\) is given by: \[ \