Log-Sum-Exp

📐 Definition

For $x = (x_1,\dots,x_n) \in \mathbb{R}^n$ ,

\operatorname{LSE}(x) = \log \sum_{i=1}^{n} e^{x_i}

To avoid overflow or underflow, use a shift $c = \max_i x_i$ and compute

\operatorname{LSE}(x) = c + \log \sum_{i=1}^{n} e^{x_i - c}, \qquad c = \max_i x_i

Domain and Codomain

Domain: real vectors or arrays. Codomain: real values; extends to complex arguments via the principal logarithm.

⚙️ Key Properties

Invariance to uniform shifts: $\operatorname{LSE}(x + c\mathbf{1}) = c + \operatorname{LSE}(x)$ . Gradients are softmax weights:

\frac{\partial}{\partial x_i} \operatorname{LSE}(x) = \frac{e^{x_i}}{\sum_{j} e^{x_j}}

The function is convex and provides a smooth approximation to $\max_i x_i$ .

🎯 Special Cases and Limits

Dominant entry: if one $x_i$ is much larger than the rest, $\operatorname{LSE}(x)\approx \max_i x_i$ .
Equal entries: if $x_i=a$ for all $i$ , then $\operatorname{LSE}(x)=a+\log n$ .
Two values: $\operatorname{LSE}(x_1,x_2)$ is a smooth max with a soft transition region.

Log-sum-exp is built from the exponential and logarithm and is tightly coupled to softmax (its gradient). It is also related to log-absolute-value as a stabilized log-domain primitive.

Usage in Oakfield

Oakfield does not currently expose a dedicated “log-sum-exp” operator, but the same stabilization pattern shows up in a few places:

Math utilities: core/math_utils.h provides sim_logsumexp2_double and sim_logsumexp2_complex as reusable helpers for numerically stable log-domain accumulation.
Softplus-style clamps: the thermostat operator uses log1p(exp(kx)), which is a special case of log-sum-exp (log(exp(0) + exp(kx))) used for smooth, stable clamping.
Future-facing: these primitives are intended for operator kernels that need stable “smooth max / free energy” style reductions without overflow.

Historical Foundations

📜 Free Energy and Partition Functions

Expressions of the form $\log\sum_i e^{x_i}$ arise naturally in statistical mechanics (log partition functions) and large-deviation/Laplace principles, where they summarize ensembles in a stable log domain.

🌍 Modern Perspective

Log-sum-exp is a standard numerical stabilization primitive in optimization and machine learning, avoiding overflow/underflow while retaining differentiability.

📚 References

Boyd & Vandenberghe, Convex Optimization
Cover & Thomas, Elements of Information Theory