From convex conjugates
to variational optimization.
Expectations from samples, convex conjugates, optimization over functions, and empirical divergence objectives.
This continuation follows the remaining Week 1 chapter: expectations from samples, convex conjugates, optimization over functions, and the variational representation of f-divergence. Additional calculations are worked teaching examples, not additional syllabus topics.
Study sections
Follow the order or choose a sectionWhy we use expectations instead of unknown density formulas
The chapter assumes that we can draw samples from the real-data distribution P_X and from the generator-induced distribution P_θ, even…
02Supremum and infimum: the language of optimization
A maximum must be achieved by an element of the set. A supremum is the least upper bound and need not be achieved. Likewise, a minimum…
03Convex conjugates: definition, optimization and geometry
For a function f, its convex conjugate is f*(t) = supᵤ {t u − f(u)}, with the supremum taken over the domain of f. We fix t, vary u, an…
04Why the biconjugate representation matters
For a proper, convex, lower-semicontinuous function f, the biconjugate identity is f(u)=supₜ {t u−f*(t)}. This converts an expression i…
05Why the optimizing scalar becomes a function T(x)
In f-divergence the density ratio p(x)/q(x) depends on the location x. Therefore the optimizing scalar t can also depend on x. Write t=…
06The variational representation of f-divergence
Let P be the real distribution and Q the model distribution. Under the standard regularity and admissibility assumptions, the variation…
07Turning expectations into empirical objectives
For observed real samples x₁,…,xₙ and generated samples x̃₁,…,x̃ₘ, estimate J(T) by Ĵ(T)=(1/n)ΣᵢT(xᵢ)−(1/m)Σⱼf*(T(x̃ⱼ)). This is an em…
08Week 1 synthesis
Real samples reveal the data distribution indirectly. A generator turns random latent inputs into model samples. We want the model dist…