BSNotes.in — Notes for IIT Madras BS StudentsSearch your notes⌘ K
SECTION 02 / 7

The core generative-modeling problem

2 min read
On this page 4 headings

Given samples from an unknown PXP_{X}, our goal is to learn a mechanism that produces samples with a distribution close to PXP_{X}. The example’s setup assumes a parameterized model distribution PθP_\theta , defines a discrepancy D(PX∥Pθ)D(P_{X} \| P_\theta ), and optimizes the model parameters to reduce it.

Figure 08. Generator from latent noise.
Figure 08. Generator from latent noise.
Open image ↗
Illustration: The core generative-modeling problem
θ∗∈arg min⁡θD(PX ∥ Pθ)\theta^* \in \operatorname*{arg\,min}_{\theta} D(P_X\,\|\,P_\theta)

Here argmin returns parameter choices, not the smallest divergence value itself. We should not assume the optimum is reached or that training always succeeds.

Prerequisite refresher

Gaussian noise and the generator

A common starting point is z∼N(0,I)z\sim\mathcal{N}(0,I): a standard multivariate Gaussian with zero mean vector and identity covariance. The neural generator maps latent noise to a synthetic sample:

x~=gθ(z),z∼N(0,I)\tilde{x}=g_\theta(z),\qquad z\sim\mathcal{N}(0,I)
Figure 09. Standard Gaussian latent input.
Figure 09. Standard Gaussian latent input.
Open image ↗
Illustration: The core generative-modeling problem

The probability distribution of x~\tilde{x} is induced by the random input z and the mapping gθg_\theta . It is written PθP_\theta . Sampling means drawing a fresh z and passing it through the trained generator.

Figure 10. Real and generated distributions.
Figure 10. Real and generated distributions.
Open image ↗
Illustration: The core generative-modeling problem
Worked example

What changes when the generator changes?

  1. Let z∼N(0,1)z\sim\mathcal{N}(0,1) and gθ(z)=θ+zg_\theta(z)=\theta+z.

  2. Then x~∼N(θ,1)\tilde{x}\sim\mathcal{N}(\theta,1): changing θ\theta shifts the generated distribution without changing the noise source.

  3. If the target were N(3,1)N(3,1), choosing θ=3\theta = 3 would match this simplified target exactly. Real neural generators are far more complicated.

Worked example

Turning latent randomness into data

Let Z take values −1-1 and +1+1, each with probability 12\frac{1}{2}, and define gθ(z)=θ+2zg_\theta(z)=\theta+2z. If θ=3\theta = 3, the generator outputs 1 or 5, each with probability 12\frac{1}{2}. Thus PθP_\theta is a distribution over {1,5}\{1,5\}. If θ\theta changes to 4, the possible outputs become 2 and 6. This tiny example shows how a fixed source of randomness plus a parameterized mapping induces a model distribution.

Try it yourself

Check yourself

If gθ(z)=θ+zg_\theta(z)=\theta+z and z∈{−1,+1}z\in\{-1,+1\} with equal probabilities, what does the model generate when θ=10\theta = 10?

Show solution

9 and 11, each with probability 12\frac{1}{2}.

YOUR STUDY PROGRESS

Finished this section? Save your progress.

Back to top ↑