BSNotes.in — Notes for IIT Madras BS StudentsSearch your notes⌘ K
SECTION 04 / 7

Detour D — Optimization, minimum and argmin (detailed)

2 min read
On this page 3 headings
Figure 16. Generator learning loop.
Figure 16. Generator learning loop.
Open image ↗
Illustration: Detour D — Optimization, minimum and argmin (detailed)

An objective is a number we want to make small (or large). A minimum is the smallest attained objective VALUE. An argmin is the INPUT, or set of inputs, where that value is attained. A supremum is the least upper bound and need not be attained; we will explore it in detail in Part 2.

Figure 13. An optimization objective.
Figure 13. An optimization objective.
Open image ↗
Illustration: Detour D — Optimization, minimum and argmin (detailed)
Worked example

Minimum versus argmin

  1. Take

    L(θ)=(θ−2)2+1L(\theta)=(\theta-2)^2+1

    Since every square is nonnegative, L(θ)≥1L(\theta ) \geq 1.

Figure 14. Minimum versus argmin.
Figure 14. Minimum versus argmin.
Open image ↗
Illustration: Detour D — Optimization, minimum and argmin (detailed)

  1. Equality holds only when θ−2=0\theta - 2 = 0, so θ=2\theta = 2.

  2. Therefore

    min⁡θL(θ)=1\min_\theta L(\theta)=1

    whereas

    arg min⁡θL(θ)={2}\operatorname*{arg\,min}_\theta L(\theta)=\{2\}

For L(t)=(t+4)2+7L(t)=(t+4)^2+7, find the minimum value and argmin.

Show solution

minimum 7; argmin {−4}\{-4\}.

Worked example

The optimizer versus the optimum

Example A:

L(t)=(t+1)2+4L(t)=(t+1)^2+4

The square is at least zero, so minL=4min L = 4 at t=−1t = -1;

argmin⁡L={−1}\operatorname{argmin} L = \{-1\}

Example B:

L(t)=(t2−1)2L(t)=(t^2-1)^2

The minimum is 0, achieved at t=−1t = -1 and t=1t = 1;

argmin⁡L={−1,1}\operatorname{argmin} L = \{-1,1\}

One objective can have more than one minimizing input.

Watch out

Common mistake

Writing

argmin⁡L=4\operatorname{argmin} L = 4

for Example A confuses an input with an objective value. Another mistake is to assume that writing an optimization problem means a training algorithm has already solved it. The example states the goal; finding a practical algorithm is a separate question.

YOUR STUDY PROGRESS

Finished this section? Save your progress.

Back to top ↑