Let f be a differentiable and μ-strongly-convex function whose minimum is achieved at x*. Let us assume that the variance on the gradients is controlled: There exists σ > 0 and L≥ 0 such that E, [||Vfi(x)|||xk| ≤ 0² + L ||xk - x* ||². Prove the following statements: 1. If σ > 0 and L = 0, SGD with step size where nk satisfies · E [||x − x. [2] + Σ;=0 130² E[f (zk) - f*]≤ k 2-oj (1) Σj=0jxj Zk= (2) ak In particular, E[f (zk) - f*] converges to 0 if and only if Σ; ni 2. If σ > 0 and L > 0, SGD with a constant step size n satisfies = ∞ and = 0. E ||xk+1 - x* ||² ≤ (1-2μ + m² L)*E ||x − x*||² + (1 − 2nµ + n²L); - ησε 2μ-nL (3) What is the restriction on the stepsize? 3. Let us observe by definition, SGD with step size n satisfies: |xk|1x| = || − x 2 + nổ |Vf(x)|| – 20k (k – x, Vfi(x)). -x -x+ Derive the optimal step size and comment on it. - - (4)
Let f be a differentiable and μ-strongly-convex function whose minimum is achieved at x*. Let us assume that the variance on the gradients is controlled: There exists σ > 0 and L≥ 0 such that E, [||Vfi(x)|||xk| ≤ 0² + L ||xk - x* ||². Prove the following statements: 1. If σ > 0 and L = 0, SGD with step size where nk satisfies · E [||x − x. [2] + Σ;=0 130² E[f (zk) - f*]≤ k 2-oj (1) Σj=0jxj Zk= (2) ak In particular, E[f (zk) - f*] converges to 0 if and only if Σ; ni 2. If σ > 0 and L > 0, SGD with a constant step size n satisfies = ∞ and = 0. E ||xk+1 - x* ||² ≤ (1-2μ + m² L)*E ||x − x*||² + (1 − 2nµ + n²L); - ησε 2μ-nL (3) What is the restriction on the stepsize? 3. Let us observe by definition, SGD with step size n satisfies: |xk|1x| = || − x 2 + nổ |Vf(x)|| – 20k (k – x, Vfi(x)). -x -x+ Derive the optimal step size and comment on it. - - (4)
Advanced Engineering Mathematics
10th Edition
ISBN:9780470458365
Author:Erwin Kreyszig
Publisher:Erwin Kreyszig
Chapter2: Second-order Linear Odes
Section: Chapter Questions
Problem 1RQ
Related questions
Question
![Let f be a differentiable and μ-strongly-convex function whose minimum is achieved at x*. Let us assume that the
variance on the gradients is controlled: There exists σ > 0 and L≥ 0 such that E, [||Vfi(x)|||xk| ≤ 0² + L ||xk - x* ||².
Prove the following statements:
1. If σ > 0 and L = 0, SGD with step size
where
nk satisfies
· E [||x − x. [2] + Σ;=0 130²
E[f (zk) - f*]≤
k
2-oj
(1)
Σj=0jxj
Zk=
(2)
ak
In particular, E[f (zk) - f*] converges to 0 if and only if Σ; ni
2. If σ > 0 and L > 0, SGD with a constant step size n satisfies
= ∞ and
= 0.
E ||xk+1 - x* ||² ≤ (1-2μ + m² L)*E ||x − x*||² + (1 − 2nµ + n²L);
-
ησε
2μ-nL
(3)
What is the restriction on the stepsize?
3. Let us observe by definition, SGD with step size n satisfies:
|xk|1x| = || − x 2 + nổ |Vf(x)|| – 20k (k – x, Vfi(x)).
-x
-x+
Derive the optimal step size and comment on it.
-
-
(4)](/v2/_next/image?url=https%3A%2F%2Fcontent.bartleby.com%2Fqna-images%2Fquestion%2F08792f8d-91a6-4520-a2bb-5d9564b28bda%2Fa6bb0320-8107-4734-ae1f-5156c9868d41%2Fgw2q1wf_processed.png&w=3840&q=75)
Transcribed Image Text:Let f be a differentiable and μ-strongly-convex function whose minimum is achieved at x*. Let us assume that the
variance on the gradients is controlled: There exists σ > 0 and L≥ 0 such that E, [||Vfi(x)|||xk| ≤ 0² + L ||xk - x* ||².
Prove the following statements:
1. If σ > 0 and L = 0, SGD with step size
where
nk satisfies
· E [||x − x. [2] + Σ;=0 130²
E[f (zk) - f*]≤
k
2-oj
(1)
Σj=0jxj
Zk=
(2)
ak
In particular, E[f (zk) - f*] converges to 0 if and only if Σ; ni
2. If σ > 0 and L > 0, SGD with a constant step size n satisfies
= ∞ and
= 0.
E ||xk+1 - x* ||² ≤ (1-2μ + m² L)*E ||x − x*||² + (1 − 2nµ + n²L);
-
ησε
2μ-nL
(3)
What is the restriction on the stepsize?
3. Let us observe by definition, SGD with step size n satisfies:
|xk|1x| = || − x 2 + nổ |Vf(x)|| – 20k (k – x, Vfi(x)).
-x
-x+
Derive the optimal step size and comment on it.
-
-
(4)
Expert Solution
This question has been solved!
Explore an expertly crafted, step-by-step solution for a thorough understanding of key concepts.
Step by step
Solved in 2 steps
Similar questions
- Recommended textbooks for youAdvanced Engineering MathematicsAdvanced MathISBN:9780470458365Author:Erwin KreyszigPublisher:Wiley, John & Sons, Incorporated
Numerical Methods for EngineersAdvanced MathISBN:9780073397924Author:Steven C. Chapra Dr., Raymond P. CanalePublisher:McGraw-Hill Education
Introductory Mathematics for Engineering Applicat…Advanced MathISBN:9781118141809Author:Nathan KlingbeilPublisher:WILEYAdvanced Engineering MathematicsAdvanced MathISBN:9780470458365Author:Erwin KreyszigPublisher:Wiley, John & Sons, IncorporatedNumerical Methods for EngineersAdvanced MathISBN:9780073397924Author:Steven C. Chapra Dr., Raymond P. CanalePublisher:McGraw-Hill EducationIntroductory Mathematics for Engineering Applicat…Advanced MathISBN:9781118141809Author:Nathan KlingbeilPublisher:WILEY
Mathematics For Machine TechnologyAdvanced MathISBN:9781337798310Author:Peterson, John.Publisher:Cengage Learning,
