Let f be a differentiable and μ-strongly-convex function whose minimum is achieved at x*. Let us assume that the variance on the gradients is controlled: There exists σ > 0 and L≥ 0 such that E; [||Vf; (x)||² | xk] ≤0² + L ||xk − x* ||². Prove the following statements: 1. If σ > 0 and L = 0, SGD with step size ŋk satisfies | E || xo - Ef (zk) - f*] ≤ =0 (1) j=0 where Σj=0jxj Zk= (2) ΣΕ In particular, E[f (zk) - f*] converges to 0 if and only if Σ, n; = ∞ and 2. If σ > 0 and L > 0, SGD with a constant step size n satisfies = 0. E||xk+1 - x* ||² ≤ (1 - 2nμ+n²L)*E ||xo-x* ||² + (1 − 2nµ + n²L); - ησε 2μ-nL (3) What is the restriction on the stepsize? 3. Let us observe by definition, SGD with step size n satisfies: ||K+1 – x = ||xk - xu t ng Vi(x)|| − 20k (k – x*, Vf(x)). (4) Derive the optimal step size and comment on it.

Let f be a differentiable and μ-strongly-convex function whose minimum is achieved at x. Let us assume that the variance on the gradients is controlled: There exists σ > 0 and L≥ 0 such that E; [||Vf; (x)||² | xk] ≤0² + L ||xk − x ||². Prove the following statements: 1. If σ > 0 and L = 0, SGD with step size ŋk satisfies | E || xo - Ef (zk) - f] ≤ =0 (1) j=0 where Σj=0jxj Zk= (2) ΣΕ In particular, E[f (zk) - f] converges to 0 if and only if Σ, n; = ∞ and 2. If σ > 0 and L > 0, SGD with a constant step size n satisfies = 0. E||xk+1 - x* ||² ≤ (1 - 2nμ+n²L)E ||xo-x ||² + (1 − 2nµ + n²L); - ησε 2μ-nL (3) What is the restriction on the stepsize? 3. Let us observe by definition, SGD with step size n satisfies: ||K+1 – x = ||xk - xu t ng Vi(x)|| − 20k (k – x*, Vf(x)). (4) Derive the optimal step size and comment on it.

Advanced Engineering Mathematics

10th Edition

ISBN:9780470458365

Author:Erwin Kreyszig

Publisher:Erwin Kreyszig

Chapter2: Second-order Linear Odes

Section: Chapter Questions

Problem 1RQ

See similar textbooks

Similar questions

Given thatf(x)={xcosx1/x x≠0, show thatfis not of bounded variation on[0,π,/2]
Solve for the general solution of the equation: 2x’’+ 200x = 0. Show all math steps.
* 8. Consider linear spline function L(x): Let f(x) = ln(x + 2), Given (0, 2), (-1, 4) and (1, 6). then the approximate errors f(x) - L(x) | is (a) f(x) - L(x)| ≤ 1/ (b) f(x) - L(x)| ≤ 1 (c) f(x) - L(x)| ≤ (d) None of the above. a b d
Suppose that f(x)=βx−3 if x≤2βx+2 if x>2 Find the value of β so that f(x) is continuous everywhere along the real line.
14. Suppose that 0