Chapter 8, Problem 8.15TE

If f and g are density functions that are positive over the same region, then the kullback-leiber divergence from density f to density g is defined by $KL\left(f,g\right)=E_f\left[\log \left(\frac{f\left(X\right)}{g\left(X\right)}\right)\right]={\displaystyle \int \log \left(\frac{f\left(x\right)}{g\left(x\right)}\right)}f\left(x\right)dx$ where the notation $E_f\left[h\left(X\right)\right]$ is used to indicate that X has density function f.

a. Show that $KL\left(f,f\right)=0$

b. Use Jensen’s inequality and the Identity $\log \left(\frac{f\left(x\right)}{g\left(x\right)}\right)=-\log \left(\frac{g\left(x\right)}{f\left(x\right)}\right)$, to show that $KL\left(f,g\right)\ge 0$