PS#4

Q1 Statistics and Exponential Family Basics 31 Points Solve the following questions in order. You could find the relevant hint for solving the next questions by studying and answering to the previous questions. Q1.1 7 Points Given a set of samples , which of the following is a statistic (not necessarily sufficient)? Select ALL correct options. Q1.2 5 Points A probability density function in the exponential family is of the form: . Which of the following is incorrect? { y , ... y } (1) ( m ) (i.e., sample mean)  = y ˉ y / m ∑ i =1 m ( i ) (i.e., sample variance)  s = 2 ( y − ∑ i =1 m ( i ) ) y ˉ 2  T = 1 max { y , y , ..., y } (1) (2) ( m )  T = 2 median { y , y , ..., y } (1) (2) ( m )  T = 3 5 p ( y ; η ) = b ( y ) exp ( η T ( y ) − T a ( η ))

Q1.3 7 Points The univariate Gaussian density function is . Recall how we transformed this density into a form of exponential family in the lecture. Choose ALL correct descriptions. Q1.4 could be a vector for a certain distribution .  y p is always a scalar (i.e., is a scalar-valued function of )  b ( y ) b ( y ) y The dimension of is equal to the dimension of .  T ( y ) y makes as a valid probability distribution (i.e., sum to 1)  a ( η ) p is always equal to if provided with an input vector and the model parameter .  η θ x T x θ p ( y ; μ , σ ) = 2 exp σ 2 π 1 ( −2 σ 2 ( y − μ ) 2 ) are called the standard parameter (i.e., parameters in the standard density format)  ( μ , σ ) The same density could be represented by multiple different forms.  Natural parameter is the canonical parameter in the perspective of exponential family.  The dimension of natural parameter could be different from the dimension of standard parameter for some distributions in the exponential family. If each outcome is a result given an input , we can think model parameter associates the input to the output (e.g., ).  y x θ y = θ x T

Q2.1 5 Points What condition must be generally assumed to use GLM for building a prediction model? Q2.2 5 Points To follow the GLM recipe, we define the hypothesis as the conditional expectation . Q2.3 5 Points If your outcome is one of four classes , which of the following distributions you would better choose to incorporate in the GLM? Q3 Generalized Linear Model a distribution in the exponential family.  y ∼ a distribution in the expoenntial family.  y ∣ x ∼ None of the above.  h ( x ) θ E [ y ∣ x ; θ ] True  False  ∈ { A , B , C , D } Poisson distribution  Multinomial distribution  Gaussian distribution  Uniform distribution  Binomial distribution 

54 Points In this problem, your output will be a different type of random variable. It models the number of times that an event occurs in an interval of time or space. It's probability density is of the form: , where $\labmda$ indicates the rate of the event which occurs independently. (For example, if you receive in average 4 postal mails every day, then receiving any particular mail will not affect the arrival times of the future mails. ) Q3.1 20 Points Prove or disprove whether this distribution is in the exponential family. If so, you should derive and clearly write down natural parameter, sufficient statistic, log partition function, and the base measure. Q3.1.pdf  Download p ( y ; λ ) = y ! e λ − λ y λ = 4 

1 of 1 Q3.3 17 Points Suppose you are given with a training data . Evaluate the partial derivative of one single example with respect to . Derive a stochastic gradient ascent algorithm for learning the model parameter . (Hint: associate the user parameter to the natural parameter , which is always assumed to be linearly hypothesized by through the model parameters and the example ) D = {( x , y )∣1 ≤ ( i ) ( i ) i ≤ m } p ( y ∣ x ; θ ) ( i ) ( i ) θ j θ λ η θ x T θ x

Q3.3.pdf  Download 3 of 3  GRADED Problem Set (PS) #04 