Practice Questions

CS 583 - Deep Learning Practice Questions Question 1 s start s ans s quit s end Answer Quit V Ans ( s ans ) =? V Quit ( s quit ) =? For the MDP above (same as the one we had in class), we randomly selected a policy and generated four (4) episode. What will be the values after each episode if we use the Model-free Monte Carlo method. You should write down the two (2) utility values for each question. ( V Ans ( s ans ) =? and V Quit ( s quit ) =?). i) Policy = Ans , Data = s start ; Ans, 4 , s start ; Ans, 4 , s end ii) Policy = Quit, Data = s start ; Quit, 10 , s end iii) Policy = Ans , Data = s start ; Ans, 4 , s end iv) Policy = Ans , Data = s start ; Ans, 4 , s start ; Ans, 4 , s start ; Ans, 4 , s end v) Policy = Quit, Data = s start ; Quit, 10 , s end 1

Question 2 s start s end Ans: 3 4 : $ 5 Ans: 1 4 : $ 5 Quit: 1: $ 20 The MDP for the gameshow (given in Question 1) can also be drawn as above without the chance nodes. Labels for each edge denote three things; the policy, the transition probability, and the reward. Suppose you start at the s start state and stop when you reach the s end state, and the discount factor γ = 0 . 5. Answer the following questions. i) V Ans ( s end ) =? ii) V Ans ( s start ) =? Note: You may assume that you take a total of n steps to reach s end . iii) V Quit ( s end ) =? iv) V Quit ( s start ) =? v) V opt ( s start ) =? vi) What is the optimal policy of the above MDP? vii) Define the value function V π ( s ) in an MDP? Question 3 i) What is the difference between the optimization functions for Supervised Multi-task Learning and Multi-task Reinforcement Learning? ii) Give an example of domain shift when using Transfer Learning for Reinforce- ment Learning. What is a possible solution? iii) There are two generators in cycle GANs. What is the loss function each of the generator is trying to minimize? iv) Suppose you have trained a model which identifies the genre of a given movie plot. Now you plan to expand your model and include movies which can have multiple genres; e.g. ‘Back to the Future’ can be classified both as a Western 2

v) Which of the following is the cost for the generator for GANs ( G is the gen- erator and D is the discriminator). (A) J ( G ) = 1 m ∑ m i =1 log(1 − D ( G ( z ( i ) ))) (B) J ( G ) = − 1 m ∑ m i =1 log( D ( G ( z ( i ) ))) (C) J ( G ) = 1 m ∑ m i =1 log(1 − G ( D ( z ( i ) ))) (D) J ( G ) = − 1 m ∑ m i =1 log( G ( D ( z ( i ) ))) vi) What does K-L divergence compute? Explain in regards to the second loss term in the Variational Auto-Encoder’s (VAE) loss function: D KL ( q ϕ ( z | x ) || p ( z ))? Question 5 i) What is the difference between binary cross entropy loss function and the categorical cross-entropy loss function? ii) Draw the computation graph for a one-hidden layer neural network with ReLU activation for the hidden layer and sigmoid activation for the output layer. Write all the derivatives. You are given the following: • the derivative of Relu is f ′ ( x ) = ( 0 if x = 0 1 otherwise , and • the derivative for sigmoid is σ ′ ( x ) = σ ( x )(1 − σ ( x )), iii) After training your neural network you observe a gap between the training accuracy (100%) and test accuracy (41%). Which of the following method is used to reduce this gap? (A) GANs (B) Batch size tuning (C) Adam optimizer (D) Dropout 4