Homework 8C

1 Homework 8 Problem 3 Suppose that we use bootstraping (i.e., sample with replacement) to sample 5 data points out of the training dataset { x 1 ,x 2 ,...,x 10 }. Answer the following questions 1. Is it possible for the sampled bootstrap to be { x 1 ,x 2 ,x 3 ,x 4 ,x 5}? If no, please explain why; otherwise calculate the probability of getting this specific bootstrap sample. Yes, x 1 ,x 2 ,x 3 ,x 4 ,x 5 can be obtained as a bootstrap sample because bootstrapping involves sampling with replacement, which means that each data point has a chance of being selected multiple times. This formula can be used to calculate the likelihood of receiving this specific bootstrap sample: • Probability of selecting x 1 in one draw = 1/10 (since there are 10 data points) • Probability of selecting x 2 ,x 3 ,x 4 and x 5 in subsequent draws = 1/10 each • Probability of getting {x 1 ,x 2 ,x 3 ,x 4 ,x 5 } = (1/10) * (1/10) * (1/10) * (1/10) * (1/10) = (1/10)^5 = 1/100,000 2. Is it possible for the sampled bootstrap to be {x 1, x 1, x 1, x 1, x 1 }? If no, please explain why; otherwise calculate the probability of getting this specific bootstrap sample. No, the sampled bootstrap cannot be x 1, x 1, x 1, x 1, x 1 . This is because bootstrap sampling is done with replacement, which means that each data point can be chosen multiple times. However, in this case, the sample contains only one unique data point (x1) repeated five times. Because the sampling is done with replacement, this specific bootstrap sample cannot be obtained. 3. Compute the probability of a data sample, say x 1 , NOT being included in the bootstrap. Because there are 9 other data points besides x1, the probability of x1 not being included in a single draw is 9/10. The probability of x1 not being included in any of the five draws in bootstrapping with replacement is: • Probability of x1 not being selected in one draw = 9/10

2 • Probability of x1 not being selected in all five draws = (9/10)^5 = 59049/100000 = 0.59049 = 59.05 %

4 • In the average approach, the final classification is based on the average of the individual probabilities. • 𝑨𝒗𝒆𝒓?𝒈𝒆 𝑷𝒓𝒐???𝒊𝒍𝒊𝒕𝒚 = (0.1+0.15+0.2+0.2+0.55+0.6+0.6+0.65+0.7+0.75) 10 ≈ 0.47 • Since 0.46 is less than 0.5, the final classification under the average probability approach is -1. As a result, the final classification under the majority vote approach is +1, whereas the final classification under the average probability approach is -1 for the specific test input x based on the given set of probability estimates.

5 Problem 5 Single-layer perceptrons are simplified neural networks. In this exercise, we use the sign function (the “hard” version of the Logistic function) as the activation function and assume that the input layer has two variables x 1 and x 2 only. , Formally, such perceptron can be expressed in the following way: G p ( x 1 ,x 2 ) := sign ( w 1 x 1 + w 2 x 2 + w 3 ) , where w 1 ,w 2 ,w 3 represent the parameters. 1. Given two binary variables x 1 and x 2 , the AND operator is defined as: {1 if x 1 = x 2 = 1, AND( x 1 ,x 2 ) := (0 otherwise. Is it possible to find parameters w 1 ,w 2 ,w 3 such that G p ( x 1 ,x 2 ) = AND( x 1 ,x 2 ) for all x 1 ,x 2 ∈ {0 , 1}? If no, please explain why; otherwise identify the parameters and show that your answer is correct. Yes, it is possible to find w 1 ,w 2 ,w 3 parameters such that G p ( x 1 ,x 2 ) = AND( x 1 ,x 2 ) for all x 1 ,x 2 ∈ {0 , 1}. Set the parameters to w 1 = 1, w 2 = 1, and w 3 = -1.5. The perceptron will output 1 when both x 1 and x 2 are 1, and 0 otherwise with these parameter values. This is consistent with the behavior of the AND operator. 2. Given two binary variables x 1 and x 2 , the OR operator is defined as: {1 if x 1 = 1 or x 2 = 1, OR( x 1 ,x 2 ) := 0 otherwise.