HomeWork1_ML_Assignment

pdf

School

New York University *

*We aren’t endorsed by this school

Course

6913

Subject

Computer Science

Date

Jan 9, 2024

Type

pdf

Pages

Uploaded by HighnessCobraMaster699

Introduction to Machine Learning Homework 1 1 Problem (10 points) Solution: Plotting with different d values: (by Splitting data into 2 equal halves using random split) for d = 1 For d = 2 For d = 5

For d = 6 for d = 7 For d = 12

For d = 15 For d = 18 By running cross validation for the range of D (1 to 100):

Your preview ends here

Eager to read complete document? Join bartleby learn and gain access to the full version

Access to all documents
Unlimited textbook solutions
24/7 expert homework help

With d value as 7, testing error reaches its lowest. Code : import numpy as np import matplotlib.pyplot as plt from numpy.linalg import inv, pinv from scipy.io import loadmat def split_function(x_train, x_test, y_train, y_test): indices_total = np.random.permutation(x.shape[0]) size = int(x.shape[0]*50/100) indices_train, indices_test = indices_total[:size], indices_total[size:] x_train = x[indices_train] x_test = x[indices_test] y_train, y_test = y[indices_train], y[indices_test] return x_train, x_test, y_train, y_test def polyreg(x_train, x_test, y_train, y_test, D): D_Power = np.array(range(D+1)) X_TRAIN = np.power(x_train, D_Power) model = np.linalg.pinv(X_TRAIN).dot(y_train) train_loss = (0.5/X_TRAIN.shape[0])*np.sum(np.power(y_train-X_TRAIN.dot(model),2)) X_TEST = np.power(x_test, D_Power)

if D in range (1,20): plt.figure(D) plt.plot(x_train, y_train, 'r.', label='Train data') plt.plot(x_test, y_test, 'g.', label='Test data') plt.plot(x_train, X_TRAIN.dot(model), 'b.', label='Prediction') plt.title('d='+str(D)) plt.legend() plt.show() test_loss = (0.5/X_TEST.shape[0])*np.sum(np.power(y_test-X_TEST.dot(model),2)) return train_loss, test_loss Dataset_matlab = loadmat('problem1.mat') x = Dataset_matlab['x'] y = Dataset_matlab['y'] splitdata = split_function(x_train, x_test, y_train, y_test) max_dim = 100 loss = [polyreg(x_train, x_test, y_train, y_test, i) for i in range(100)] train_errs = [loss[0] for loss in loss] test_errs = [loss[1] for loss in loss] min_val = np.min(test_errs) min_ind = np.argmin(test_errs) plt.figure(0) plt.plot(range(100), train_errs,'b',label='Train') plt.plot(range(100), test_errs, 'r',label='Test') plt.xlabel('Dimension'); plt.ylabel('Error') plt.title('Cross-validation') plt.show() print('minimum loss value:',min_val,' ','minimum loss indice:', min_ind) 2 Problem (10 points) R reg ( 𝜃 ) = R emp ( 𝜃 ) + Penalty( 𝜃 )

= 1/N (Sum of 1 to N) L (y i , f(x i , 𝜃 ))+ λ/2N With gradient = 0, empirical risk 𝜃 * = (X T X + λI) -1 X T y Two-fold cross validation to find the best λ. As can be seen from the graph above, increasing λ causes the test error to decrease rapidly (while increasing). training error). The minimum λ value in this case was ≈ 999, but it also depends. Code: import numpy as np import matplotlib.pyplot as plt from numpy.linalg import inv, pinv, norm from scipy.io import loadmat def split_function(x_train, x_test, y_train, y_test): indices_total = np.random.permutation(x.shape[0]) size = int(x.shape[0]*50/100) indices_train, indices_test = indices_total[:size], indices_total[size:] x_train = x[indices_train] x_test = x[indices_test]

Your preview ends here

Eager to read complete document? Join bartleby learn and gain access to the full version

Access to all documents
Unlimited textbook solutions
24/7 expert homework help

y_train, y_test = y[indices_train], y[indices_test] return x_train, x_test, y_train, y_test def polyreg(x_train, x_test, y_train, y_test, L): theta = inv(x_train.T.dot(x_train) + L *np.identity(x_train.shape[- 1])).dot(x_train.T).dot(y_train) norm_theta = norm(theta) train_loss = (0.5/x_train.shape[0]) * (np.sum(np.power(y_train- x_train.dot(theta),2)) + L * norm_theta * norm_theta) test_loss = (0.5/x_test.shape[0]) * (np.sum(np.power(y_test- x_test.dot(theta),2)) + L * norm_theta * norm_theta) return train_loss, test_loss Dataset_matlab = loadmat('problem2.mat') x = Dataset_matlab['x'] y = Dataset_matlab['y'] splitdata = split_function(x_train, x_test, y_train, y_test) loss = [polyreg(x_train, x_test, y_train, y_test, i) for i in range(1000)] train_errs = [loss[0] for loss in loss] test_errs = [loss[1] for loss in loss] min_ind = np.argmin(test_errs) min_val = np.min(test_errs) plt.figure(0) plt.title('Loss') plt.plot(range(1000), train_errs,'b', label='Train data') plt.plot(range(1000), test_errs,'r', label='Test data') plt.xlabel('lambda'); plt.ylabel('error') plt.legend() plt.show() print('minimum loss value:',min_val,' ','minimum loss indice:', min_ind) 2. Problem 3 (10 points) Logistic Squashing Function. The logistic squashing function is given by g ( z ) = 1 / (1 + exp( − z )) . Show that it satisfies the property g ( − z ) = 1 − g ( z ). Also show that

its inverse is given by g − 1 ( y ) = ln( y/ (1 − y ). Answer: Given, g(z) = 1/(1 + exp(-z)) and we need to prove g(-z) = 1 – g(z) now, g(-z) = 1/ (1 + exp(z)) = 1 + exp(z)-exp(z) / (1 + exp(z)) = 1 + exp(z)/(1 + exp(z)) - exp(z)/(1 + exp(z)) = 1 - exp(z)/(1 + exp(z)) = 1 – 1/ ((1/exp(z))+ 1) = 1 – 1/(exp(-z) + 1) = 1 – g(z) Therefore, g(-z) = 1 – g(z) Now we need to prove the inverse of the logistic function i.e. g -1 (y) = ln (y/(1-y)) ln (y/(1-y)) = ln (1/(1 + exp(-z)) / 1 – 1/(1 + exp(-z))) = ln (1/(1 + exp(-z)) / 1+exp(-z)/1 + exp(-z) – 1/(1 + exp (-z)) = ln (1 / 1 + exp(-z) - 1) = ln (1/exp(-z)) = ln (exp(z)) = z g -1 (g(z)) = z. 3. Problem 4 (20 points) Solution:

Your preview ends here

Eager to read complete document? Join bartleby learn and gain access to the full version

Access to all documents
Unlimited textbook solutions
24/7 expert homework help

Learning rate (η) = 0 . 1 , Tolerance (s) = 0 . 005, Iterations count: 250 until convergence, Accuracy is 80%. Graph showing the decision boundary is: Poor performance, learning rate is small to tolerance so that stops iteration.

Learning rate (η) = 1 , Tolerance (s) = 0 . 004, Iterations count: 2980 until convergence and accuracy of 90%. Graph showing decision boundary is as below: Learning rate ( η) = 1 , Tolerance (s) = 0 . 005, Iterations count: 2089, until convergence, accuracy is 98%. Graph showing decision boundary is as below:

accuracy is good, but the decision boundary may not seem perfect. η = 1 , s = 0 . 003, Iterations count: 4623, until convergence, accuracy is 100%. Graph is showing the decision boundary is as below: excellent performance, iteration stops just after binary error gets to the value zero. Code : import numpy as np import matplotlib.pyplot as plt from scipy.io import loadmat from numpy import exp, log, absolute from numpy.linalg import norm def sigmoid(X): return 1.0/(1+exp(-X)) def learningrate(x, y, opts): num_samples, num_features = x.shape lrate = opts['lrate'] tolerance = opts['tolerance'] Iterations = 0 weights = np.random.random_sample((num_features, 1)) Errors = [] Risk = [] while True: op = sigmoid(x.dot(weights)) err = y - op Errors.append(error_calculate(y, op, num_samples)) Risk.append(risk_calculate(y, op, num_samples)) dec = lrate * 1/num_samples * x.T.dot(err)

Your preview ends here

Eager to read complete document? Join bartleby learn and gain access to the full version

Access to all documents
Unlimited textbook solutions
24/7 expert homework help

weights = weights + dec Iterations += 1 if norm(dec)<tolerance: break return weights, Iterations, Errors, Risk def risk_calculate(y, y_pred, N): assert(y.shape[-1]==1) risk = 0 for i in range(y.shape[0]): if y[i,0] == 1: risk -= log(y_pred[i,0]) else: risk -= log(1-y_pred[i,0]) return risk/N def error_calculate(y, y_pred, N): return float(np.count_nonzero((y_pred>0.5)!=y))/N dataset_matlab = loadmat('dataset4.mat') x = dataset_matlab['X'] y = dataset_matlab['Y'] opts = {'lrate':1, 'tolerance':0.003} #(0.1, 0.005), (1, 0.005), (1, 0.001),(1, 0.003) weights, Iterations, Errors, Risk= learningrate(x, y, opts) print(Iterations) plt.figure(0) samples_num = x.shape[0] for i in range(samples_num): plt.plot(x[i, 0], x[i, 1], 'r.') minimum_x = min(x[:, 0]) maximum_x = max(x[:, 0]) y_minimum_x = float(-weights[2] - weights[0] * minimum_x) / weights[1] y_maximum_x = float(-weights[2] - weights[0] * maximum_x) / weights[1] plt.plot([minimum_x, maximum_x], [y_minimum_x, y_maximum_x], '-b') plt.xlabel('X0'); plt.ylabel('X1') plt.show() plt.figure(1) plt.plot(range(Iterations), Errors,'r',label='Error') plt.plot(range(Iterations), Risk, 'g',label='Risk') plt.xlabel('Iterations') plt.title('Error/Risk-Iteration') plt.legend() plt.show()

Related Documents

APIPA.pdf

IT-FP3355_HinesJon_AssesmentNumber1-Attempt2.docx

ACS-1903-050-F2023 Assignment 3.pdf

Case Study week main 3.docx

Screenshot 5.8.2 2.png

assignment3_MECHTRON_3X03.pdf

McCurley-Shimer_Assignment #4.docx

MSIT 3150 - Quiz 5.docx

MSIT 3150-Assignment 3 .docx

Textbook Assignment 3.docx

McCurley-Shimer_Assignment #2.docx

Recommended textbooks for you

C++ Programming: From Problem Analysis to Program...

Computer Science

ISBN:9781337102087

Author:D. S. Malik

Publisher:Cengage Learning

C++ for Engineers and Scientists

Computer Science

ISBN:9781133187844

Author:Bronson, Gary J.

Publisher:Course Technology Ptr

EBK JAVA PROGRAMMING

Computer Science

ISBN:9781337671385

Author:FARRELL

Publisher:CENGAGE LEARNING - CONSIGNMENT

Programming Logic & Design Comprehensive

Computer Science

ISBN:9781337669405

Author:FARRELL

Publisher:Cengage

COMPREHENSIVE MICROSOFT OFFICE 365 EXCE

Computer Science

ISBN:9780357392676

Author:FREUND, Steven

Publisher:CENGAGE L

Microsoft Visual C#

Computer Science

ISBN:9781337102100

Author:Joyce, Farrell.

Publisher:Cengage Learning,

SEE MORE TEXTBOOKS

Recommended textbooks for you

C++ Programming: From Problem Analysis to Program...
Computer Science
ISBN:9781337102087
Author:D. S. Malik
Publisher:Cengage Learning
C++ for Engineers and Scientists
Computer Science
ISBN:9781133187844
Author:Bronson, Gary J.
Publisher:Course Technology Ptr
EBK JAVA PROGRAMMING
Computer Science
ISBN:9781337671385
Author:FARRELL
Publisher:CENGAGE LEARNING - CONSIGNMENT
Programming Logic & Design Comprehensive
Computer Science
ISBN:9781337669405
Author:FARRELL
Publisher:Cengage
COMPREHENSIVE MICROSOFT OFFICE 365 EXCE
Computer Science
ISBN:9780357392676
Author:FREUND, Steven
Publisher:CENGAGE L
Microsoft Visual C#
Computer Science
ISBN:9781337102100
Author:Joyce, Farrell.
Publisher:Cengage Learning,