vertopal.com_mnist_logreg

[0, 0, 0, ..., 0, 0, 0], [0, 0, 0, ..., 0, 0, 0], [0, 0, 0, ..., 0, 0, 0]], dtype=uint8), 'testY': array([[7, 2, 1, ..., 4, 5, 6]], dtype=uint8), 'trainY': array([[5, 0, 4, ..., 5, 6, 8]], dtype=uint8), 'trainX': array([[0, 0, 0, ..., 0, 0, 0], [0, 0, 0, ..., 0, 0, 0], [0, 0, 0, ..., 0, 0, 0], ..., [0, 0, 0, ..., 0, 0, 0], [0, 0, 0, ..., 0, 0, 0], [0, 0, 0, ..., 0, 0, 0]], dtype=uint8)} (b) Coding gradient descent. Do not use scikit-learn or other built in tools for this exercise. Please only use the packages that are already imported in the notebook. 1 Open mnist logreg.ipynb. We will use logistic regression to direntiate 4's from 9's, a notoriously tricky problem. Run the rst box to see what the data looks like. In the second box I have set up the problem for you by pulling out the train and test set, selecting out only the data samples related to 4's and 9's. I have not altered the data in any other way. While other normalization tricks will help you improve the accuracy, for the purposes of this exercise we will forgo them, so that it's easy to compare everyone's solutions. Fill in the next box by providing code that will return the loss function value and the gradient. Make sure that everything is normalized, e.g. don't forget the 1=m term in the front of our loss function. Run the script. If done correctly, you should see 45.192, 12343.177 Write a script that returns the classication accuracy given . Use gradient descent to minimize the logistic loss for this classication problem. Use a step size of 10􀀀6. (1.5 pts) Run for 1500 iterations. In your report, give the plot of the train loss and train/test misclassi- cation rate, plotted as a function of iterations. Report the nal train and test accuracy values. def sigmoid(val): return 1 / ( 1 + np.exp( - val)) def getLossFunction(theta): val = ytrain * np.dot(Xtrain, theta) return - np.mean(np.log(sigmoid(val))) def getGradient(theta): val = ytrain * np.dot(Xtrain, theta) return - np.mean(( 1 - sigmoid(val))[:, np.newaxis] * ytrain[:, np.newaxis] * Xtrain, axis = 0 ) def predict(theta,x): return np.where(np.dot(x, theta) >= 0 , 1 , - 1 ) def classification_accuracy(theta, x, y): y_pred = np.array(predict(theta, x)).reshape( - 1 ) y_true = np.array(y).reshape( - 1 ) tp = sum ( 1 for pred, true in zip (y_pred, y_true) if pred == true) return tp / len (y)

#Computing Train and Test Accuracy train_acc = classification_accuracy(theta_values[ - 1 ], Xtrain, ytrain) test_acc = classification_accuracy(theta_values[ - 1 ], Xtest, ytest) print ( f"Final Training Accuracy: { train_acc * 100 :.2f} %" ) print ( f"Final Test Accuracy: { test_acc * 100 :.2f} %" ) Final Training Accuracy: 96.45% Final Test Accuracy: 96.63% 1(c) Coding stochastic gradient descent. Do not use scikit-learn or other built in tools for this exercise. Please only use the packages that are already imported in the notebook. Now, ll in the next box a function that takes in and a minibatch B as either a list of numbers or as an np.array, and returns the minibatch gradient rBf() = 1 jBj X i2B rfi() where fi() is the contribution to the gradient from datapoint i: fi = 􀀀log((yixTi )): Run the script. If done correctly, you should see the number 5803.5 printed out. def getStochGradient(theta, minibatch): grad = np.zeros_like(theta) for i in minibatch: y_i = ytrain[i] x_i = Xtrain[i]

<matplotlib.legend.Legend at 0x1db816a7550> #Plotting Train Accuracy #gd_train_acc = [classification_accuracy(theta, Xtrain, ytrain) for theta in gd_theta_values][:len(gd_loss_values)] sgd_train_acc = [classification_accuracy(theta, Xtrain, ytrain) for theta in sgd_theta_values][: len (sgd_loss_values)] plt.plot(gd_iterations, gd_train_acc, label = "Gradient Descent" ) plt.plot(sgd_iterations, sgd_train_acc, label = "Stochastic Gradient Descent" ) plt.title( "Train Accuracy vs. Iterations" ) plt.legend() <matplotlib.legend.Legend at 0x1db9eab0d50> #Plotting Test Accuracy gd_test_acc = [classification_accuracy(theta, Xtest, ytest) for theta in gd_theta_values][: len (gd_loss_values)] sgd_test_acc = [classification_accuracy(theta, Xtest, ytest) for theta in sgd_theta_values][: len (sgd_loss_values)] plt.plot(gd_iterations, gd_test_acc, label = "Gradient Descent" ) plt.plot(sgd_iterations, sgd_train_acc, label = "Stochastic Gradient Descent" ) plt.title( "Test Accuracy vs. Iterations" ) plt.legend() <matplotlib.legend.Legend at 0x1db94a07d50>

val = softmax(np.dot(X, theta)) s = X.shape[ 0 ] loss = - np. sum (np.log(val[ range (s), y.astype( int )])) / s return loss def gradient(theta, X, y): val = softmax(np.dot(X, theta)) grad = np.dot(X.T, val - (y[:, np.newaxis] == np.arange(theta.shape[ 1 ]))) return grad / X.shape[ 0 ] def compute_accuracy(theta, X, y): y_pred = predict(theta, X) tp = sum ( 1 for pred, true in zip (y_pred, y) if pred == true) return tp / len (y) def train(X, y, lr, epochs): theta = np.zeros((X.shape[ 1 ], len (np.unique(y)))) for epoch in range (epochs): grad = gradient(theta, X, y) theta = theta - (lr * grad) train_acc.append(compute_accuracy(theta, X_train, y_train)) test_acc.append(compute_accuracy(theta, X_test, y_test)) train_loss.append(compute_loss(theta, X_train, y_train)) test_loss.append(compute_loss(theta, X_test, y_test)) if (epoch + 1 ) % 50 == 0 : print ( f'Epoch { epoch + 1 } / { epochs } , Loss: { compute_loss(theta, X, y) } ' ) return theta def predict(theta, X): return np.argmax(softmax(np.dot(X,theta)), axis = 1 ) final_theta = train(X_train, y_train, lr = pow ( 10 , - 5 ), epochs = 500 ) Epoch 50/500, Loss: 0.43303781750474707 Epoch 100/500, Loss: 0.377047373840449 Epoch 150/500, Loss: 0.35266453445686136 Epoch 200/500, Loss: 0.3381561296393 Epoch 250/500, Loss: 0.32823523531012044 Epoch 300/500, Loss: 0.3208876171754041 Epoch 350/500, Loss: 0.31515416111898353 Epoch 400/500, Loss: 0.31051141521398773 Epoch 450/500, Loss: 0.30664603309909005 Epoch 500/500, Loss: 0.30335748284346775