IE6400_Day23

IE6400 Foundations of Data Analytics Engineering ¶ Fall 2023 ¶ Module 4: Introduction to Machine Learning ¶ Machine Learning Overview ¶ Machine learning (ML) is a branch of artificial intelligence (AI) that focuses on building systems that can learn from data. Rather than being explicitly programmed to perform a task, a machine learning algorithm uses statistical techniques to learn patterns in data and make predictions or decisions based on it. Key Aspects of Machine Learning ¶ 1. Supervised Learning ¶ • Description : An algorithm is trained on a labeled dataset, where the data comes with the correct answers. The algorithm makes predictions and is corrected if those predictions are wrong, leading it to learn over time. • Common Tasks : Classification (categorizing items) and regression (predicting numerical values). 2. Unsupervised Learning ¶ • Description : The algorithm is given data without any explicit instructions on what to do with it. It tries to learn patterns and the structure from the data. • Common Tasks : Clustering (grouping similar items) and association (finding rules that describe data). 3. Reinforcement Learning ¶ • Description : An agent learns how to behave in an environment by performing actions and receiving rewards or penalties. • Analogy : Teaching a dog new tricks. The dog is the agent, the environment is where the dog can perform tricks, and rewards (or penalties) are treats (or lack of treats). 4. Semi-Supervised and Active Learning ¶ • Description : These methods use both labeled and unlabeled data for training. Typically, a small amount of labeled data and a large amount of unlabeled data are used. 5. Deep Learning ¶ • Description : A subset of ML, deep learning models data with deep neural networks, which are algorithms inspired by the structure of the brain. Particularly powerful for tasks like image and speech recognition. Applications of Machine Learning ¶ Machine learning has a myriad of applications, including: • Web search engines • Recommendation systems (e.g., Netflix, Amazon) • Image and speech recognition • Medical diagnosis • Financial forecasting The core idea behind machine learning is that machines take data and "learn" from it, thereby improving their performance over time without being explicitly programmed for the task at hand. Popular Machine Learning Algorithms ¶ Machine learning encompasses a wide range of algorithms used for various tasks. Here's an overview of some of the most popular ones:

1. Linear Regression ¶ • Type : Supervised • Use Case : Predicting a continuous target variable based on one or more input features. • Description : Assumes a linear relationship between the inputs and the target. It tries to find the best-fit straight line that accurately predict the output values within a range. 2. Logistic Regression ¶ • Type : Supervised • Use Case : Binary classification problems. • Description : Estimates the probability that a given instance belongs to a particular category. Despite its name, it's used for classification, not regression. 3. Decision Trees ¶ • Type : Supervised • Use Case : Classification and regression tasks. • Description : Splits the data into subsets based on the value of input features. This process is repeated recursively, resulting in a tree-like model of decisions. 4. Random Forest ¶ • Type : Supervised • Use Case : Classification and regression. • Description : An ensemble method that creates a 'forest' of decision trees. Each tree is trained on a random subset of the data and makes its own predictions. The random forest algorithm then aggregates these predictions to produce a final result. 5. Support Vector Machines (SVM) ¶ • Type : Supervised • Use Case : Classification and regression. • Description : Tries to find a hyperplane that best separates the classes of data. It's particularly useful for classifying complex but small- or medium-sized datasets. 6. K-Means Clustering ¶ • Type : Unsupervised • Use Case : Clustering similar data points together. • Description : Partitions the data into 'K' number of clusters where each data point belongs to the cluster with the nearest mean. 7. Neural Networks (Deep Learning) ¶ • Type : Supervised, Unsupervised • Use Case : Complex tasks like image and speech recognition. • Description : Composed of layers of nodes or 'neurons'. Can automatically learn and extract features from raw data. 8. Naive Bayes ¶ • Type : Supervised • Use Case : Classification tasks, often used for text data. • Description : Based on Bayes' theorem with the 'naive' assumption of conditional independence between every pair of features. 9. Principal Component Analysis (PCA) ¶ • Type : Unsupervised • Use Case : Dimensionality reduction. • Description : Transforms the original variables into a new set of variables (the principal components) which are orthogonal (and linearly independent) and which reflect the maximum variance in the data.

predictors have a linear relationship with the dependent variable, the predictors are not highly correlated with each other, and when the data is homoscedastic, meaning the residuals are equal across the regression line. Exercise 1 Understanding Linear Regression in Machine Learning ¶ Problem Statement ¶ In this exercise, we will build a Linear Regression model to predict the median house values in California districts. The California Housing dataset includes metrics like median income and housing median age, which we will use as predictors. Our goal is to understand the relationship between the features and the median house value and to evaluate the performance of our regression model. Step-by-Step Guide ¶ Step 1: Import Libraries ¶ In [1]: # Step 1: Import necessary libraries import numpy as np import pandas as pd import matplotlib.pyplot as plt from sklearn.datasets import fetch_california_housing from sklearn.model_selection import train_test_split from sklearn.linear_model import LinearRegression from sklearn.metrics import mean_squared_error, r2_score import warnings # Settings the warnings to be ignored warnings.filterwarnings('ignore') Step 2: Load the Dataset ¶ In [2]: california = fetch_california_housing() california_df = pd.DataFrame(california.data, columns=california.feature_names) california_df['MedHouseVal'] = california.target Step 3: Exploratory Data Analysis ¶ In [3]: print(california_df.head()) # Display the first five rows of the dataset print(california_df.describe()) # Get the summary statistics MedInc HouseAge AveRooms AveBedrms Population AveOccup Latitude \ 0 8.3252 41.0 6.984127 1.023810 322.0 2.555556 37.88 1 8.3014 21.0 6.238137 0.971880 2401.0 2.109842 37.86 2 7.2574 52.0 8.288136 1.073446 496.0 2.802260 37.85 3 5.6431 52.0 5.817352 1.073059 558.0 2.547945 37.85 4 3.8462 52.0 6.281853 1.081081 565.0 2.181467 37.85 Longitude MedHouseVal 0 -122.23 4.526 1 -122.22 3.585 2 -122.24 3.521 3 -122.25 3.413 4 -122.25 3.422 MedInc HouseAge AveRooms AveBedrms Population \ count 20640.000000 20640.000000 20640.000000 20640.000000 20640.000000 mean 3.870671 28.639486 5.429000 1.096675 1425.476744 std 1.899822 12.585558 2.474173 0.473911 1132.462122 min 0.499900 1.000000 0.846154 0.333333 3.000000 25% 2.563400 18.000000 4.440716 1.006079 787.000000

50% 3.534800 29.000000 5.229129 1.048780 1166.000000 75% 4.743250 37.000000 6.052381 1.099526 1725.000000 max 15.000100 52.000000 141.909091 34.066667 35682.000000 AveOccup Latitude Longitude MedHouseVal count 20640.000000 20640.000000 20640.000000 20640.000000 mean 3.070655 35.631861 -119.569704 2.068558 std 10.386050 2.135952 2.003532 1.153956 min 0.692308 32.540000 -124.350000 0.149990 25% 2.429741 33.930000 -121.800000 1.196000 50% 2.818116 34.260000 -118.490000 1.797000 75% 3.282261 37.710000 -118.010000 2.647250 max 1243.333333 41.950000 -114.310000 5.000010 Step 4: Split the Data into Training and Testing Sets ¶ In [4]: X = california_df.drop('MedHouseVal', axis=1) y = california_df['MedHouseVal'] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) Step 5: Build the Linear Regression Model ¶ In [5]: lin_reg = LinearRegression() lin_reg.fit(X_train, y_train) Out[5]: LinearRegression() Step 6: Make Predictions ¶ In [6]: # Make predictions on the testing set y_pred = lin_reg.predict(X_test) Step 7: Evaluate the Model ¶ In [7]: # Calculate the Mean Squared Error (MSE) and Coefficient of Determination (R^2) mse = mean_squared_error(y_test, y_pred) r2 = r2_score(y_test, y_pred) print(f'Mean Squared Error: {mse:.2f}') print(f'R^2 Score: {r2:.2f}') Mean Squared Error: 0.56 R^2 Score: 0.58 Step 8: Visualize the Results ¶ In [8]: # Plotting the true values vs predicted values plt.scatter(y_test, y_pred) plt.plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()], '--k') plt.xlabel('Actual Values') plt.ylabel('Predicted Values') plt.title('Actual vs Predicted Housing Values') plt.show()

• ( $\beta_0$ ) is the intercept, • ( $\beta_1$, $\beta_2$, $\ldots$, $\beta_n$ ) are the coefficients, • ( $x_1$, $x_2$, $\ldots$, $x_n$ ) are the independent variables. Estimating Probabilities ¶ The logistic regression function ( $p$ ) is the sigmoid function of ( $\beta_0 + \ beta_1x_1 + \beta_2x_2 + \ldots + \beta_nx_n$ ), which yields values between 0 and 1. These values represent the probability that the dependent variable equals a 1. Model Training ¶ The parameters of the logistic regression model are estimated from the training data using the method of maximum likelihood estimation (MLE). Evaluation Metrics ¶ The performance of a logistic regression model is typically evaluated using metrics like accuracy, precision, recall, F1 score, and the ROC-AUC curve. Application in Python ¶ In Python's sklearn library, the LogisticRegression class is used to perform logistic regression. The model is trained using the .fit() method, and predictions can be made with the .predict() method for classification, or .predict_proba() for obtaining the probability estimates. Conclusion ¶ Logistic Regression is a powerful statistical way of modeling a binomial outcome with one or more explanatory variables. It measures the relationship between the categorical dependent variable and one or more independent variables by estimating probabilities using a logistic function. Exercise 2 Logistic Regression Model on the Iris Dataset ¶ Problem Statement ¶ The goal of this exercise is to build a logistic regression model to predict the species of an iris flower based on the sepal length, sepal width, petal length, and petal width. Dataset ¶ We will use the Iris dataset, which is a classic in the field of machine learning. The dataset contains 150 observations of iris flowers, each with four features and a label indicating the species of the iris. Objectives ¶ • Perform exploratory data analysis • Visualize the data • Prepare the data for modeling • Train a logistic regression model • Evaluate the model's performance • Interpret the results In [9]: # Import necessary libraries import numpy as np import pandas as pd import matplotlib.pyplot as plt from sklearn.datasets import load_iris from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split from sklearn.metrics import confusion_matrix, classification_report import seaborn as sns Load and Explore the Iris Dataset ¶ In [10]: # Load the dataset iris = load_iris()

X = iris.data y = iris.target # Convert to DataFrame for easier analysis iris_df = pd.DataFrame(X, columns=iris.feature_names) iris_df['species'] = pd.Categorical.from_codes(iris.target, iris.target_names) # Display the first 5 rows of the DataFrame iris_df.head() Out[10]: sepal length (cm) sepal width (cm) petal length (cm) petal width (cm) species 0 5.1 3.5 1.4 0.2 setosa 1 4.9 3.0 1.4 0.2 setosa 2 4.7 3.2 1.3 0.2 setosa 3 4.6 3.1 1.5 0.2 setosa 4 5.0 3.6 1.4 0.2 setosa Data Visualization ¶ Let's visualize the relationships between the different features and the species. In [11]: # Pairplot to visualize the relationships between features sns.pairplot(iris_df, hue='species') plt.show()

Preparing Data for Logistic Regression ¶ We will split the data into a training set and a testing set. In [12]: # Split the data into training and test sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42) Training the Logistic Regression Model ¶ In [13]: # Initialize the Logistic Regression model logreg = LogisticRegression(max_iter=200) # Fit the model with the training data logreg.fit(X_train, y_train) Out[13]: LogisticRegression(max_iter=200) Model Evaluation ¶ We will now evaluate the model's performance on the test set using a confusion matrix and classification report. In [14]: # Predict the labels for the test set y_pred = logreg.predict(X_test) # Classification report print(classification_report(y_test, y_pred)) precision recall f1-score support 0 1.00 1.00 1.00 19 1 1.00 1.00 1.00 13 2 1.00 1.00 1.00 13 accuracy 1.00 45 macro avg 1.00 1.00 1.00 45 weighted avg 1.00 1.00 1.00 45 In [15]: from sklearn.metrics import confusion_matrix import matplotlib.pyplot as plt # Assuming y_test are the true labels and y_pred are the predicted labels from your classifier # Compute confusion matrix conf_matrix = confusion_matrix(y_test, y_pred) # Set up the matplotlib figure fig, ax = plt.subplots(figsize=(8, 6)) # Adjust the size as needed # Use imshow to display the confusion matrix cax = ax.matshow(conf_matrix, cmap=plt.cm.Blues) # Add colorbar for reference fig.colorbar(cax)

# Add annotations with the confusion matrix values for (i, j), val in np.ndenumerate(conf_matrix): ax.text(j, i, f'{val:d}', ha='center', va='center', color='white' if val > conf_matrix.max()/2 else 'black') # Set labels for axes ax.set_xlabel('Predicted Label') ax.set_ylabel('True Label') # Adjust the tick labels on x and y axis ax.set_xticks(np.arange(len(np.unique(y_test)))) ax.set_yticks(np.arange(len(np.unique(y_test)))) ax.set_xticklabels(np.unique(y_test)) ax.set_yticklabels(np.unique(y_test)) # Show the plot plt.show() Interpretation ¶ The confusion matrix shows the number of correct and incorrect predictions for each class. The classification report provides key metrics such as precision, recall, and F1- score, which help us understand the accuracy of our model. Conclusion ¶ This exercise demonstrated the process of implementing a logistic regression model

Attribute Selection Measures ¶ Attribute selection measure is a heuristic for selecting the splitting criterion that partition data into the best possible manner. It is also known as splitting rules because it helps us to determine breakpoints for tuples on a given node. ASM provides a rank to each feature (or attribute) by explaining the given dataset. The best ASM is used to split the node. Common measures are: • Information Gain • Gini Index • Chi-Square • Reduction in Variance Conclusion ¶ Decision Trees are a powerful tool for classification and regression. They are intuitive and easy to explain but can become complex and overfit the data. Therefore, understanding how to tune and prune a Decision Tree is essential for effective modeling. Exercise 3 Understanding Decision Trees with the Iris Dataset ¶ Introduction ¶ In this exercise, we'll use the Iris dataset to build a decision tree classifier. We'll visualize the tree and interpret the results to gain insights into how decision trees make predictions. Load the Dataset ¶ The Iris dataset is available through the sklearn.datasets module. We'll start by loading the data and taking a look at its structure. In [16]: from sklearn.datasets import load_iris import pandas as pd # Load the iris dataset iris = load_iris() df_iris = pd.DataFrame(iris.data, columns=iris.feature_names) df_iris['species'] = pd.Categorical.from_codes(iris.target, iris.target_names) # Display the first 5 rows of the dataset df_iris.head() Out[16]: sepal length (cm) sepal width (cm) petal length (cm) petal width (cm) species 0 5.1 3.5 1.4 0.2 setosa 1 4.9 3.0 1.4 0.2 setosa 2 4.7 3.2 1.3 0.2 setosa 3 4.6 3.1 1.5 0.2 setosa 4 5.0 3.6 1.4 0.2 setosa Exploratory Data Analysis (EDA) ¶ Before training our model, it's important to explore the dataset to understand the distribution of the different features and classes.

In [17]: import seaborn as sns import matplotlib.pyplot as plt # Pairplot to visualize the relationships between features sns.pairplot(df_iris, hue='species') plt.show()

Splitting the Data ¶ We'll split our dataset into a training set and a test set to evaluate the performance of our model. In [18]: from sklearn.model_selection import train_test_split # Split the data X = df_iris.iloc[:, :-1] # Features y = df_iris['species'] # Target variable X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42) Training the Decision Tree Classifier ¶ With our data split, we can train a decision tree classifier. In [19]: from sklearn.tree import DecisionTreeClassifier # Initialize the classifier dt_classifier = DecisionTreeClassifier(random_state=42) # Train the classifier dt_classifier.fit(X_train, y_train) Out[19]: DecisionTreeClassifier(random_state=42) Visualizing the Decision Tree ¶ We can visualize the decision tree using the plot_tree function from sklearn . In [20]: from sklearn.tree import plot_tree # Convert the class names from a NumPy array to a list class_names_list = iris.target_names.tolist() # Visualize the decision tree plt.figure(figsize=(20,10)) plot_tree(dt_classifier, feature_names=iris.feature_names, class_names=class_names_list, filled=True) plt.show()

Evaluating the Classifier ¶ Finally, we evaluate the performance of the classifier using the test data. In [21]: from sklearn.metrics import classification_report # Predictions y_pred = dt_classifier.predict(X_test) # Classification report print(classification_report(y_test, y_pred)) precision recall f1-score support

How it Works ¶ 1. Bootstrap Sampling : Random Forest starts by selecting random samples from the dataset using bootstrap sampling. 2. Building Trees : It then builds a decision tree for each sample. The trees are grown to the largest extent possible and there is no pruning. 3. Random Feature Selection : During the construction of trees, only a random subset of features is considered for splitting at each node. 4. Aggregation : For classification, each tree votes and the most popular class is chosen as the final result. For regression, the average prediction of all trees is used. Applications ¶ • Banking : For detecting customers likely to default on loans. • Medicine : To identify the correct combination of components in medicine. • Stock Market : To predict stock behavior. • E-commerce : For predicting whether a customer will like a product recommendation. Random Forests are a powerful tool in the machine learning toolkit but should be used with consideration of their complexity and computational cost. Exercise 4 Predicting Wine Quality with Random Forest ¶ Introduction ¶ In this exercise, we'll use the Wine Quality dataset to build a Random Forest classifier that predicts the quality of white wines. We'll explore the importance of different physicochemical properties in determining wine quality. Load the Dataset ¶ The Wine Quality dataset can be downloaded from the UCI Machine Learning Repository. We'll load the data and inspect the first few rows to understand its structure. In [22]: import pandas as pd # Load the dataset url = 'http://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/ winequality-white.csv' df_wine = pd.read_csv(url, sep=';') # Display the first 5 rows of the dataset df_wine.head() Out[22]: fixed acidity volatile acidity citric acid residual sugar chlorides free sulfur dioxide total sulfur dioxide density pH sulphates a 0 7.0 0.27 0.36 20.7 0.045 45.0 170.0 1.0010 3.00 0.45 8 1 6.3 0.30 0.34 1.6 0.049 14.0 132.0 0.9940 3.30 0.49 9 2 8.1 0.28 0.40 6.9 0.050 30.0 97.0 0.9951 3.26 0.44 1 3 7.2 0.23 0.32 8.5 0.058 47.0 186.0 0.9956 3.19 0.40 9 4 7.2 0.23 0.32 8.5 0.058 47.0 186.0 0.9956 3.19 0.40 9

fixed acidity volatile acidity citric acid residual sugar chlorides free sulfur dioxide total sulfur dioxide density pH sulphates a Exploratory Data Analysis (EDA) ¶ Let's perform EDA to visualize the distribution of wine qualities and the relationship between the physicochemical properties and quality. In [23]: import seaborn as sns import matplotlib.pyplot as plt # Visualize the distribution of wine quality ratings sns.countplot(x='quality', data=df_wine) plt.title('Distribution of Wine Quality') plt.show() # Box plots for each feature df_wine.drop('quality', axis=1).plot(kind='box', subplots=True, layout=(3,4), figsize=(20,10), title='Boxplot of physicochemical properties') plt.show()