1. Decision How some classification methods can be used to generate rankings instead of class labels,
and how these rankings can be used to generate multiple confusion matrices for the same classifier.
When it comes to decision making there are strategies that can be used. One approach is to use a
classification method to organize groups of cases and then act based on the ranked cases. This method helps
in selecting the model for making the best choice, in exceptional situations relying on their predicted values.
Another option is to rank instances based on their scores and then act upon the cases that have the rankings.
Ranking cases according to the likelihood of belonging to a category works well when the focus is on
instances with the expected value assuming consistent costs and benefits for each category. Combining a
classifier with a threshold results in a confusion matrix. In scenarios, there may be limitations or constraints
on actions, such as having a fixed budget for a competition where targeting the most qualified individuals is
paramount. Ranking cases according to their likelihood of belonging to a category is appropriate when
aiming at targeting cases with potential value while maintaining consistent costs and benefits, across
categories.
2. How Profit Curves can be use to extend these multiple confusion matrices into multiple expected
values.
As we make changes, to the inputs the profit also changes accordingly. This change depends on the costs and
benefits associated with a model. Both approaches are part of what we call the profit curve. To create this
curve, we compile a list of instances each with an expected score arranged in descending order based on
their scores. Then we use a classifier to estimate the profit. By selecting cut points on this list we can
determine the portion that's likely to generate positive results and calculate an approximate profit, for each
cut point. These data points are then plotted to generate the profit curve. Basically, each curve represents
how adjusting a classifier's threshold value at positions can impact its performance.
3. How ROC Curves can be used to visualize the performances of the various confusion matrices.
A ROC curve is a representation that shows how well a classification model performs at classification
thresholds. It considers two factors; the rate of identified positives and the rate of falsely identified positives.
When it comes to predicting profits, accuracy depends on the costs and benefits associated with categories,
in the cost-benefit matrix. In real-life situations like detecting credit card fraud occurrence can vary by
region. Change throughout the month. The ROC graph provides insights into how a classifier balances,
between identifying positives (benefits) and incorrectly identifying negatives (costs). What makes the ROC
graph unique is its ability to showcase the classifier's performance independently from settings. Even though
costs, benefits, and class distributions may shift over time the fundamental shape of the curve remains
consistent.
4. How AUC (Area Under the ROC Curve) can summarize an ROC Curve in a single number.
AUC, which stands for "Area Under the ROC Curve " is a metric used to evaluate performance, in
classification tasks. It calculates the area beneath the ROC curve, which covers all classification thresholds
from (0,0) to (1,1). AUC provides insight into the probability of the model ranking a selected example
higher, than a randomly chosen negative example. This probability exceeds what would be expected by
chance.
5. How Cumulative Response & Lift Curves can summarize how well a model performs as compared to
a random-guessing classifier.
To assess the model's classification performance, evaluating class probabilities and scoring ROC curves
prove to be a tool. These curves offer an established method that simplifies assumptions and provides a
comprehensive understanding of performance. However, it's crucial to proceed with caution when utilizing
these curves particularly if the initial proportion of instances, in the population is uncertain or inaccurately
reflected in the test results.
