wconf: Weighted Confusion Matrix

Alexandru Monahov

2023-12-12

The wconf package

wconf is a package that allows users to create weighted confusion matrices and accuracy scores

Used to improve the model selection process, the package includes several weighting schemes which can be parameterized, as well as the option for custom weight configurations. Furthermore, users can decide whether they wish to positively or negatively affect the accuracy score as a result of applying weights to the confusion matrix. “wconf” integrates with the “caret” package, but it can also work standalone when provided data in matrix form.

About confusion matrices

Confusion matrices are used to visualize the performance of classification models in tabular format. A confusion matrix takes the form of an “n x n” matrix depicting:

  1. the reference category, in columns;

  2. the predicted category, in rows;

  3. the number of observation corresponding to each combination of “reference - predicted” category couples, as cells of the matrix.

Visually, the simplest binary classification confusion matrix takes on the form:

\[ A = \begin{bmatrix}TP & FP \\FN & TN\\ \end{bmatrix} \] where:

\(TP\) - True Positives - the number of observations that were “positive” and were correctly predicted as being “positive”

\(TN\) - True Negatives - the number of originally “negative” observations that were correctly predicted by the model as being “negative”.

\(FP\) - False Positives - also called “Type 1 Error” - represents observations that are in fact “negative”, but were incorrectly classified by the model as being “positive”.

\(FN\) - False Negatives - also called “Type 2 Error” - represents observations that are in fact “positive”, but were incorrectly classified by the model as being “negative”.

The traditional accuracy metric is compiled by adding the true positives and true negatives, and dividing them by the total number of observations.

\[ A = \frac{TP + TN} {N} \]

A weighted confusion matrix consists in attributing weights to all classification categories based on their distance from the correctly predicted category. This is important for multi-category classification problems (where there are three or more categories), where distance from the correctly predicted category matters.

The weighted confusion matrix, for the simple binary classification, takes the form:

\[ A = \begin{bmatrix}w1*TP & w2*FP \\w2*FN & w1*TN\\ \end{bmatrix} \]

In the case of the weighted confusion matrix, a weighted accuracy score can be calculated by summing up all of the elements of the matrix and dividing the resulting amount by the number of observations.

\[ A = \frac{w1*TP + w2*FP + w2*FN + w1*TN} {N} \]

References

For more details on the method, see the paper:

Monahov, A. (2023). wconf: The weighted confusion matrix and accuracy scores package for R

Functions

weightmatrix - configure and visualize a weight matrix

This function compiles a weight matrix according to one of several weighting schemas and allows users to visualize the impact of the weight matrix on each element of the confusion matrix.

In R, simply call the function:

weightmatrix(n, weight.type = "arithmetic", weight.penalty = FALSE, standard.deviation = 2, geometric.multiplier = 2, interval.high=1, interval.low = -1, custom.weights = NA, plot.weights = FALSE) {

The function takes as input:

n – the number of classes contained in the confusion matrix.

weight.type – the weighting schema to be used. Can be one of: “arithmetic” - a decreasing arithmetic progression weighting scheme, “geometric” - a decreasing geometric progression weighting scheme, “normal” - weights drawn from the right tail of a normal distribution, “interval” - weights contained on a user-defined interval, “custom” - custom weight vector defined by the user.

weight.penalty – determines whether the weights associated with non-diagonal elements generated by the “normal”, “arithmetic” and “geometric” weight types are positive or negative values. By default, the value is set to FALSE, which means that generated weights will be positive values.

standard.deviation – standard deviation of the normal distribution, if the normal distribution weighting schema is used.

geometric.multiplier – the multiplier used to construct the geometric progression series, if the geometric progression weighting scheme is used.

interval.high – the upper bound of the weight interval, if the interval weighting scheme is used.

interval.low – the lower bound of the weight interval, if the interval weighting scheme is used.

custom.weights – the vector of custom weights to be applied, is the custom weighting scheme was selected. The vector should be equal to “n”, but can be larger, with excess values being ignored.

plot.weights – optional setting to enable plotting of weight vector, corresponding to the first column of the weight matrix

The function outputs a matrix:

w the nxn weight matrix.

wconfusionmatrix - compute a weighted confusion matrix

This function calculates the weighted confusion matrix by multiplying, element-by-element, a weight matrix with a supplied confusion matrix object.

In R, simply call the function:

wconfusionmatrix(m, weight.type = "arithmetic", weight.penalty = FALSE, standard.deviation = 2, geometric.multiplier = 2, interval.high=1, interval.low = -1, custom.weights = NA, print.weighted.accuracy = FALSE) {

The function takes as input:

m – the caret confusion matrix object or simple matrix.

weight.type – the weighting schema to be used. Can be one of: “arithmetic” - a decreasing arithmetic progression weighting scheme, “geometric” - a decreasing geometric progression weighting scheme, “normal” - weights drawn from the right tail of a normal distribution, “interval” - weights contained on a user-defined interval, “custom” - custom weight vector defined by the user.

weight.penalty – determines whether the weights associated with non-diagonal elements generated by the “normal”, “arithmetic” and “geometric” weight types are positive or negative values. By default, the value is set to FALSE, which means that generated weights will be positive values.

standard.deviation – standard deviation of the normal distribution, if the normal distribution weighting schema is used.

geometric.multiplier – the multiplier used to construct the geometric progression series, if the geometric progression weighting scheme is used.

interval.high – the upper bound of the weight interval, if the interval weighting scheme is used.

interval.low – the lower bound of the weight interval, if the interval weighting scheme is used.

custom.weights – the vector of custom weights to be applied, is the custom weighting scheme was selected. The vector should be equal to “n”, but can be larger, with excess values being ignored.

print.weighted.accuracy – optional setting to print the weighted accuracy metric, which represents the sum of all weighted confusion matrix cells divided by the total number of observations.

The function outputs a matrix:

w_m the nxn weighted confusion matrix.

Examples

Producing a weighted confusion matrix in conjunction with the caret package

This example provides a real-world usage example of the wconf package on the Iris dataset included in R.

We will attempt the more difficult task of predicting petal length from sepal width. In addition, for this task, we are only given categorical information about the length of the petals, specifically that they are:

Numeric data is available for the sepal width.

Using caret, we train a multinomial logistic regression model to fit the numeric sepal width onto our categorical petal length data. We run 10-fold cross-validation, repeated 3 times to avoid overfitting and find optimal regression coefficient values for various data configurations.

Finally, we extract the confusion matrix. We wish to weigh the confusion matrix to represent preference for observations fitted closer to the correct value. We would like to assign some degree of positive value to observations that are incorrectly classified, but are close to the correct category. Since our categories are equally spaced, we can use an arithmetic weighing scheme.

Let’s first visualize what this weighting schema would look like:

# View the weight matrix and plot for a 3-category classification problem, using the arithmetic sequence option.

weightmatrix(3, weight.type = "arithmetic", plot.weights = TRUE)
#> Error in weightmatrix(3, weight.type = "arithmetic", plot.weights = TRUE): could not find function "weightmatrix"

To obtain the weighted confusion matrix, we run the “wconfusionmatrix” command and provide it the confusion matrix object generated by caret, a weighting scheme and, optionally, parameterize it to suit our objectives. Using the “wconfusionmatrix” function will automatically determine the dimensions of the weighing matrix and the user need only specify the parameters associated with their weighting scheme of choice.

The following block of code produces the weighted confusion matrix, to out specifications.

# Load libraries and perform transformations
library(caret)
#> Warning: package 'caret' was built under R version 4.2.3
#> Loading required package: ggplot2
#> Warning: package 'ggplot2' was built under R version 4.2.3
#> Loading required package: lattice
#> Warning: package 'lattice' was built under R version 4.2.3
data(iris)
iris$Petal.Length.Cat = cut(iris$Petal.Length, breaks=c(1, 3, 5, 7), right = FALSE)

# Train multinomial logistic regression model using caret
set.seed(1)
control <- trainControl(method="repeatedcv", number=10, repeats=3)
model <- train(Petal.Length.Cat ~ Sepal.Width, data=iris, method="multinom", trace = FALSE, trControl=control)

# Extract original data, predicted values and place them in a table
y = iris$Petal.Length.Cat
yhat = predict(model)
preds = table(data=yhat, reference=y)

# Construct the confusion matrix
confmat = confusionMatrix(preds)

# Compute the weighted confusion matrix and display the weighted accuracy score
wconfusionmatrix(confmat, weight.type = "arithmetic", print.weighted.accuracy = TRUE)
#> Error in wconfusionmatrix(confmat, weight.type = "arithmetic", print.weighted.accuracy = TRUE): could not find function "wconfusionmatrix"

About the author

The wconf: Weighted Confusion Matrix package was programmed by Dr. Alexandru Monahov.

Alexandru Monahov holds a PhD in Economics from the University Cote d’Azur (Nice, France) and a Professional Certificate in Advanced Risk Management from the New York Institute of Finance (New York, United States). His Master’s Degree in International Economics and Finance and his Bachelor’s Degree in Economics and Business Administration were completed at the University of Nice (Nice, France).

His professional activity includes working for the Bank of England as a Research Economist and as Expert Consultant at the National Bank of Moldova, within the Financial Stability Division. Alexandru also provides training for professionals in finance from Central Banks and Ministries of Finance at the Center of Excellence in Finance (Ljubljana, Slovenia) and the Centre for Central Banking Studies (London, UK). Previously, he worked as assistant and, subsequently, associate professor at the University of Nice and IAE in France, where he taught Finance, Economics, Econometrics and Business Administration. He developed training and professional education curricula for the Chambers of Commerce and Industry and directed several continuing education programs.

Dr. Monahov was awarded funding for continuing professional education by the World Bank through the Reserve Advisory & Management Partnership Program, a PhD scholarship by the Doctoral School of Nice and a scholarship of the French Government.

Copyright Alexandru Monahov, 2023.

You may use, modify and redistribute this code, provided that you give credit to the author and make any derivative work available to the public for free.