Performance Metric of fitted model

Authors

Pankaj Das (https://orcid.org/0000-0003-1672-2502)

Introduction

Performance metrics are essential tools for evaluating the accuracy and effectiveness of predictive models. These metrics provide quantifiable measures to assess how well a model’s predictions match the actual outcomes. Here are some commonly used performance metrics:

1.Mean Absolute Error(MAE): It is the average distance between Predicted and original values. Basically, it gives how we have predicted from the actual output. However, there is one limitation i.e. it doesn’t give any idea about the direction of the error which is whether we are under-predicting or over-predicting our data.low MAE values indicate that the model is correctly predicting. Larger MAE values indicate that the model is poor at prediction.

2.Mean Squared Error(MSE): It is similar to mean absolute error but the difference is it takes the square of the average of between predicted and original values. The main advantage to take this metric is here, it is easier to calculate the gradient whereas, in the case of mean absolute error, it takes complicated programming tools to calculate the gradient. By taking the square of errors it pronounces larger errors more than smaller errors, we can focus more on larger errors.

3.Root Mean Square Error(RMSE): We can say that RMSE is a metric that can be obtained by just taking the square root of the MSE value. As we know that the MSE metrics are not robust to outliers and so are the RMSE values. This gives higher weightage to the large errors in predictions.

4.R-squared (R²): The coefficient of determination also called the R-Square is used to evaluate the performance of a linear regression model. It is the amount of variation in the output-dependent attribute which is predictable from the input independent variable(s). It is used to check how well-observed results are reproduced by the model, depending on the ratio of total deviation of results described by the model.

5.Accuracy Model accuracy is a measure of how well a machine learning (ML) model is able to make predictions or decisions based on data. It is a common metric used to evaluate the performance of ML models, and can be used to compare the performance of different models or to assess the effectiveness of a particular model for a given task.

6.CVRME: Coefficient of variation of Root mean square is measure of variation of the error in a prediction model, normalized by the mean of the target variable.CVRMSE is defined as the ratio of the root mean square error (RMSE) to the mean of the target variable, multiplied by 100% to express it as a percentage.

CVRMSE=(RMSE/mean(target variable))*100 %

CVRMSE measures the average distance between predicted values and the true values and thus reflects the accuracy of the model. CVRMSE reflects the relative size of errors in the model, compared to the scale of the target variable. CVRMSE is useful metric when target variable has a large scale of the target variable has a large scale or a wide range of values , as it provides a standardized measure of model’s performance. A lower CVRMSE indicates a more accurate and reliable model, while a higher CVRMSE indicates a less accurate and more variable model. CVRMSE values can be used in combination with other metrics, such as R-Squared and Mean absolute error (MAE), to provide a comprehensive evaluation of the model’s performance.

7.NRMSE: Normalized root mean square error is a measure of the accuracy of a prediction model, representing the ratio of the root mean squared error to the range of the dependent variable.It is used to compare the performance of different models or to evaluate the accuracy of a model over time. The formula of NRMSE is

NRMSE= RMSE/ (Y_max - Y_min)

where Y_max is maximum value of target variable Y_min is minimum value of target variable

NRMSE values range from 0 to 1 with lower values indicating better model accuracy.The values can be used to compare the performance of different regression models or to track the performance of the same model over time.

8.MAPE: Mean Absolute Percentage Error (MAPE) is a metric used to measure the accuracy of a predictive model. It represents the average of the absolute percentage errors between the actual values and the predicted values.MAPE is expressed as a percentage, making it easy to interpret. It indicates the average percentage error made by the model in its predictions. Lower MAPE values represent better predictive accuracy.

ME: Mean Error (ME) is a metric that measures the average difference between the actual values and the predicted values. Unlike other metrics that only consider the magnitude of errors, ME retains the sign of the errors, indicating whether predictions are generally over or under the actual values. ME can be positive or negative. A positive ME indicates that, on average, the model’s predictions are lower than the actual values (underestimation). A negative ME indicates that, on average, the model’s predictions are higher than the actual values (overestimation). An ME close to zero indicates that the model does not have a significant bias in either direction.

Functions in the R package

RMSE(actual, predicted): It help in RMSE calculation of the fitted model

NRMSE(actual, predicted): It help inNRMSE calculation of the fitted model.

CVRMSE(actual, predicted): It help in CVRMSE calculation of the fitted model.

R2(actual, predicted): It help in RSquare calculation of the fitted model.

accuracy(actual, predicted): It help in Accuracy calculation of the fitted model.

ME(actual, predicted): It help inMaximum error calculation of the fitted model.

MAPE(actual, predicted): It help in MAPE calculation of the fitted model.

MAE(actual, predicted): It help in MAE calculation of the fitted model.

Uses of the Performance Metrics

1.Model Evaluation: Purpose: Metrics such as MAE, MSE, RMSE, and R² help evaluate how well a predictive model performs.

Application: By comparing these metrics across different models, data scientists can determine which model provides the best fit for the data.

2.Model Selection: Purpose: Performance metrics guide the selection of the most appropriate model for a given problem.

Application: When multiple models are developed, metrics are used to select the model that minimizes error and maximizes explained variance.

3.Model Tuning: Purpose: Metrics assist in hyperparameter tuning to improve model performance.

Application: During processes like cross-validation, performance metrics are used to adjust model parameters to achieve optimal performance.

4.Benchmarking: Purpose: Metrics provide benchmarks to compare current models against new or alternative models.

Application: Metrics allow for tracking performance improvements over time or comparing against industry standards.

5.Error Analysis: Purpose: Identifying the type and magnitude of prediction errors helps in understanding model limitations.

Application: Metrics like MSE and RMSE highlight large errors, while MAE indicates average error, guiding further investigation and refinement.

6.Decision Making: Purpose: Performance metrics inform business and operational decisions based on model predictions.

Application: Metrics help stakeholders understand the reliability of predictions, influencing strategic decisions in areas like finance, healthcare, marketing, and more.

7.Model Reporting: Purpose: Clear communication of model performance to stakeholders.

Application: Metrics provide quantitative evidence of model effectiveness, useful for reports, presentations, and documentation.

8.Algorithm Comparison: Purpose: Metrics facilitate comparison between different types of algorithms (e.g., linear regression vs. decision trees).

Application: By evaluating metrics, one can determine which algorithm better suits the data characteristics and problem requirements.

9.Ensuring Model Robustness: Purpose: To ensure that a model performs well on unseen data.

Application: Metrics help in validating the robustness and generalizability of the model by comparing performance on training and test datasets.

Reference

Das, P., Jha, G. K., Lama, A., Parsad, R. and Mishra, D. (2020). Empirical Mode Decomposition based Support Vector Regression for Agricultural Price Forecasting. Indian Journal of Extension Education, 56(2):7-12.(http://krishi.icar.gov.in/jspui/handle/123456789/44138).

Bharti, Das,P.,Banerjee, R., Ahmad, T., Devi, S. and Verma G. (2023).Machine-LearningBased Apple Yield Prediction using Morphological Characters. Horticulturae. 9, (4), 436.https://doi.org/10.3390/horticulturae9040436

Das,P., Jha, G. K., Lama, A.and Parsad, R.(2023). Crop Yield Prediction using Hybrid Machine Learning approach: A Case study of Lentil (Lens culinaris Medik.) Agriculture, 13 (3), 596. https://doi.org/10.3390/agriculture13030596

Das, P. Jha, G. K. and Lama, A. (2023). Empirical Mode Decomposition Based Ensemble Hybrid Machine Learning Models for Agricultural Commodity Price Forecasting. Statistics and Applications, 21(1),99-112.(http://krishi.icar.gov.in/jspui/handle/123456789/77772).

Das, P., Jha, G. K., Lama, A. and Bharti (2022). EMD-SVR Hybrid Machine Learning Model and its Application in Agricultural Price Forecasting. Bhartiya Krishi Anusandhan Patrika. (DOI: 10.18805/BKAP385)

Das, P. (2019). Study On Machine Learning Techniques Based Hybrid Model for Forecasting in Agriculture. Published Ph.D. Thesis.

Choudhury, K., Jha, G. K., Das, P. and Chaturvedi, K. K. (2019). Forecasting Potato Price using Ensemble Artificial Neural Networks. Indian Journal of Extension Education, 55(1):71-77.(http://krishi.icar.gov.in/jspui/handle/123456789/44873).

Das, P., Lama, A. and Jha, G. K. (2022). Variational Mode Decomposition based Machine Learning Models Optimized with Genetic Algorithm for Price Forecasting. Journal of the Indian Society of Agricultural Statistics, 76(3), 141-150. (http://krishi.icar.gov.in/jspui/handle/123456789/76648)

#
library(PerMat)
##Example how the package works

#Dataset generation

actual <- c(100, 150, 200, 250, 300, 350, 400, 450, 500, 550)
predicted <- c(95, 148, 210, 245, 290, 360, 395, 440, 510, 540)

#RMSE calculation of the fitted model
RMSE(actual, predicted)
#> Mean Squared Error: 67.9
#> Root Mean Squared Error: 8.24014562978107

#NRMSE calculation of the fitted model
NRMSE(actual, predicted)
#> Normalised Root Mean Squared Error:0.0183114347328468

#CVRMSE calculation of the fitted model

CVRMSE(actual, predicted)
#> Coefficient of Variation of Root Mean Squared Error:0.0253542942454802
#> Coefficient of Variation of Root Mean Squared Error (In percentage):2.53542942454802

#RSquare calculation of the fitted model
R2(actual, predicted)
#> Coefficient of Determination (R-Square):0.996851326601227
#> Percentage of Variation Expressed by The Model:99.6851326601227



#Maximum error calculation of the fitted model
ME(actual, predicted)
#> Maximum Error: 10

#MAPE calculation of the fitted model
MAPE(actual, predicted)
#> Mean Absolute Percentage Error: 0.0268142135642136
#> Mean Absolute Percentage Error (In percentage): 2.68142135642136

#MAE calculation of the fitted model
MAE(actual, predicted)
#> Mean Absolute Error: 7.7

#Accuracy calculation of the fitted model
accuracy(actual, predicted)
#> Accuracy of Model: 0.992900072150072