Regression performance metrics and indices

Description

The metrica package compiles +80 functions to assess regression (continuous) and classification (categorical) prediction performance from multiple perspectives.

For regression models, it includes 4 plotting functions (scatter, tiles, density, & Bland-Altman plots), and 48 prediction performance scores including error metrics (MBE, MAE, RAE, RMAE, MAPE, SMAPE, MSE, RMSE, RRMSE, RSR, PBE, iqRMSE), error decomposition (MLA, MLP, PLA, PLP, PAB, PPB, SB, SDSD, LCS, Ub, Uc, Ue), model efficiency (NSE, E1, Erel, KGE), indices of agreement (d, d1, d1r, RAC, AC, lambda), goodness of fit (r, R2, RSS, TSS, RSE), adjusted correlation coefficients (CCC, Xa, distance correlation-dcorr-, maximal information coefficient -MIC-), variability (uSD, var_u), and symmetric regression coefficients (B0_sma, B1_sma). Specifically for time-series predictions, metrica also includes the Mean Absolute Scaled Error (MASE).

For supervised models, always keep in mind the concept of “cross-validation” since predicted values should ideally come from out-of-bag samples (unseen by training sets) to avoid overestimation of the prediction performance.

Using the functions.

There are two basic arguments common to all metrica functions: (i) obs(Oi; observed, a.k.a. actual, measured, truth, target, label), and (ii) pred (Pi; predicted, a.k.a. simulated, fitted, modeled, estimate) values.

Optional arguments include data that allows to call an existing data frame containing both observed and predicted vectors, and tidy, which controls the type of output as a list (tidy = FALSE) or as a data.frame (tidy = TRUE).

For regression, some specific functions for regression also require to define the axis orientation. For example, the slope of the symmetric linear regression describing the bivariate scatter (SMA).

List of regression prediction performance metrics (continuous variables)

#	Metric	Definition	Details	Formula
01	RSS	Residual sum of squares (a.k.a. as sum of squares)	The sum of squared differences between predicted and observed values. It represents the base of many error metrics using squared scale such as the MSE	\(RSS = \sum{(O_i - P_i)^2}\)
02	TSS	Total sum of squares	The sum of the squared differences between the observations and its mean. It is used as a reference error, for example, to estimate explained variance	\(TSS = \sum{(O_i - \bar{O})^2}\)
03	var_u	Sample variance, uncorrected	The mean of sum of squared differences between values of an x and its mean (divided by n, not n-1)	\(var_u = \frac{1}{n}\sum{(x - \bar{x})^2}\)
4	uSD	Sample standard deviation, uncorrected	The square root of the mean of sum of squared differences between values of an x and its mean (divided by n, not n-1)	\(uSD = \sqrt{\frac{1}{n}\sum{(x_i - \bar{x})^2}}\)
04	B0	Intercept of SMA regression	SMA is a symmetric linear regression (invariant results/interpretation to axis orientation) recommended to describe the bivariate scatter instead of OLS regression (classic linear model, which results vary with the axis orientation). B0 could be used to test agreement along with B1 (H0: B0 = 0, B1 = 1) . Warton et al. (2006)	\(\beta_{0_{PO}} = \bar{P} - \frac{S_P}{S_O} \, ;\, \beta_{0_{OP}} = \bar{O} - \frac{S_O}{S_P}\)
06	B1	Slope of SMA regression	SMA is a symmetric linear regression (invariant results/interpretation to axis orientation) recommended to describe the bivariate scatter instead of OLS regression (classic linear model, which results vary with the axis orientation). B1 could be used to test isometry of the PO scatter (H0: B1 = 1). B1 also represents the ratio of standard deviations (So and Sp). Warton et al. (2006)	\(\beta_{1_{PO}} = \frac{S_P}{S_O}\, ;\, \beta_{1_{OP}} = \frac{S_O}{S_P}\)
07	r	Pearson’s correlation coefficient	Strength of linear association between P and O. However, it measures “precision” but no accuracy. Kirch (2008)	\(r = \frac{{S}_{PO}}{{S}_{P}{S}_{O}}\)
8	R2	Coefficient of determination	Strength of linear association between P and O. However, it measures “precision” but no accuracy	\(R^{2} = \frac{{S^2}_{PO}}{{S^2}_{P}{S^2}_{O}}\)
09	Xa	Accuracy coefficient	Measures accuracy. Used to adjust the precision measured by r to estimate agreement	$ X_a = $
10	CCC	Concordance correlation coefficient	Tests agreement. It presents both precision (r) and accuracy (Xa) components. Easy to interpret. Lin (1989)	\(CCC = r * X_a\)
11	MAE	Mean Absolute Error	Measures both lack of accuracy and precision in absolute scale. It keeps the same units than the response variable. Less sensitive to outliers than the MSE or RMSE. Willmott & Matsuura (2005)	\(MAE = \frac{1}{n} \sum{|O_i - P_i|}\)
12	RMAE	Relative Mean Absolute Error	Normalizes the MAE with respect to the mean of observations	\(RMAE = \frac{\frac{1}{n} \sum{|O_i - P_i|}}{\bar{O}}\)
13	MAPE	Mean Absolute Percentage Error	Percentage units (independent scale). Easy to explain and to compare performance across models with different response variables. Asymmetric and unbounded.	\(MAPE = \frac{1}{n}\sum{|\frac{O_i-P_i}{O_i}|}\)
14	SMAPE	Symmetric Mean Absolute Percentage Error	SMAPE tackles the asymmetry issues of MAPE and includes lower (0%) and upper (200%) bounds. Makridakis (1993)	\(SMAPE = \frac{1}{n}\sum{|\frac{|O_i-P_i|}{(O_i+P_i)/2}|}\)
15	RAE	Relative Absolute Error	RAE normalizes MAE with respect to the total absolute error. Lower bound at 0 (perfect fit) and no upper bound (infinity)	\(RAE = \frac{\sum{|P_i-O_i|}}{\sum{|O_i-\bar{O}|}}\)
16	RSE	Relative Squared Error	Proportion of the total sum of squares that corresponds to differences between predictions and observations (residual sum of squares)	\(RSE = \frac{\sum{(P_i-O_i)^2}}{\sum{(O_i-\bar{O})^2}}\)
17	MBE	Mean Bias Error	Main bias error metric. Same units as the response variable. Related to differences between means of predictions and observations. Negative values indicate overestimation. Positive values indicate underestimation. Unbounded. Also known as average error. Janssen & Heuberger (1995)	\(MBE = \frac{1}{n} \sum{(O_i - P_i)} = \bar{O}-\bar{P}\)
18	PBE	Percentage Bias Error	Useful to identify systematic over or under predictions. Percentage units. As the MBE, PBE negative values indicate overestimation, while positive values indicate underestimation. Unbounded. Gupta et al. (1999)	\(PBE = 100 \frac{\sum{(P_i-O_i)}}{\sum O_i}\)
19	PAB	Percentage Additive Bias	Percentage of the MSE related to systematic additive issues on the predictions. Related to difference of the means of predictions and observations	\(PAB = 100\frac{(\bar{O}-\bar{P})^2}{\frac{1}{n} \sum{(P_i - O_i)^2}}\)
20	PPB	Percentage Proportional Bias	Percentage of the MSE related to systematic proportionality issues on the predictions. Related to slope of regression line describing the bivariate scatter	\(PPB = 100 \frac{S_O S_P}{\frac{1}{n} \sum{(P_i - O_i)^2}}\)
21	MSE	Mean Squared Error	Comprises both accuracy and precision. High sensitivity to outliers	\(MSE = \frac{1}{n} \sum{(P_i - O_i)^2}\)
22	RMSE	Root Mean Squared Error	Comprises both precision and accuracy, has the same units than the variable of interest. Very sensitive to outliers	\(RMSE = \sqrt{\frac{1}{n} \sum{(P_i - O_i)^2}}\)
23	RRMSE	Relative Root Mean Squared Error	RMSE normalized by the mean of observations	\(RRMSE = \frac{\sqrt{\frac{1}{n} \sum{(P_i - O_i)^2}}}{\bar{O}}\)
24	RSR	Root Mean Standard Deviation Ratio	RMSE normalized by the standard deviation of observations. Moriasi et al. (2007)	\(RSR = \frac{1}{n} \sum{(P_i - O_i)^2}\)
25	iqRMSE	Inter-quartile Normalized Root Mean Squared Error	RMSE normalized by the interquartile range length (between percentiles 25th and 75th)	\(iqRMSE = \frac{\sqrt{\frac{1}{n} \sum{(P_i - O_i)^2}}}{[75^{th} percentile - 25^{th} percentile]}\)
26	MLA	Mean Lack of Accuracy	Bias component of MSE decomposition. Correndo et al. (2021)	\(MLA = (\bar{O}-\bar{P})^2 + (S_O - S_P)^2\)
27	MLP	Mean Lack of Precision	Variance component of MSE decomposition. Correndo et al. (2021)	\(MLP = 2 S_O S_P (1-r)\)
28	RMLA	Root Mean Lack of Accuracy	Bias component of MSE decomposition expressed on the original units of interest. Correndo et al. (2021)	\(RMLA = \sqrt{(\bar{O}-\bar{P})^2 + (S_O - S_P)^2}\)
29	RMLP	Root Mean Lack of Precision	Variance component of MSE decomposition expressed on the original units of interest. Correndo et al. (2021)	\(RMLP = \sqrt{2 S_O S_P (1-r)}\)
30	PLA	Percentage Lack of Accuracy	Percentage of the MSE related to lack of accuracy (systematic differences) on the predictions. Correndo et al. (2021)	\(PLA = 100 \frac{(\bar{O}-\bar{P})^2 + (S_O - S_P)^2}{\frac{1}{n} \sum{(P_i - O_i)^2} }\)
31	PLP	Percentage Lack of Precision	Percentage of the MSE related to lack of precision (unsystematic differences) on the predictions. Correndo et al. (2021)	\(PLP = 100 \frac{2 S_O S_P (1-r)}{\frac{1}{n} \sum{(P_i - O_i)^2} }\)
32	SB	Squared Bias	Additive bias component, MSE decomposition. Kobayashi and Salam (2000)	\(SB=(\bar{O}-\bar{P})^2\)
33	SDSD	Product of Standard Deviations	Proportional bias component, MSE decomposition. Kobayashi and Salam (2000)	\(SDSD = S_O S_P\)
34	LCS	Lack of Correlation	Random error component, MSE decomposition. Kobayashi and Salam (2000)	\(LCS = 2 S_P S_O (1-r)\)
35	Ue	Random error proportion	The Ue estimates the proportion of the total sum of squares related to the random error (unsystematic error or variance) following the sum of squares decomposition suggested by Smith and Rose (1995) also known as Theil’s partial inequalities	\(Ue = \frac{2n(1-r)S_O S_P}{\sum{(O_i-P_i)^2}}\)
36	Uc	Lack of Consistency error proportion	The Uc estimates the proportion of the total sum of squares related to the lack of consistency (proportional bias) following the sum of squares decomposition suggested by Smith and Rose (1995) also known as Theil’s partial inequalities	\(Uc = \frac{n(S_O-S_P)^2}{\sum{(O_i-P_i)^2}}\)
37	Ub	Mean Bias error proportion	The Ub estimates the proportion of the total sum of squares related to the mean bias following the sum of squares decomposition suggested by Smith and Rose (1995) also known as Theil’s partial inequalities	\(Ub = \frac{n(\bar{O}-\bar{P})^2}{\sum{(O_i-P_i)^2}}\)
38	NSE	Nash and Sutcliffe’s Model Efficiency	Model efficiency using squared residuals normalized by the variance of observations. Nash and Sutcliffe (1970)	\(NSE = 1- \frac{\frac{1}{n} \sum{(P_i - O_i)^2}}{\frac{1}{n} \sum{(O_i - \bar{O})^2}}\)
39	E1	Absolute Model Efficiency	Model efficiency. Modification of NSE using absolute residuals instead of squared residuals. Legates and McCabe (1999)	\(E1 = 1- \frac{\sum{|P_i - O_i|}}{\sum{|O_i - \bar{O}|}}\)
40	Erel	Relative Model Efficiency	Compared to the NSE, the Erel is suggested as more sensitive to systematic over- or under-predictions. Krause et al. (2005)	\(Erel = 1- \frac{\sum{(\frac{P_i - O_i}{Oi})^2}}{\sum{(\frac{O_i - \bar{O}}{Oi})^2}}\)
41	KGE	Kling-Gupta Model Efficiency	Model efficiency with accuracy, precision, and consistency components. Kling et al. (2012)	\(KGE = 1- \sqrt{(r-1)^2+ (\frac{S_P}{S_O}-1)^2+(\frac{\bar{P}}{\bar{O}}-1)^2}\)
42	d	Index of Agreement	Measures accuracy and precision using squared residuals. Dimensionless (normalized). Bounded [0;1]. Asymmetric Willmott (1981)	\(d = 1- \frac{\sum{(O_i - P_i)^2}}{\sum{(|O_i - \bar{P}| + |P_i - \bar{O}|})^2}\)
43	d1	Modified Index of Agreement	Measures accuracy and precision using absolute residuals(1). Dimensionless (normalized). Bounded [0;1]. Asymmetric Willmott et al. (1985)	\(d1 = 1- \frac{\sum{|O_i - P_i}|}{\sum{(|O_i - \bar{O}| + |P_i - \bar{O}|})}\)
44	d1r	Refined Index of Agreement	Refines d1 by a modification on the denominator (potential error) to normalize absolute error. Willmott et al. (2012)	\(d1r = 1- \frac{\sum{|O_i - P_i}|}{2\sum{|O_i - \bar{O}|}}\)
45	RAC	Robinson’s Agreement Coefficient	RAC measures both accuracy and precision (general agreement). Dimensionless (normalized). Bounded [0;1]. Symmetric. Robinson (1957; 1959)	\(RAC = 1- \frac{\sum{(O_i - Z_i})^2 + \sum{(P_i - Z_i})^2}{\sum{(O_i - \bar{Z})^2 + \sum{(P_i - \bar{Z})^2} } }\) where \(Zi = \frac{O_i + P_i}{2}; \bar{Z} = \bar{O} + \bar{P}\)
46	AC	Ji and Gallo’s Agreement Coefficient	AC measures both accuracy and precision (general agreement). Dimensionless (normalized). Positively bounded [-infinity;1]. Symmetric. Ji and Gallo (2006)	\(AC = 1 - \frac{\sum{(O_i-P_i)^2}}{\sum{[(|\bar{P}-\bar{O}|+|O_i-\bar{O}|)(|\bar{P}-\bar{O}|+|P_i-\bar{P}|)]}}\)
47	lambda	Duveiller’s Lambda Coefficient	lambda measures both accuracy and precision. Dimensionless (normalized). Bounded [-1;1]. Symmetric. Equivalent to CCC when r is greater or equal to 0. Duveiller et al. (2016)	\(\lambda = 1 - \frac{\frac{1}{n}\sum(O_i-P_i)^2}{S^2_P+S^2_O+(\bar{O}-\bar{P})^2+ n^{-1}k}\) where \(k = 0\,\, if\,\, r \geq{0}\), otherwise \(k = 2|\sum[(O_i-\bar{O})(P_i-\bar{P})]|\)
48	dcorr	Distance correlation	Measures the dependency between to random vectors. Compared to Pearson’s r, it offers the advantage of considering both linear and nonlinear association patterns. It is based on a matrix of centered Euclidean distances compared to the distance of many shuffles of the data. It is dimensionless, bounded [0;1], and symmetric. dcorr = 0 characterizes independence between vectors. The closest to 1 the better. A disadvantage for the predicted-observed case is that values can be negatively correlated but producing a dcorr close to 1. Székely (2007)	\(dcorr = \sqrt{\frac{\mathcal{V}^2_n~(\mathbf{P,O})}{ {\sqrt{\mathcal{V}^2_n (\mathbf{P}) \mathcal{V}^2_n(\mathbf{O})} } }}\) See Székely (2007) for full details
49	MIC	Maximal Information Coefficient	Measures association between two variables based on “binning” (a.k.a. data bucketing) to reduce the influence of small observation errors. It is based on the “mutual information” concept of information theory, which measures the mutual dependence between two variables. It is dimensionless (normalized), bounded [0;1], and symmetric. Reshef et al. (2011)	\(MIC(D) = max_{PO<B(n)} M(D)_{X,Y} = max_{PO<B(n)} \frac{I^{(D,P,O)}}{log(\min{P,O})}\) where \(B(n) = n^{\alpha}\) is the search-grid size, \(I^{(D,P,O)}\) is the maximum mutual information over all grids P-by-O, of the distribution induced by \(D\) on a grid having P and O bins (where the probability mass on a cell of the grid is the fraction of points of D falling in that cell). See Reshef et al. 2011, for full details.
50	MASE	Mean Absolute Scaled Error	The MASE is especially well suited for time series predictions, as it scales (or normalize) the error based on in-sample MAE from the naive forecast method (a.k.a. random walk). It is dimensionless (normalized) and symmetric. The reference score is MASE = 1, which indicates that the model performs the same than a naive forecast (error with respect to previous historical observation). MASE <1 indicates that the model performs better than naive forecast, and MASE > 1 indicates a bad performance of the predictions. See Hyndman & Koehler (2006)	\(MASE = \frac{1}{n}(\frac{|O_i-P_i|}{ \frac{1}{T-1} \sum^T_{t=2}~|O_t - O_{t-1}| })\)

References:

1. Correndo et al. (2021). Revisiting linear regression to test agreement in continuous predicted-observed datasets. Agric. Syst. 192, 103194.
   

2. Duveiller et al. (2016). Revisiting the concept of a symmetric index of agreement for continuous datasets. Sci. Rep. 6, 1-14.
   

3. Gupta et al. (1999). Status of automatic calibration for hydrologic models: Comparison with multilevel expert calibration. J. Hydrologic Eng. 4(2): 135-143.
   

4. Janssen & Heuberger (1995). Calibration of process-oriented models. Ecol. Modell. 83, 55-66.
   

5. Ji & Gallo (2006). An agreement coefficient for image comparison. Photogramm. Eng. Remote Sensing 7, 823–833.
   

6. Kling et al. (2012). Runoff conditions in the upper Danube basin under an ensemble of climate change scenarios. J. Hydrol., 424-425, 264-277.
   

7. Kirch (2008). Pearson’s Correlation Coefficient. In: Kirch W. (eds) Encyclopedia of Public Health. Springer, Dordrecht.
   

8. Krause et al. (2005). Comparison of different efficiency criteria for hydrological model assessment. Adv. Geosci. 5, 89–97.
   

9. Kobayashi & Salam (2000). Comparing simulated and measured values using mean squared deviation and its components. Agron. J. 92, 345–352.
   

10. Legates & McCabe (1999). Evaluating the use of “goodness-of-fit” measures in hydrologic and hydroclimatic model validation. Water Resour. Res.
    

11. Lin (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics 45 (1), 255–268.
    

12. Makridakis (1993). Accuracy measures: theoretical and practical concerns. Int. J. Forecast. 9, 527-529.
    

13. Moriasi et al. (2007). Model Evaluation Guidelines for Systematic Quantification of Accuracy in Watershed Simulations. Trans. ASABE 50, 885–900.
    

14. Nash & Sutcliffe (1970). River flow forecasting through conceptual models part I - A discussion of principles. J. Hydrol. 10(3), 292-290.
    

15. Robinson (1957). The statistical measurement of agreement. Am. Sociol. Rev. 22(1), 17-25.
    

16. Robinson (1959). The geometric interpretation of agreement. Am. Sociol. Rev. 24(3), 338-345.
    

17. Smith & Rose (1995). Model goodness-of-fit analysis using regression and related techniques. Ecol. Model. 77, 49–64.
    

18. Warton et al. (2006). Bivariate line-fitting methods for allometry. Biol. Rev. Camb. Philos. Soc. 81, 259–291.
    

19. Willmott (1981). On the validation of models. Phys. Geogr. 2, 184–194.
    

20. Willmott et al. (1985). Statistics for the evaluation and comparison of models. J. Geophys. Res. 90, 8995.
    

21. Willmott & Matsuura (2005). Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Clim. Res. 30, 79–82.
    

22. Willmott et al. (2012). A refined index of model performance. Int. J. Climatol. 32, 2088–2094.
    

23. Yang et al. (2014). An evaluation of the statistical methods for testing the performance of crop models with observed data. Agric. Syst. 127, 81-89.
    

24. Szekely, G.J., Rizzo, M.L., and Bakirov, N.K. (2007). Measuring and testing dependence by correlation of distances. Annals of Statistics, Vol. 35(6): 2769-2794.
    

25. Reshef, D., Reshef, Y., Finucane, H., Grossman, S., McVean, G., Turnbaugh, P., Lander, R., Mitzenmacher, M., and Sabeti, P. (2011). Detecting novel associations in large datasets. Science 334, 6062.

26. Hyndman, R.J., Koehler, A.B. (2006). Another look at measures of forecast accuracy. Int. J. Forecast