Performance Metrics
This document summarizes and compares some metrics used in model performance evaluation. These metrics are:
- Mean Absolute Error (MAE)
- Mean Absolute Percentage Error (MAPE)
- Weighted Mean Absolute Percentage Error (WMAPE)
The Mean Absolute Error is calculated as the average of the absolute differences between the predicted and actual values for each observation.
$$ MAE = \sum_{i=1}^{N} |T_i - G_i| $$
The Mean Absolute Percentage Error is calculated as the average of the absolute percentage differences between the predicted and actual values for each observation.
$$ MAPE = \sum_{i=1}^{N} \frac{|T_i - G_i|}{G_i} $$
The Weighted Mean Absolute Percentage Error is calculated as the ratio of the sum of the absolute differences between the predicted and actual values to the sum of the actual values.
$$ WMAPE = \frac{\sum_{i=1}^{N} |T_i - G_i|}{\sum_{i=1}^{N} G_i} $$
N = Number of observations, \( T_i \) = Predicted value for the i-th observation, \( G_i \) = Actual value for the i-th observation
The calculation of these metrics is also demonstrated numerically on the example data below.
| Observation | Predicted | Actual | Absolute Error | Absolute Percentage Error |
|---|---|---|---|---|
| 1 | 55.000 | 57.000 | 2.000 | %3.51 |
| 2 | 47.000 | 45.000 | 2.000 | %4.44 |
| 3 | 60.000 | 55.000 | 5.000 | %9.09 |
| 4 | 10.000 | 11.000 | 1.000 | %9.09 |
| 5 | 22.000 | 21.000 | 1.000 | %4.76 |
| 6 | 4.000 | 0.050 | 3.950 | %7900 |
| 7 | 50.000 | 56.000 | 6.000 | %10.71 |
| Total | 248.000 | 245.050 | 20.950 | %7941.61 |
| Average | 35.429 | 35.007 | 2.993 | %1134.52 |
For the given example:
- MAE: Mean Absolute Error = 2.993
- MAPE: Mean Absolute Percentage Error = %1134.52
- WMAPE: Weighted Mean Absolute Percentage Error = \( \frac{20.950}{245.050} \) = %8.54
These are calculated as follows. Comparing these metrics:
- First, MAE provides a result in the unit of measurement of the predicted and actual values rather than a percentage. Especially in wind power generation forecasts, where production capacities vary between plants, MAE does not produce an easily interpretable result. For example, in wind forecasting, the MAE unit will be in MWh.
- MAPE, despite being a percentage expression, produces unusually high percentage errors when the actual value is low, as seen in the example (observation 6). In cases like wind power forecasting, where observation scales constantly change and actual values frequently approach zero, MAPE does not produce reliable and comparable results.
- To enhance interpretability, WMAPE adds the sum of actual values to the denominator in addition to the MAE metric, converting the result into a percentage expression. This increases both its reliability and interpretability.
In light of these observations, WMAPE stands out as both an interpretable and reliable metric.
