staccuracy

Standardized accuracy (staccuracy) is a framework for expressing accuracy scores such that 50% represents a reference level of performance and 100% is a perfect prediction. The {staccuracy} package provides tools for creating staccuracy functions as well as some recommended staccuracy measures. It also provides functions for some classic performance metrics such as mean absolute error (MAE), root mean squared error (RMSE), and area under the receiver operating characteristic curve (AUCROC), as well as their winsorized versions when applicable.

Installation

You can install the official CRAN version of {staccuracy}:

install.packages('staccuracy')

The development version of {staccuracy} is thoroughly tested, but it might not be thoroughly documented. You can install it like so:

# install.packages("pak")
pak::pak("tripartio/staccuracy")

Example

The basic challenge that {staccuracy} addresses is not only to measure the accuracy of model predictions but to intuitively indicate how relevant the accuracy scores are:

library(staccuracy)
#> 
#> Attaching package: 'staccuracy'
#> The following object is masked from 'package:stats':
#> 
#>     mad

# Here's some data
actual_1 <- c(2.3, 4.5, 1.8, 7.6, 3.2)

# Here are some predictions of that data
predicted_1 <- c(2.5, 4.2, 1.9, 7.4, 3.0)

# Mean Absolute Error (MAE) measures the average error in the predictions
mae(actual_1, predicted_1)
#> [1] 0.2

# But how good is that? 
# Mean Absolute Deviation (MAD) gives the natural variation in the actual data around the mean; this is a point of comparison for the MAE.
mad(actual_1)
#> [1] 1.736

# So, our predictions are better (lower) than the MAD, but how good, really?
# Create a standardized accuracy function to give us an easily interpretable metric:
my_mae_vs_mad_sa <- staccuracy(mae, mad)
my_mae_vs_mad_sa(actual_1, predicted_1)
#> [1] 0.9423963

# That's 94.2% standardized accuracy compared to the MAD. Pretty good!

# This and other convenient standardized accuracy scores are already built in
sa_mae_mad(actual_1, predicted_1)  # staccuracy of MAE on MAD
#> [1] 0.9423963
sa_rmse_sd(actual_1, predicted_1)  # staccuracy of RMSE on SD
#> [1] 0.95477