Tutorial to the fairness R package

Installation

Install the latest stable package version from CRAN:

install.packages('fairness')
library(fairness)

…or get the most recent development version from Github:

library(devtools)
devtools::install_github('kozodoi/fairness')
library(fairness)

Data description

This package includes two datasets to study algorithmic fairness: compas and germancredit. In this tutorial, you will be able to use a simplified version of the landmark COMPAS dataset. You can read more about the dataset here. To load the dataset, all you need to do is:

data('compas')

The compas dataframe contains nine columns: The outcome is Two_yr_Recidivism, i.e. whether an individual will commit a crime in two years or not. Variables exist in the data about prior criminal record (Number_of_Priors and Misdemeanor) and basic features such as age, categorized (Age_Above_FourtyFive and Age_Below_TwentyFive), sex (Female) and ethnicity (ethnicity). You don’t really need to delve into the data much, we have already ran a prediction model using all variables to predict Two_yr_Recidivism and concatenated the predicted probabilities (probability) and predicted classes (predicted) to the data. You will be able to use the probability and predicted columns directly in your analysis.

Please feel free to set up other prediction models (e.g. excluding sensitive group information, such as sex and ethnicity) and use your generated predicted probabilities or classes to assess group fairness.

An outlook on the confusion matrix

Most fairness metrics are calculated based on a confusion matrix produced by a classification model. The confusion matrix is comprised of four classes:

• True positives (TP): the true class is positive and the prediction is positive (correct classification)
• False positives (FP): the true class is negative and the prediction is positive (incorrect classification)
• True negatives (TN): the true class is negative and the prediction is negative (correct classification)
• False negatives (FN): the true class is positive and the prediction is negative (incorrect classification)

Fairness metrics are calculated by comparing one or more of these measures across sensitive subgroups (e.g., male and female). For a detailed overview of measures coming from the confusion matrix and precise definitions, click here or here.

Fairness metrics functions

The package implements 11 fairness metrics. Many of these are mutually exclusive: results for a given classification problem often cannot be fair in terms of all metrics. Depending on a context, it is important to select an appropriate metric to evaluate fairness.

Below, we describe functions used to compute the implemented metrics. Every function has a similar set of arguments:

• data: data.frame containing the input data and model predictions
• group: column name indicating the sensitive group (factor variable)
• base: base level of the sensitive group for fairness metrics calculation
• outcome: column name indicating the binary outcome variable
• outcome_base: base level of the outcome variable (i.e., negative class) for fairness metrics calculation

We also need to supply model predictions. Depending on the metric, we need to provide either probabilistic predictions as probs or class predictions as preds. The model predictions can be appended to the original data.frame or provided as a vector. In this tutorial, we will use probabilistic predictions with all functions. When working with probabilistic predictions, some metrics require a cutoff value to convert probabilities into class predictions supplied as cutoff.

Before looking at different metrics, we will create a binary numeric version of the outcome variable that we will supply as outcome in fairness metrics functions:

compas$Two_yr_Recidivism_01 <- ifelse(compas$Two_yr_Recidivism == 'yes', 1, 0) 

dem_parity(data    = compas, 
           outcome = 'Two_yr_Recidivism_01',
           group   = 'ethnicity',
           probs   = 'probability', 
           cutoff  = 0.5, 
           base    = 'Caucasian')

prop_parity(data    = compas, 
            outcome = 'Two_yr_Recidivism_01',
            group   = 'ethnicity',
            probs   = 'probability', 
            cutoff  = 0.5, 
            base    = 'Caucasian')

equal_odds(data    = compas, 
           outcome = 'Two_yr_Recidivism_01', 
           group   = 'ethnicity',
           probs   = 'probability', 
           cutoff  = 0.5, 
           base    = 'African_American')

pred_rate_parity(data    = compas, 
                 outcome = 'Two_yr_Recidivism_01', 
                 group   = 'ethnicity',
                 probs   = 'probability', 
                 cutoff  = 0.5, 
                 base    = 'African_American')

acc_parity(data    = compas, 
           outcome = 'Two_yr_Recidivism_01', 
           group   = 'ethnicity',
           probs   = 'probability', 
           preds   = NULL,
           cutoff  = 0.5, 
           base    = 'African_American')

fnr_parity(data    = compas, 
           outcome = 'Two_yr_Recidivism_01', 
           group   = 'ethnicity',
           probs   = 'probability', 
           cutoff  = 0.5, 
           base    = 'African_American')

fpr_parity(data    = compas, 
           outcome = 'Two_yr_Recidivism_01', 
           group   = 'ethnicity',
           probs   = 'probability', 
           cutoff  = 0.5, 
           base    = 'African_American')

npv_parity(data    = compas, 
           outcome = 'Two_yr_Recidivism_01', 
           group   = 'ethnicity',
           probs   = 'probability', 
           cutoff  = 0.5, 
           base    = 'African_American')

spec_parity(data    = compas, 
            outcome = 'Two_yr_Recidivism_01', 
            group   = 'ethnicity',
            probs   = 'probability', 
            cutoff  = 0.5, 
            base    = 'African_American')

roc_parity(data    = compas, 
           outcome = 'Two_yr_Recidivism_01', 
           group   = 'ethnicity',
           probs   = 'probability', 
           base    = 'African_American')

mcc_parity(data    = compas, 
           outcome = 'Two_yr_Recidivism_01', 
           group   = 'ethnicity',
           probs   = 'probability', 
           cutoff  = 0.5, 
           base    = 'African_American')

Output and visualizations

All functions output results and matching barcharts that provide visual cues about the parity metrics for the defined sensitive subgroups. For instance, let’s look at predictive rate parity with ethnicity being set as the sensitive group and considering Caucasians as the ‘base’ group:

output$Metric
#>                          Caucasian African_American      Asian   Hispanic
#> Precision                 0.577381        0.6652475  0.5000000   0.590604
#> Predictive Rate Parity    1.000000        1.1521812  0.8659794   1.022902
#> Group size             2103.000000     3175.0000000 31.0000000 509.000000
#>                        Native_American       Other
#> Precision                     0.600000   0.6117647
#> Predictive Rate Parity        1.039175   1.0595512
#> Group size                   11.000000 343.0000000

In the upper row, the raw precision values are shown for all ethnicities, and in the row below, the relative precisions compared to Caucasians (1) are shown. Note that in case an other ethnic group is set as the base group (e.g. Hispanic), the raw precision values do not change, only the relative metrics:

output$Metric
#>                          Hispanic    Caucasian African_American      Asian
#> Precision                0.590604    0.5773810        0.6652475  0.5000000
#> Predictive Rate Parity   1.000000    0.9776109        1.1263849  0.8465909
#> Group size             509.000000 2103.0000000     3175.0000000 31.0000000
#>                        Native_American       Other
#> Precision                     0.600000   0.6117647
#> Predictive Rate Parity        1.015909   1.0358289
#> Group size                   11.000000 343.0000000

A standard output is a barchart that shows the relative metrics for all subgroups. For the previous case (when Hispanic is defined as the base group), this plot would look like this:

output$Metric_plot

When probabilities are defined, an extra density plot will be output with the distributions of probabilities of all subgroups and the user-defined cutoff:

output$Probability_plot

Another example would be comparing males vs. females in terms of recidivism prediction and defining a 0.4 cutoff:

output$Probability_plot

The function related to ROC AUC comparisons will output ROC curves for each subgroups. Let’s look at the plot, also comparing males vs. females:

output$ROCAUC_plot

Closing words

You have read through the fairness R package tutorial and by now, you have a solid grip on algorithmic group fairness metrics. If something is not clear, check out this blogpost with a more detailed tutorial. We hope that you will be able to use this R package in your data analysis! Please let us know if you have any issues here - fairness GitHub - or contact the authors if you have any feedback!