What is a “small” dataset?

An obvious question is, “How small is ‘small’?” This is a very complex question and it is way beyond the scope of this vignette to try to answer it rigorously. But we can simply say that the key issue at stake is that applying the training-test split that is common in machine learning is a crucial technique for increasing the generalizability of data analysis. So, the question becomes focused to, “How small is too small for a training-test split for machine learning analysis?” The rule of thumb I am familiar with is that machine learning requires at least 200 rows of data for each predictor variable. So, for example, if you have five input variables, you would need at least 1000 rows of data. But note that this does not refer to the size of the entire dataset but to the minimum size of the training subset. So, if you carry out an 80-20 split on the full dataset (that is, 80% training set), then you would need at least 1000 rows for the training set and another 250 rows for the test set, for a minimum of 1250 rows. (And if you carry out hyperparameter tuning with cross validation on that training set, then you need even more data.) If you see where this is headed, you might quickly realize that most datasets of less than 2000 rows are probably “small”. You can see that even many datasets that are more than 2000 rows are nonetheless “small”, and so probably need the techniques mentioned in this vignette.

We begin by loading the necessary libraries.

library(ale)

attitude dataset

Most analyses use the attitude dataset, built-in with R: “From a survey of the clerical employees of a large financial organization, the data are aggregated from the questionnaires of the approximately 35 employees for each of 30 (randomly selected) departments.” Since we’re talking about “small” datasets, we figure that we might as well demonstrate principles with extremely small examples.

str(attitude)
#> 'data.frame':    30 obs. of  7 variables:
#>  $ rating    : num  43 63 71 61 81 43 58 71 72 67 ...
#>  $ complaints: num  51 64 70 63 78 55 67 75 82 61 ...
#>  $ privileges: num  30 51 68 45 56 49 42 50 72 45 ...
#>  $ learning  : num  39 54 69 47 66 44 56 55 67 47 ...
#>  $ raises    : num  61 63 76 54 71 54 66 70 71 62 ...
#>  $ critical  : num  92 73 86 84 83 49 68 66 83 80 ...
#>  $ advance   : num  45 47 48 35 47 34 35 41 31 41 ...

summary(attitude)
#>      rating        complaints     privileges       learning         raises     
#>  Min.   :40.00   Min.   :37.0   Min.   :30.00   Min.   :34.00   Min.   :43.00  
#>  1st Qu.:58.75   1st Qu.:58.5   1st Qu.:45.00   1st Qu.:47.00   1st Qu.:58.25  
#>  Median :65.50   Median :65.0   Median :51.50   Median :56.50   Median :63.50  
#>  Mean   :64.63   Mean   :66.6   Mean   :53.13   Mean   :56.37   Mean   :64.63  
#>  3rd Qu.:71.75   3rd Qu.:77.0   3rd Qu.:62.50   3rd Qu.:66.75   3rd Qu.:71.00  
#>  Max.   :85.00   Max.   :90.0   Max.   :83.00   Max.   :75.00   Max.   :88.00  
#>     critical        advance     
#>  Min.   :49.00   Min.   :25.00  
#>  1st Qu.:69.25   1st Qu.:35.00  
#>  Median :77.50   Median :41.00  
#>  Mean   :74.77   Mean   :42.93  
#>  3rd Qu.:80.00   3rd Qu.:47.75  
#>  Max.   :92.00   Max.   :72.00

ALE for ordinary least squares regression (multiple linear regression)

Ordinary least squares (OLS) regression is the most generic multivariate statistical technique. Thus, we use it as a baseline illustration to help motivate the value of ALE for interpreting the analysis of small data samples.

We train an OLS model to predict average rating:

lm_attitude <- lm(rating ~ ., data = attitude)

summary(lm_attitude)
#> 
#> Call:
#> lm(formula = rating ~ ., data = attitude)
#> 
#> Residuals:
#>      Min       1Q   Median       3Q      Max 
#> -10.9418  -4.3555   0.3158   5.5425  11.5990 
#> 
#> Coefficients:
#>             Estimate Std. Error t value Pr(>|t|)    
#> (Intercept) 10.78708   11.58926   0.931 0.361634    
#> complaints   0.61319    0.16098   3.809 0.000903 ***
#> privileges  -0.07305    0.13572  -0.538 0.595594    
#> learning     0.32033    0.16852   1.901 0.069925 .  
#> raises       0.08173    0.22148   0.369 0.715480    
#> critical     0.03838    0.14700   0.261 0.796334    
#> advance     -0.21706    0.17821  -1.218 0.235577    
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Residual standard error: 7.068 on 23 degrees of freedom
#> Multiple R-squared:  0.7326, Adjusted R-squared:  0.6628 
#> F-statistic:  10.5 on 6 and 23 DF,  p-value: 1.24e-05

At the very least, the ale is useful for visualizing the effects of model variables.

Before starting, we recommend that you enable progress bars to see how long procedures will take. Simply run the following code at the beginning of your R session:

# Run this in an R console; it will not work directly within an R Markdown or Quarto block
progressr::handlers(global = TRUE)
progressr::handlers('cli')

If you forget to do that, the {ale} package will do it automatically for you with a notification message.

Note that for now, we run ale with no bootstrapping (the default) because small samples require a special bootstrap approach, as explained below. For now, all we are doing is using ALE to accurately visualize what the model estimates.

ale_lm_attitude_simple <- ale(
  attitude, lm_attitude,
  parallel = 2  # CRAN limit (delete this line on your own computer)
)

# Print all plots
ale_lm_attitude_simple$plots |> 
  patchwork::wrap_plots(ncol = 2)

This visualization confirms what we see in the model coefficients above: complaints have a strong positive effect on ratings and learning has a more moderate effect. However, the ALE indicates a stronger effect of advance than the regression coefficients suggest. The other variables have relatively little effect on ratings. We will see shortly that proper bootstrapping of the model can shed some light on the discrepancies.

What is unique about ALE compared to other approaches is that it visualizes the effect of each variable irrespective of interactions that might or might not exist with other variables, whether these other interacting variables are included in the model or not.

We can use ale_ixn() to visualize the possible existence of any interactions:

ale_lm_attitude_ixn <- ale_ixn(
  attitude, lm_attitude,
  parallel = 2  # CRAN limit (delete this line on your own computer)
)

# Print plots
ale_lm_attitude_ixn$plots |>
  # extract list of x1 ALE outputs
  purrr::walk(\(.x1) {  
    # plot all x2 plots in each .x1 element
    patchwork::wrap_plots(.x1, ncol = 2) |> 
      print()
  })

This is a powerful use-case for the ale package: it can be used to explore the existence of interactions after the fact; they do not need to be hypothesized beforehand. However, without bootstrapping, such findings cannot be considered reliable. In this case, there are no interactions in this dataset, so we will not continue exploring them.

Full model bootstrapping

We have referred frequently to the importance of bootstrapping. None of our model results, with or without ALE, should be considered reliable without being bootstrapped. For large datasets whose models have been properly trained and evaluated on separate subsets before ALE analysis, ale() bootstraps the ALE results of the final deployment model on the full dataset. However, when a dataset is too small to be subdivided into training and test sets, then the entire model should be bootstrapped, not just the ALE data from a single deployment model. That is, multiple models should be trained, one on each bootstrap sample. The reliable results are the average results of all the bootstrap models, however many there are.

The model_bootstrap() function automatically carries out full-model bootstrapping suitable for small datasets. Specifically, it:

• Creates multiple bootstrap samples (default 100; the user can specify any number);
• Creates a model on each bootstrap sample;
• Calculates model overall statistics, variable coefficients, and ALE values for each model on each bootstrap sample;
• Calculates the mean, median, and lower and upper confidence intervals for each of those values across all bootstrap samples.

model_bootstrap() has two required arguments. Consistent with tidyverse conventions, its first argument is a dataset, data. The second argument is the model object to be analyzed. For objects that follow standard R modelling conventions, model_bootstrap() should be able to automatically recognize and parse the model object. So, here is the call to model_bootstrap():

mb_lm <- model_bootstrap(
  attitude, 
  lm_attitude,
  boot_it = 10,  # 100 by default but reduced here for a faster demonstration
  parallel = 2  # CRAN limit (delete this line on your own computer)
)

By default, model_bootstrap() creates 100 bootstrap samples of the provided dataset and creates 100 + 1 models on the data (one for each bootstrap sample and then once for the original dataset). (However, so that this illustration runs faster, we demonstrate it here with only 10 iterations.) Beyond ALE data, it also provides bootstrapped overall model statistics (provided through broom::glance()) and bootstrapped model coefficients (provided through broom::tidy()). Any of the default options for broom::glance(), broom::tidy(), and ale() can be customized, along with defaults for model_bootstrap(), such as the number of bootstrap iterations. You can consult the help file for these details with help(model_bootstrap).

model_bootstrap() returns a list with the following elements (depending on values requested in the output argument:

• model_stats: bootstrapped results from broom::glance()
• model_coefs: bootstrapped results from broom::tidy()
• ale_data: bootstrapped ALE data and plots
• boot_data: full bootstrap data (not returned by default)

Here are the bootstrapped overall model statistics:

mb_lm$model_stats
#> # A tibble: 8 × 7
#>   name            estimate     conf.low       mean      median conf.high      sd
#>   <chr>              <dbl>        <dbl>      <dbl>       <dbl>     <dbl>   <dbl>
#> 1 r.squared      0.767      0.665        0.767      0.775       0.864    6.23e-2
#> 2 adj.r.squared  0.707      0.577        0.707      0.716       0.828    7.86e-2
#> 3 sigma          6.57       5.33         6.57       6.77        7.95     8.60e-1
#> 4 statistic     13.9        7.62        13.9       13.2        25.6      5.80e+0
#> 5 p.value        0.0000257  0.000000104  0.0000257  0.00000200  0.000150 5.59e-5
#> 6 df             6          6            6          6           6        0      
#> 7 df.residual   23         23           23         23          23        0      
#> 8 nobs          30         30           30         30          30        0

Here are the bootstrapped model coefficients:

mb_lm$model_coefs
#> # A tibble: 7 × 7
#>   term        estimate conf.low    mean  median conf.high std.error
#>   <chr>          <dbl>    <dbl>   <dbl>   <dbl>     <dbl>     <dbl>
#> 1 (Intercept)   3.05   -9.92     3.05    3.64      13.5       8.29 
#> 2 complaints    0.613   0.323    0.613   0.589      0.871     0.175
#> 3 privileges   -0.0588 -0.292   -0.0588 -0.114      0.353     0.204
#> 4 learning      0.334   0.0358   0.334   0.320      0.605     0.174
#> 5 raises        0.0604 -0.267    0.0604  0.0599     0.337     0.206
#> 6 critical      0.159  -0.00270  0.159   0.158      0.298     0.110
#> 7 advance      -0.260  -0.756   -0.260  -0.152      0.104     0.294

Here we can visualize the results of the ALE plots.

mb_lm$ale$plots |> 
  patchwork::wrap_plots(ncol = 2)

The key to interpreting the effects in these models is contrasting the grey bootstrapped confidence bands surrounding the average (median) ALE effect with the thin horizontal grey band labelled ‘median \(\pm\) 2.5%’. Anything within \(\pm\) 2.5% of the median is in the 5% middle of the data. Only bootstrapped effects that are clearly beyond this middle band may be considered significant. By this criteria, considering that the median rating was 65.5%, we can conclude that:

• Complaints that were handled at below around 68% led to below-average overall ratings; complaints that were handled above around 72% are associated with above-average overall ratings.
• The 95% bootstrapped confidence intervals of every other variable fully overlap the entire 5% median band. Thus, despite the general trends of some of the data (in particular learning’s positive trend and advance’s negative trend), the data does not support claims that any other factor had a convincingly meaningful effect on ratings.

Although this is a basic demonstration, it readily shows how crucial proper bootstrapping is to make meaningful inferences from data analysis.

ALE for general additive models (GAM)

A major limitation of OLS regression is that it models all relationships between the x variables and y as straight lines. But it is unlikely that all relationships are truly linear. OLS cannot accurately capture non-linear relationships.

Because the samples here are relatively small, we will use general additive models (GAM) for the modelling. To grossly oversimplify things, GAM is an extension of statistical regression analysis that lets the model fit flexible patterns in the data instead of being restricted to the best-fitting straight line. It is an ideal approach for samples that are too small for machine learning because it provides flexible curves unlike ordinary least squares regression yet will not overfit excessively as would most machine learning techniques when working with such small samples.

With GAM, the variables that we want to become flexible need to be wrapped in the s (smooth) function, e.g., s(complaints). For this example, we will smooth all our numerical input variables:

gam_attitude <- mgcv::gam(rating ~ complaints + privileges + s(learning) +
                            raises + s(critical) + advance,
                          data = attitude)
summary(gam_attitude)
#> 
#> Family: gaussian 
#> Link function: identity 
#> 
#> Formula:
#> rating ~ complaints + privileges + s(learning) + raises + s(critical) + 
#>     advance
#> 
#> Parametric coefficients:
#>             Estimate Std. Error t value Pr(>|t|)    
#> (Intercept) 36.97245   11.60967   3.185 0.004501 ** 
#> complaints   0.60933    0.13297   4.582 0.000165 ***
#> privileges  -0.12662    0.11432  -1.108 0.280715    
#> raises       0.06222    0.18900   0.329 0.745314    
#> advance     -0.23790    0.14807  -1.607 0.123198    
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Approximate significance of smooth terms:
#>               edf Ref.df     F p-value  
#> s(learning) 1.923  2.369 3.761  0.0312 *
#> s(critical) 2.296  2.862 3.272  0.0565 .
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> R-sq.(adj) =  0.776   Deviance explained = 83.9%
#> GCV = 47.947  Scale est. = 33.213    n = 30

By comparing the adjusted R² of the OLS model (0.663) with that of the GAM model (0.776), we can readily see that the GAM model provides a superior fit to the data.

To understand which variables were responsible for this relationship, the results for the smooth terms in GAM are not readily interpretable. They need to be visualized for effective interpretation—ALE is perfect for such purposes.

ale_gam_attitude_simple <- ale(
  attitude, gam_attitude,
  parallel = 2  # CRAN limit (delete this line on your own computer)
)

ale_gam_attitude_simple$plots |> 
  patchwork::wrap_plots(ncol = 2)

Compared to the OLS results above, the GAM results provide quite a surprise concerning the shape of the effect of employees’ perceptions that their department is too critical–it seems that both low criticism and very high criticism negatively affect ratings. However, before trying to interpret these results, we must remember that results that are not bootstrapped are simply not reliable. So, let us see what bootstrapping will give us.

mb_gam <- model_bootstrap(
  attitude, 
  gam_attitude, 
  boot_it = 10,  # 100 by default but reduced here for a faster demonstration
  parallel = 2  # CRAN limit (delete this line on your own computer)
  )
mb_gam$model_stats
#> # A tibble: 5 × 7
#>   name          estimate conf.low   mean median conf.high     sd
#>   <chr>            <dbl>    <dbl>  <dbl>  <dbl>     <dbl>  <dbl>
#> 1 df              16.0     10.6   16.0    16.5       18.8 2.77  
#> 2 df.residual     14.0     11.2   14.0    13.5       19.4 2.77  
#> 3 nobs            30       30     30      30         30   0     
#> 4 adj.r.squared    0.964    0.846  0.964   1.00       1   0.0619
#> 5 npar            23       23     23      23         23   0

mb_gam$model_coefs
#> # A tibble: 2 × 7
#>   term        estimate conf.low  mean median conf.high std.error
#>   <chr>          <dbl>    <dbl> <dbl>  <dbl>     <dbl>     <dbl>
#> 1 s(learning)     6.96     2.05  6.96   8.23      8.93      2.67
#> 2 s(critical)     4.02     1.29  4.02   3.81      7.22      1.95

mb_gam$ale$plots |> 
  patchwork::wrap_plots(ncol = 2)

The bootstrapped GAM results tell a rather different story from the OLS results. In this case, the bootstrap confidence bands of all the variables (even of complaints) fully overlap with the entirety of the median non-significance region. Even the average slopes have vanished from all variables except for complaint, where it remains positive, yet insignificant because of the wide confidence interval.

So, what should we conclude? First, it is tempting to retain the OLS results because they tell a more interesting story. But we consider that this would be irresponsible since the GAM model is clearly superior in terms of adjusted R²: it is the model that far more reliably tells us what is really going on. And what does it tell us?

• There seems to be a positive effect of handled complaints on ratings (the higher the percentage of complaints that are handled, the higher the average rating), but the data does not allow us to be sufficiently certain to generalize these results.
• There is insufficient evidence that any of the other variables have any effect at all.

No doubt, the inconclusive results are because the dataset is so small (only 30 rows). A dataset even double that size might show significant effects at least for complaints, if not for other variables.

model_call_string argument for non-standard models

model_bootstrap() accesses the model object and internally modifies it to retrain the model on bootstrapped datasets. It should be able to automatically manipulate most R model objects that are used for statistical analysis. However, if an object does not follow standard conventions for R model objects, model_bootstrap() might not be able to manipulate it. If so, the function will fail early with an appropriate error message. In that case, the user must specify the model_call_string argument with a character string of the full call for the model with boot_data as the data argument for the call. (boot_data is a placeholder for the bootstrap datasets that model_bootstrap() will internally work with.)

To show how this works, let’s pretend that the mgcv::gam object needs such special treatment. To construct, the model_call_string, we must first execute the model and make sure that it works. We did that earlier but we repeat it here for this demonstration

gam_attitude_again <- mgcv::gam(rating ~ complaints + privileges + s(learning) +
                                  raises + s(critical) + advance,
                                data = attitude)
summary(gam_attitude_again)
#> 
#> Family: gaussian 
#> Link function: identity 
#> 
#> Formula:
#> rating ~ complaints + privileges + s(learning) + raises + s(critical) + 
#>     advance
#> 
#> Parametric coefficients:
#>             Estimate Std. Error t value Pr(>|t|)    
#> (Intercept) 36.97245   11.60967   3.185 0.004501 ** 
#> complaints   0.60933    0.13297   4.582 0.000165 ***
#> privileges  -0.12662    0.11432  -1.108 0.280715    
#> raises       0.06222    0.18900   0.329 0.745314    
#> advance     -0.23790    0.14807  -1.607 0.123198    
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Approximate significance of smooth terms:
#>               edf Ref.df     F p-value  
#> s(learning) 1.923  2.369 3.761  0.0312 *
#> s(critical) 2.296  2.862 3.272  0.0565 .
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> R-sq.(adj) =  0.776   Deviance explained = 83.9%
#> GCV = 47.947  Scale est. = 33.213    n = 30

Once we’re sure that the model call works, then the model_call_string is constructed with three simple steps:

1. Wrap the entire call (everything to the right of the assignment operator <-) in quotes.
2. Replace the dataset in the data argument with boot_data.
3. Pass the quoted string to model_bootstrap() as the model_call_string argument (the argument must be explicitly named).

So, here is the form of the call to model_bootstrap() for a non-standard model object type:

mb_gam_non_standard <- model_bootstrap(
  attitude,
  gam_attitude_again,
  model_call_string = 'mgcv::gam(rating ~ complaints + privileges + s(learning) +
                                  raises + s(critical) + advance,
                                data = boot_data)', 
  boot_it = 10,  # 100 by default but reduced here for a faster demonstration
  parallel = 2  # CRAN limit (delete this line on your own computer)
  )
mb_gam_non_standard$model_stats
#> # A tibble: 5 × 7
#>   name          estimate conf.low   mean median conf.high     sd
#>   <chr>            <dbl>    <dbl>  <dbl>  <dbl>     <dbl>  <dbl>
#> 1 df              16.0     10.6   16.0    16.5       18.8 2.77  
#> 2 df.residual     14.0     11.2   14.0    13.5       19.4 2.77  
#> 3 nobs            30       30     30      30         30   0     
#> 4 adj.r.squared    0.964    0.846  0.964   1.00       1   0.0619
#> 5 npar            23       23     23      23         23   0

Everything else works as usual.