The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.

Custom models and performance metrics

The CCI test only needs two things: a way to train a model on part of the data, and a way to measure how well it predicts the rest. The package has four built-in learners (rf, xgboost, svm, KNN) and three built-in metrics (RMSE, Kappa, LogLoss), but both can be replaced:

In both cases you must tell CCI which direction is “better” with the tail argument:

We use the same kind of data as in vignette("Testing-CI-with-CCI", package = "CCI"), where \(Y \perp\!\!\!\perp X \mid Z_1, Z_2\) is true and \(Y \perp\!\!\!\perp X \mid Z_1\) is false.

Custom performance metrics with `metricfunc`

A metric function takes the observed values of the test data and the model’s predictions, and returns a single number:

my_metric <- function(actual, predictions) {
  # compute and return one number
}

It may also have a ... argument, in which case additional arguments given to CCI.test() are passed on to it. What actual and predictions contain depends on the learner and on the type of \(Y\):

`method`	Numeric \(Y\) (regression)	Categorical \(Y\) (classification)
`"rf"`, `"svm"`, `"KNN"`	numeric, numeric predictions	factor, predicted classes (factor)
`"xgboost"`	numeric, numeric predictions	factor, class probabilities: the probability of the second class for two classes, an \(n \times K\) matrix with the classes as column names for more

Regression: \(R^2\)

\(R^2\) is higher for better predictions, so tail = "right":

r_squared <- function(actual, predictions) {
  1 - sum((actual - predictions)^2) / sum((actual - mean(actual))^2)
}
res_r2 <- CCI.test(Y ~ X | Z1, data = dat, metricfunc = r_squared, tail = "right",
                   seed = 1, progress = FALSE)
summary(res_r2)
#> 
#> Computational Conditional Independence Test
#> --------------------------------------------
#> Method:    CCI test using rf 
#> Formula:   Y ~ X | Z1 
#> Permutations:  160 
#> Metric:    r_squared 
#> Tail:      right 
#> Statistic: 0.3268 
#> P-value:   0.006211 
#> 
#> MC sample:   1

The summary shows the name of the metric function. The mean absolute error (MAE) is lower for better predictions, so tail = "left". Here with the KNN learner:

mae <- function(actual, predictions) mean(abs(actual - predictions))
summary(CCI.test(Y ~ X | Z1 + Z2, data = dat, method = "KNN", metricfunc = mae, tail = "left",
                 seed = 1, progress = FALSE))
#> 
#> Computational Conditional Independence Test
#> --------------------------------------------
#> Method:    CCI test using KNN 
#> Formula:   Y ~ X | Z1 + Z2 
#> Permutations:  160 
#> Metric:    mae 
#> Tail:      left 
#> Statistic: 0.9363 
#> P-value:   0.6646 
#> 
#> MC sample:   1

Classification: balanced accuracy and Brier score

With a categorical \(Y\), the built-in learners except xgboost give the predicted classes. Balanced accuracy, the average share of correct predictions within each class, is robust to unequal class sizes:

set.seed(2)
cat_data <- normal_data(500)
cat_data$Y <- factor(ifelse(cat_data$Y > 1, "high", "low"))   # unequal class sizes
table(cat_data$Y)
#> 
#> high  low 
#>  164  336

balanced_accuracy <- function(actual, predictions) {
  mean(tapply(as.character(predictions) == as.character(actual), actual, mean))
}
summary(CCI.test(Y ~ X | Z1, data = cat_data, metricfunc = balanced_accuracy, tail = "right",
                 seed = 1, progress = FALSE))
#> 
#> Computational Conditional Independence Test
#> --------------------------------------------
#> Method:    CCI test using rf 
#> Formula:   Y ~ X | Z1 
#> Permutations:  160 
#> Metric:    balanced_accuracy 
#> Tail:      right 
#> Statistic: 0.7319 
#> P-value:   0.006211 
#> 
#> MC sample:   1

xgboost gives class probabilities instead. For two classes, predictions is the probability of the second class level ("low" here), which allows metrics like the Brier score (lower is better):

brier <- function(actual, predictions) {
  mean((as.numeric(actual == levels(actual)[2]) - predictions)^2)
}
summary(CCI.test(Y ~ X | Z1, data = cat_data, method = "xgboost", nrounds = 100, eta = 0.1,
                 metricfunc = brier, tail = "left", seed = 1, progress = FALSE))
#> 
#> Computational Conditional Independence Test
#> --------------------------------------------
#> Method:    CCI test using xgboost 
#> Formula:   Y ~ X | Z1 
#> Permutations:  160 
#> Metric:    brier 
#> Tail:      left 
#> Statistic: 0.1942 
#> P-value:   0.03106 
#> 
#> MC sample:   1

Custom learners with `mlfunc`

An mlfunc function trains a model on the training rows, predicts the test rows, and returns the performance as a single number. It must have these arguments:

formula: a regression formula Y ~ X + Z1 + ... (the | is replaced by +);
data: the data, with \(X\) permuted when the null distribution is built;
train_indices and test_indices: the rows to train on and to evaluate on;
...: additional arguments given to CCI.test(), e.g. tuning parameters for the model.

The general structure is:

my_wrapper <- function(formula, data, train_indices, test_indices, ...) {
  model <- train_model(formula, data = data[train_indices, ], ...)
  predictions <- predict(model, data[test_indices, ])
  actual <- data[test_indices, all.vars(formula)[1]]
  compute_metric(actual, predictions)
}

The formula and data include the polynomial and interaction terms that CCI.test() adds to \(Z\) (see poly and interaction). Set poly = FALSE and interaction = FALSE if your model should only see the original variables.

A linear model

With a linear regression as learner, the test compares how well a linear model predicts \(Y\) with and without the real \(X\). With the polynomial and interaction terms of \(Z\), this is a flexible and very fast test:

lm_wrapper <- function(formula, data, train_indices, test_indices, ...) {
  model <- lm(formula, data = data[train_indices, ])
  predictions <- predict(model, newdata = data[test_indices, ])
  actual <- data[test_indices, all.vars(formula)[1]]
  sqrt(mean((actual - predictions)^2))   # RMSE: lower is better
}
summary(CCI.test(Y ~ X | Z1 + Z2, data = dat, mlfunc = lm_wrapper, tail = "left",
                 seed = 1, progress = FALSE))
#> 
#> Computational Conditional Independence Test
#> --------------------------------------------
#> Method:    CCI test using lm_wrapper 
#> Formula:   Y ~ X | Z1 + Z2 
#> Permutations:  160 
#> Metric:    lm_wrapper 
#> Tail:      left 
#> Statistic: 1.029 
#> P-value:   0.06832 
#> 
#> MC sample:   1
summary(CCI.test(Y ~ X | Z1, data = dat, mlfunc = lm_wrapper, tail = "left",
                 seed = 1, progress = FALSE))
#> 
#> Computational Conditional Independence Test
#> --------------------------------------------
#> Method:    CCI test using lm_wrapper 
#> Formula:   Y ~ X | Z1 
#> Permutations:  160 
#> Metric:    lm_wrapper 
#> Tail:      left 
#> Statistic: 1.337 
#> P-value:   0.006211 
#> 
#> MC sample:   1

Logistic regression with extra arguments

Arguments that CCI.test() does not know are passed on to mlfunc through .... Here a logistic regression returns the log loss of the predicted probabilities, and the constant used to keep the probabilities away from 0 and 1 is given as an argument:

logistic_wrapper <- function(formula, data, train_indices, test_indices, clip = 1e-6, ...) {
  model <- glm(formula, data = data[train_indices, ], family = binomial)
  prob <- predict(model, newdata = data[test_indices, ], type = "response")
  prob <- pmin(pmax(prob, clip), 1 - clip)
  actual <- data[test_indices, all.vars(formula)[1]]
  is_second <- actual == levels(actual)[2]    # glm models the probability of the second level
  -mean(ifelse(is_second, log(prob), log(1 - prob)))   # log loss: lower is better
}
summary(CCI.test(Y ~ X | Z1, data = cat_data, mlfunc = logistic_wrapper, tail = "left",
                 clip = 1e-4, poly = FALSE, interaction = FALSE, seed = 1, progress = FALSE))
#> 
#> Computational Conditional Independence Test
#> --------------------------------------------
#> Method:    CCI test using logistic_wrapper 
#> Formula:   Y ~ X | Z1 
#> Permutations:  160 
#> Metric:    logistic_wrapper 
#> Tail:      left 
#> Statistic: 0.4914 
#> P-value:   0.006211 
#> 
#> MC sample:   1

Probability-based metrics like the log loss usually give more power than the share of correct classifications, since they use how confident the predictions are.

Any model from caret

The caret package gives a common interface to more than 200 models, which makes a general wrapper easy. The caret method and model parameters are given as arguments:

caret_wrapper <- function(formula, data, train_indices, test_indices, caret_method, ...) {
  model <- caret::train(formula, data = data[train_indices, ], method = caret_method,
                        trControl = caret::trainControl(method = "none"), ...)
  predictions <- predict(model, newdata = data[test_indices, ])
  actual <- data[test_indices, all.vars(formula)[1]]
  sqrt(mean((actual - predictions)^2))
}
summary(CCI.test(Y ~ X | Z1, data = dat, mlfunc = caret_wrapper, tail = "left",
                 caret_method = "knn", tuneGrid = data.frame(k = 15),
                 seed = 1, progress = FALSE))
#> Warning in value[[3L]](cond): Error in iteration 1: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 2: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 3: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 4: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 5: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 6: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 7: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 8: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 9: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 10: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 11: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 12: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 13: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 14: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 15: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 16: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 17: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 18: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 19: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 20: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 21: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 22: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 23: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 24: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 25: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 26: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 27: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 28: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 29: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 30: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 31: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 32: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 33: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 34: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 35: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 36: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 37: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 38: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 39: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 40: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 41: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 42: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 43: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 44: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 45: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 46: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 47: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 48: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 49: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 50: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 51: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 52: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 53: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 54: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 55: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 56: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 57: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 58: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 59: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 60: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 61: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 62: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 63: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 64: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 65: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 66: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 67: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 68: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 69: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 70: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 71: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 72: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 73: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 74: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 75: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 76: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 77: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 78: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 79: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 80: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 81: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 82: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 83: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 84: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 85: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 86: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 87: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 88: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 89: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 90: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 91: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 92: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 93: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 94: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 95: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 96: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 97: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 98: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 99: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 100: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 101: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 102: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 103: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 104: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 105: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 106: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 107: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 108: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 109: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 110: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 111: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 112: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 113: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 114: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 115: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 116: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 117: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 118: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 119: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 120: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 121: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 122: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 123: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 124: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 125: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 126: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 127: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 128: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 129: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 130: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 131: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 132: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 133: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 134: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 135: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 136: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 137: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 138: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 139: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 140: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 141: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 142: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 143: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 144: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 145: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 146: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 147: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 148: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 149: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 150: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 151: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 152: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 153: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 154: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 155: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 156: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 157: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 158: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 159: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 160: could not find function
#> "knnreg"
#> Warning in value[[3L]](cond): Error in iteration 1: could not find function
#> "knnreg"
#> Warning: The test statistic is missing (the model fit failed), so the p-value
#> is NA.
#> 
#> Computational Conditional Independence Test
#> --------------------------------------------
#> Method:    CCI test using caret_wrapper 
#> Formula:   Y ~ X | Z1 
#> Permutations:  160 
#> Metric:    caret_wrapper 
#> Tail:      left 
#> Statistic: NA 
#> P-value:   NA 
#> 
#> MC sample:   1

Tips

Return a single number. If a model fails to fit in a Monte Carlo sample, that sample is left out with a warning, so errors in your function show up as warnings about missing values in the null distribution.
Always set tail. CCI.test() stops with an error if a metricfunc or mlfunc is given without it.
The functions are stored in the result, so QQplot() repeats the test with the same custom model and metric.
Use seed in CCI.test() for reproducible results.

QQplot(res_r2, nperm = 40, progress = FALSE)

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.