Multiple Imputation and Bootstrapping - Method boot_MI
Martijn W Heymans
2020-09-24
1 Introduction
This page contains information of the boot_MI method that is implemented
in the psfmi_perform function of the psfmi package and that combines
Multiple Imputation with bootstrapping for the validation of
logistic regression / prediction models. Internal validation is done of the last model that is
selected by the function psfmi_lr. An explanation and examples of how to use the methods can be found below.
2 Method boot_MI
With the method boot_MI, first bootstrap samples are drawn from the original dataset with missing values and than multiple imputation is applied in each bootstrap sample. The pooled model is analyzed in the each bootstrap training data and subsequently tested in original multiply imputed data. The method can be performed in combination with backward or forward selection selection.
How these steps work is visualized in the Figure below.
Figure 2.1: Schematic overview of the boot_MI method
3 Examples
3.1 Method boot_MI
internal validation is done of the last model that is selected by the function psfmi_lr.
In the example below, psfmi_lr is used with p.crit set at 1. This setting is also
used and in the psfmi_perform function. This means that first the full model is pooled and
subsequently interval validation is also done of this full model.
library(psfmi)
pool_lr <- psfmi_lr(data=lbpmilr, formula = Chronic ~ Pain + JobDemands + rcs(Tampascale, 3) +
factor(Satisfaction) + Smoking, p.crit = 1, direction="FW",
nimp=5, impvar="Impnr", method="D1")
res_MI_boot <- psfmi_perform(pool_lr, val_method = "boot_MI", data_orig = lbp_orig, nboot = 5,
p.crit=1, nimp_mice = 3, direction = "BW", miceImp = miceImp,
printFlag = FALSE)##
## Boot 1##
## Boot 2##
## Boot 3##
## Boot 4##
## Boot 5##
## p.crit = 1, validation is done without variable selectionres_MI_boot## $stats_val
## Orig Apparent Test Optimism Corrected
## AUC 0.8871000 0.9033000 0.8765800 0.02672000 0.8603800
## R2 0.5605521 0.6076112 0.5307290 0.07688221 0.4836699
## Brier Scaled 0.4514569 0.4991465 0.4122721 0.08687432 0.3645826
## Slope 1.0000000 1.0000000 0.8265696 0.17343036 0.8265696
##
## $intercept_test
## intercept
## -0.09921869
##
## $res_boot
## ROC_app ROC_test R2_app R2_test Brier_sc_app Brier_sc_test
## Boot 1 0.8986 0.8857 0.5748913 0.5467043 0.4557457 0.4240552
## Boot 2 0.9178 0.8790 0.6667569 0.5412814 0.5841894 0.4343451
## Boot 3 0.8966 0.8834 0.5917839 0.5476373 0.4697460 0.4214247
## Boot 4 0.8995 0.8792 0.5881709 0.5347974 0.4991174 0.4301919
## Boot 5 0.9040 0.8556 0.6164528 0.4832243 0.4869337 0.3513437
## intercept Slope
## Boot 1 -0.10477141 0.8685152
## Boot 2 0.03905404 0.6860926
## Boot 3 -0.34670915 0.9631026
## Boot 4 0.10476835 0.8555809
## Boot 5 -0.18843528 0.7595569Back to Examples
3.2 Method boot_MI including BW selection
Internal validation is done of the last model that is selected by the function psfmi_lr.
In the example below, psfmi_lr is used with p.crit set at 1, and pooling is than done
without variable selection, i.e. first the full model is pooled. When subsequently
interval validation is done with the psfmi_perform function including BW, BW is applied in
each bootstrap sample from the full model.
library(psfmi)
pool_lr <- psfmi_lr(data=lbpmilr, Outcome="Chronic", predictors = c("Pain", "JobDemands", "Smoking"),
cat.predictors = "Satisfaction", spline.predictors = "Tampascale", nknots=3,
p.crit = 1, direction="FW", nimp=5, impvar="Impnr", method="D1")
res_MI_boot <- psfmi_perform(pool_lr, val_method = "boot_MI", data_orig = lbp_orig, nboot = 5,
p.crit=0.05, nimp_mice = 3, direction = "BW", miceImp = miceImp,
printFlag = FALSE)##
## Boot 1## Removed at Step 1 is - JobDemands## Removed at Step 2 is - Smoking## Removed at Step 3 is - rcs(Tampascale,3)##
## Selection correctly terminated,
## No more variables removed from the model##
## Boot 2## Removed at Step 1 is - JobDemands## Removed at Step 2 is - Pain## Removed at Step 3 is - Smoking##
## Selection correctly terminated,
## No more variables removed from the model##
## Boot 3## Removed at Step 1 is - JobDemands## Removed at Step 2 is - rcs(Tampascale,3)##
## Selection correctly terminated,
## No more variables removed from the model##
## Boot 4## Removed at Step 1 is - JobDemands## Removed at Step 2 is - Smoking## Removed at Step 3 is - rcs(Tampascale,3)## Removed at Step 4 is - factor(Satisfaction)##
## Selection correctly terminated,
## No more variables removed from the model##
## Boot 5## Removed at Step 1 is - JobDemands## Removed at Step 2 is - rcs(Tampascale,3)## Removed at Step 3 is - Smoking## Removed at Step 4 is - factor(Satisfaction)##
## Selection correctly terminated,
## No more variables removed from the modelres_MI_boot## $stats_val
## Orig Apparent Test Optimism Corrected
## AUC 0.8871000 0.8675400 0.8486600 0.01888000 0.8682200
## R2 0.5605521 0.5158842 0.4678360 0.04804823 0.5125039
## Brier Scaled 0.4514569 0.4257104 0.3697809 0.05592955 0.3955274
## Slope 1.0000000 1.0000000 0.8915697 0.10843035 0.8915697
##
## $intercept_test
## intercept
## 0.05263674
##
## $res_boot
## ROC_app ROC_test R2_app R2_test Brier_sc_app Brier_sc_test
## Boot 1 0.9034 0.8414 0.6081386 0.4741075 0.5137106 0.3669051
## Boot 2 0.8833 0.8556 0.5458448 0.4783700 0.4333424 0.3370175
## Boot 3 0.8725 0.8563 0.5375735 0.4863413 0.4508174 0.4038069
## Boot 4 0.8412 0.8450 0.4448085 0.4501806 0.3650045 0.3673242
## Boot 5 0.8373 0.8450 0.4430559 0.4501806 0.3656770 0.3738506
## intercept Slope
## Boot 1 0.08803902 0.7578973
## Boot 2 -0.23768888 0.7821832
## Boot 3 0.19396899 0.9352751
## Boot 4 0.30042374 0.9852542
## Boot 5 -0.08155918 0.9972385Back to Examples
3.3 Method boot_MI including FW selection
Internal validation is done of the last model that is selected by the function psfmi_lr.
In the example below, psfmi_lr is used with p.crit set at 1, and pooling is than done
without variable selection, i.e. first the full model is pooled. When subsequently
interval validation is done with the psfmi_perform function including FW, FW is applied in
each bootstrap sample from the full model.
library(psfmi)
pool_lr <- psfmi_lr(data=lbpmilr, formula = Chronic ~ Pain + JobDemands + rcs(Tampascale, 3) +
factor(Satisfaction) + Smoking, p.crit = 1, direction="FW",
nimp=5, impvar="Impnr", method="D1")
res_MI_boot <- psfmi_perform(pool_lr, val_method = "boot_MI", data_orig = lbp_orig, nboot = 5,
p.crit=0.05, nimp_mice = 3, direction = "FW", miceImp = miceImp,
printFlag = FALSE)##
## Boot 1## Entered at Step 1 is - Pain## Entered at Step 2 is - rcs(Tampascale,3)## Entered at Step 3 is - factor(Satisfaction)##
## Selection correctly terminated,
## No new variables entered the model##
## Boot 2## Entered at Step 1 is - Pain## Entered at Step 2 is - factor(Satisfaction)##
## Selection correctly terminated,
## No new variables entered the model##
## Boot 3## Entered at Step 1 is - Pain## Entered at Step 2 is - factor(Satisfaction)## Entered at Step 3 is - rcs(Tampascale,3)##
## Selection correctly terminated,
## No new variables entered the model##
## Boot 4## Entered at Step 1 is - Pain## Entered at Step 2 is - factor(Satisfaction)##
## Selection correctly terminated,
## No new variables entered the model##
## Boot 5## Entered at Step 1 is - Pain## Entered at Step 2 is - factor(Satisfaction)## Entered at Step 3 is - rcs(Tampascale,3)##
## Selection correctly terminated,
## No new variables entered the modelres_MI_boot## $stats_val
## Orig Apparent Test Optimism Corrected
## AUC 0.8871000 0.8960800 0.6618800 0.2342000 0.6529000
## R2 0.5605521 0.5952339 0.1966421 0.3985918 0.1619604
## Brier Scaled 0.4514569 0.4811972 -0.7624482 1.2436454 -0.7921885
## Slope 1.0000000 1.0000000 -0.1297440 1.1297440 -0.1297440
##
## $intercept_test
## intercept
## -3.357985
##
## $res_boot
## ROC_app ROC_test R2_app R2_test Brier_sc_app Brier_sc_test
## Boot 1 0.9042 0.5294 0.6136112 0.01522196 0.4967869 -0.7584502
## Boot 2 0.8587 0.4165 0.4987094 0.02239926 0.3865967 -0.7538605
## Boot 3 0.9281 0.8343 0.6810738 0.38082731 0.5684618 -0.7666667
## Boot 4 0.9217 0.7100 0.6582930 0.21705351 0.5491232 -0.7665967
## Boot 5 0.8677 0.8192 0.5244819 0.34770845 0.4050175 -0.7666667
## intercept Slope
## Boot 1 0.3069063 0.07602518
## Boot 2 -1.3393491 -0.17005915
## Boot 3 -5.7105637 -0.05125290
## Boot 4 -4.7142449 -0.38628532
## Boot 5 -5.3326725 -0.11714792