‘vtreat’ is a package that prepares arbitrary data frames into clean data frames that are ready for analysis. A clean data frame:
To achieve this a number of techniques are used. Principally:
For more details see: the ‘vtreat’ article and update.
The use pattern is:
designTreatmentsC() or designTreatmentsN() to design a treatment planprepare() to apply the plan to data frames.The main feature of ‘vtreat’ is that all data preparation is “y-aware”: it uses the relations of effective variables to the dependent or outcome variable to encode the effective variables.
The structure returned from designTreatmentsN() or designTreatmentsC() includes a list of “treatments”: objects that encapsulate the transformation process from the original variables to the new “clean” variables.
In addition to the treatment objects designTreatmentsC() and designTreatmentsN() also return a data frame named scoreFrame which contains columns:
varName: name of new variableorigName: name of original variable that the variable was derived from (can repeat)varMove : logical TRUE if the variable varied during training; only variables that move will be in the treated frame.PRESSRsquared : a PRESS-held out R-squared of a linear fit from each variable to the y-value. Scores of zero and below are very bad, scores near one are very good.psig : significance of observed variable ‘PRESSRsquared’ value under an in-sample permutation test.catPRSquared : for categorical outcomes: deviance based pseudo-Rsquared.csig : for categorical outcomes: significance of observed variable catPRSquared value under an in-sample permutation test.sig : csig for categorical outcomes, psig otherwise.In all cases we have two undesirable upward biases on the scores:
PRESSRsquared or logistic regression for catPRSquared). So in each of these cases we would like the regression itself to only be evaluated on held-out data. The PRESS statistic does just that (fast 1-way cross validation). The catPseudoRSquared performs explicit 1-way cross validation for small data sets and hold-out scoring for larger data sets.‘vtreat’ uses a number of cross-training and jackknife style procedures to try to mitigate these effects. The suggested best practice (if you have enough data) is to split your data randomly into at least the following disjoint data sets:
designTreatmentsC() or designTreatmentsN() step and not used again for training or test.prepare()) for training your model.prepare()) for estimating your model’s out of training performance.Taking the extra step to perform the designTreatmentsC() or designTreatmentsN() on data disjoint from training makes the training data more exchangeable with test and avoids the issue that ‘vtreat’ may be hiding a large number of degrees of freedom in variables it derives from large categoricals.
Some trivial execution examples (not demonstrating any cal/train/test split) are given below. Variables that do not move during hold-out testing are considered “not to move.”
library(vtreat)
dTrainC <- data.frame(x=c('a','a','a','b','b',NA),
z=c(1,2,3,4,NA,6),y=c(FALSE,FALSE,TRUE,FALSE,TRUE,TRUE))
head(dTrainC)## x z y
## 1 a 1 FALSE
## 2 a 2 FALSE
## 3 a 3 TRUE
## 4 b 4 FALSE
## 5 b NA TRUE
## 6 <NA> 6 TRUEdTestC <- data.frame(x=c('a','b','c',NA),z=c(10,20,30,NA))
head(dTestC)## x z
## 1 a 10
## 2 b 20
## 3 c 30
## 4 <NA> NAtreatmentsC <- designTreatmentsC(dTrainC,colnames(dTrainC),'y',TRUE)## [1] "desigining treatments Thu Apr 28 09:46:05 2016"
## [1] "design var x Thu Apr 28 09:46:05 2016"
## [1] "design var z Thu Apr 28 09:46:05 2016"
## [1] "scoring treatments Thu Apr 28 09:46:05 2016"
## [1] "have treatment plan Thu Apr 28 09:46:05 2016"
## [1] "rescoring complex variables Thu Apr 28 09:46:05 2016"
## [1] "done rescoring complex variables Thu Apr 28 09:46:05 2016"print(treatmentsC)## $treatments
## $treatments[[1]]
## [1] "vtreat 'Categoric Indicators'('x'(integer,factor)->character->'x_lev_NA','x_lev_x.a','x_lev_x.b')"
##
## $treatments[[2]]
## [1] "vtreat 'Prevalence Code'('x'(integer,factor)->character->'x_catP')"
##
## $treatments[[3]]
## [1] "vtreat 'Bayesian Impact Code'('x'(integer,factor)->character->'x_catB')"
##
## $treatments[[4]]
## [1] "vtreat 'Scalable pass through'('z'(double,numeric)->numeric->'z_clean')"
##
## $treatments[[5]]
## [1] "vtreat 'is.bad'('z'(double,numeric)->numeric->'z_isBAD')"
##
##
## $scoreFrame
## varName varMoves PRESSRsquared psig sig catPRSquared
## 1 x_lev_NA TRUE -0.48999814 1.000000 0.20766228 1.908745e-01
## 2 x_lev_x.a TRUE -0.99999000 1.000000 0.40972582 8.170417e-02
## 3 x_lev_x.b TRUE -1.51850123 1.000000 1.00000000 -2.220446e-16
## 4 x_catP TRUE -0.52729749 1.000000 0.25493078 1.558205e-01
## 5 x_catB TRUE 0.11661930 0.507669 0.05044486 4.600526e-01
## 6 z_clean TRUE -0.09178145 1.000000 0.14299775 2.579298e-01
## 7 z_isBAD TRUE -0.48999814 1.000000 0.20766228 1.908745e-01
## csig needsSplit origName code
## 1 0.20766228 FALSE x lev
## 2 0.40972582 FALSE x lev
## 3 1.00000000 FALSE x lev
## 4 0.25493078 TRUE x catP
## 5 0.05044486 TRUE x catB
## 6 0.14299775 FALSE z clean
## 7 0.20766228 FALSE z isBAD
##
## $outcomename
## [1] "y"
##
## $outcomeTarget
## [1] TRUE
##
## $outcomeType
## [1] "Binary"
##
## attr(,"class")
## [1] "treatmentplan"print(treatmentsC$treatments[[1]])## [1] "vtreat 'Categoric Indicators'('x'(integer,factor)->character->'x_lev_NA','x_lev_x.a','x_lev_x.b')"Here we demonstrate the optional scaling feature of prepare(), which scales and centers all significant variables to mean 0, and slope 1 with respect to y: In other words, it rescales the variables to “y-units”. This is useful for downstream principal components analysis. Note: variables perfectly uncorrelated with y necessarily have slope 0 and can’t be “scaled” to slope 1, however for the same reason these variables will be insignificant and can be pruned by pruneSig.
scale=FALSE by default.
dTrainCTreated <- prepare(treatmentsC,dTrainC,pruneSig=c(),scale=TRUE)
head(dTrainCTreated)## x_lev_NA x_lev_x.a x_lev_x.b x_catP x_catB z_clean z_isBAD
## 1 -0.1 -0.1666667 0 -0.2 -0.18838870 -0.38648649 -0.1
## 2 -0.1 -0.1666667 0 -0.2 -0.18838870 -0.21081081 -0.1
## 3 -0.1 -0.1666667 0 -0.2 -0.18838870 -0.03513514 -0.1
## 4 -0.1 0.1666667 0 0.1 0.05164882 0.14054054 -0.1
## 5 -0.1 0.1666667 0 0.1 0.05164882 0.00000000 0.5
## 6 0.5 0.1666667 0 0.4 0.46186845 0.49189189 -0.1
## y
## 1 FALSE
## 2 FALSE
## 3 TRUE
## 4 FALSE
## 5 TRUE
## 6 TRUEvarsC <- setdiff(colnames(dTrainCTreated),'y')
# all input variables should be mean 0
sapply(dTrainCTreated[,varsC,drop=FALSE],mean)## x_lev_NA x_lev_x.a x_lev_x.b x_catP x_catB
## 6.938894e-18 0.000000e+00 0.000000e+00 -7.401487e-17 2.312965e-18
## z_clean z_isBAD
## 2.775558e-17 6.938894e-18# all slopes should be 1 for variables with dTrainCTreated$scoreFrame$sig<1
sapply(varsC,function(c) { lm(paste('y',c,sep='~'),
data=dTrainCTreated)$coefficients[[2]]})## x_lev_NA x_lev_x.a x_lev_x.b x_catP x_catB z_clean z_isBAD
## 1 1 NA 1 1 1 1dTestCTreated <- prepare(treatmentsC,dTestC,pruneSig=c(),scale=TRUE)
head(dTestCTreated)## x_lev_NA x_lev_x.a x_lev_x.b x_catP x_catB z_clean z_isBAD
## 1 -0.1 -0.1666667 0 -0.2 -0.18838870 0.4918919 -0.1
## 2 -0.1 0.1666667 0 0.1 0.05164882 0.4918919 -0.1
## 3 -0.1 0.1666667 0 0.7 0.05164882 0.4918919 -0.1
## 4 0.5 0.1666667 0 0.4 0.46186845 0.0000000 0.5# numeric example
dTrainN <- data.frame(x=c('a','a','a','a','b','b',NA),
z=c(1,2,3,4,5,NA,7),y=c(0,0,0,1,0,1,1))
head(dTrainN)## x z y
## 1 a 1 0
## 2 a 2 0
## 3 a 3 0
## 4 a 4 1
## 5 b 5 0
## 6 b NA 1dTestN <- data.frame(x=c('a','b','c',NA),z=c(10,20,30,NA))
head(dTestN)## x z
## 1 a 10
## 2 b 20
## 3 c 30
## 4 <NA> NAtreatmentsN = designTreatmentsN(dTrainN,colnames(dTrainN),'y')## [1] "desigining treatments Thu Apr 28 09:46:06 2016"
## [1] "design var x Thu Apr 28 09:46:06 2016"
## [1] "design var z Thu Apr 28 09:46:06 2016"
## [1] "scoring treatments Thu Apr 28 09:46:06 2016"
## [1] "have treatment plan Thu Apr 28 09:46:06 2016"
## [1] "rescoring complex variables Thu Apr 28 09:46:06 2016"
## [1] "done rescoring complex variables Thu Apr 28 09:46:06 2016"print(treatmentsN)## $treatments
## $treatments[[1]]
## [1] "vtreat 'Categoric Indicators'('x'(integer,factor)->character->'x_lev_NA','x_lev_x.a','x_lev_x.b')"
##
## $treatments[[2]]
## [1] "vtreat 'Prevalence Code'('x'(integer,factor)->character->'x_catP')"
##
## $treatments[[3]]
## [1] "vtreat 'Scalable Impact Code'('x'(integer,factor)->character->'x_catN')"
##
## $treatments[[4]]
## [1] "vtreat 'Deviation Fact'('x'(integer,factor)->character->'x_catD')"
##
## $treatments[[5]]
## [1] "vtreat 'Scalable pass through'('z'(double,numeric)->numeric->'z_clean')"
##
## $treatments[[6]]
## [1] "vtreat 'is.bad'('z'(double,numeric)->numeric->'z_isBAD')"
##
##
## $scoreFrame
## varName varMoves PRESSRsquared psig sig needsSplit origName
## 1 x_lev_NA TRUE -0.379258198 1.0000000 1.0000000 FALSE x
## 2 x_lev_x.a TRUE -0.652770648 1.0000000 1.0000000 FALSE x
## 3 x_lev_x.b TRUE -1.260402813 1.0000000 1.0000000 FALSE x
## 4 x_catP TRUE -0.389486438 1.0000000 1.0000000 TRUE x
## 5 x_catN TRUE -0.266446584 1.0000000 1.0000000 TRUE x
## 6 x_catD TRUE -0.927072127 1.0000000 1.0000000 TRUE x
## 7 z_clean TRUE 0.008889632 0.8406476 0.8406476 FALSE z
## 8 z_isBAD TRUE -0.379258198 1.0000000 1.0000000 FALSE z
## code
## 1 lev
## 2 lev
## 3 lev
## 4 catP
## 5 catN
## 6 catD
## 7 clean
## 8 isBAD
##
## $outcomename
## [1] "y"
##
## $outcomeType
## [1] "Numeric"
##
## attr(,"class")
## [1] "treatmentplan"dTrainNTreated <- prepare(treatmentsN,dTrainN,
pruneSig=c(),scale=TRUE)
head(dTrainNTreated)## x_lev_NA x_lev_x.a x_lev_x.b x_catP x_catN x_catD
## 1 -0.0952381 -0.1785714 -0.02857143 -0.2 -0.17857143 -0.1785714
## 2 -0.0952381 -0.1785714 -0.02857143 -0.2 -0.17857143 -0.1785714
## 3 -0.0952381 -0.1785714 -0.02857143 -0.2 -0.17857143 -0.1785714
## 4 -0.0952381 -0.1785714 -0.02857143 -0.2 -0.17857143 -0.1785714
## 5 -0.0952381 0.2380952 0.07142857 0.2 0.07142857 0.2380952
## 6 -0.0952381 0.2380952 0.07142857 0.2 0.07142857 0.2380952
## z_clean z_isBAD y
## 1 -0.41904762 -0.0952381 0
## 2 -0.26190476 -0.0952381 0
## 3 -0.10476190 -0.0952381 0
## 4 0.05238095 -0.0952381 1
## 5 0.20952381 -0.0952381 0
## 6 0.00000000 0.5714286 1varsN <- setdiff(colnames(dTrainNTreated),'y')
# all input variables should be mean 0
sapply(dTrainNTreated[,varsN,drop=FALSE],mean) ## x_lev_NA x_lev_x.a x_lev_x.b x_catP x_catN
## -3.965082e-18 0.000000e+00 2.974054e-18 7.930164e-17 0.000000e+00
## x_catD z_clean z_isBAD
## 0.000000e+00 4.757324e-17 -3.967986e-18# all slopes should be 1 for variables with treatmentsN$scoreFrame$sig<1
sapply(varsN,function(c) { lm(paste('y',c,sep='~'),
data=dTrainNTreated)$coefficients[[2]]}) ## x_lev_NA x_lev_x.a x_lev_x.b x_catP x_catN x_catD z_clean
## 1 1 1 1 1 1 1
## z_isBAD
## 1# prepared frame
dTestNTreated <- prepare(treatmentsN,dTestN,
pruneSig=c())
head(dTestNTreated)## x_lev_NA x_lev_x.a x_lev_x.b x_catP x_catN x_catD z_clean
## 1 0 1 0 0.5714286 -0.17857143 0.5000000 7.000000
## 2 0 0 1 0.2857143 0.07142857 0.7071068 7.000000
## 3 0 0 0 0.0000000 0.00000000 0.7071068 7.000000
## 4 1 0 0 0.1428571 0.57142857 0.7071068 3.666667
## z_isBAD
## 1 0
## 2 0
## 3 0
## 4 1# scaled prepared frame
dTestNTreatedS <- prepare(treatmentsN,dTestN,
pruneSig=c(),scale=TRUE)
head(dTestNTreatedS)## x_lev_NA x_lev_x.a x_lev_x.b x_catP x_catN x_catD
## 1 -0.0952381 -0.1785714 -0.02857143 -0.2 -1.785714e-01 -0.1785714
## 2 -0.0952381 0.2380952 0.07142857 0.2 7.142857e-02 0.2380952
## 3 -0.0952381 0.2380952 -0.02857143 0.6 -1.982541e-17 0.2380952
## 4 0.5714286 0.2380952 -0.02857143 0.4 5.714286e-01 0.2380952
## z_clean z_isBAD
## 1 0.5238095 -0.0952381
## 2 0.5238095 -0.0952381
## 3 0.5238095 -0.0952381
## 4 0.0000000 0.5714286Related work: