Solved – Outer crossvalidation cycle in caret package (R)

Could somebody provide a nice example code how to best implement an outer crossvalidation cycle using the caret package in R? The package provides a convenient trainControl() argument to ajust the inner crossvalidation. However I would like to embed this into multiple outer crossvalidation cycles to get a more stable estimate of the prediction performance of the estimated models!

Inner and outer CV are used to perform classifier selection not to get a better prediction on the estimate. To get a better estimate, do a repeated cv. So to perform a 10-repeates 5-fold CV use

trainControl(method = "repeatedcv",number = 5,              ## repeated ten times              repeats = 10) 

But if what you really want is a nested CV, for example to select between a random forest or a svm) then as far as know you have to do the outer CV explicitly. What I did for an outer 5-fold, inner 10-fold was:

ntrain=length(ytrain)     train.ext=createFolds(ytrain,k=5,returnTrain=TRUE) test.ext=lapply(train.ext,function(x) (1:ntrain)[-x])  for (i in 1:5){     model<-train(Class ~ ., data = training[train.ext[[i]]],                  trControl=trainControl(method = "cv",number = 10),                  ...     ...     } 

Similar Posts:

Rate this post

Leave a Comment