Cross-validation (CV) is an essentially simple and intuitively reasonable approach to estimating the predictive accuracy of regression models. CV is developed in many standard sources on regression modeling and “machine learning”—we particularly recommend James, Witten, Hastie, & Tibshirani (2021, secs. 5.1, 5.3)—and so we will describe the method only briefly here before taking up computational issues and some examples.
Validating research by replication on independently collected data is a common scientific norm. Emulating this process in a single study by data-division is less common: The data are randomly divided into two, possibly equal-size, parts; the first part is used to develop and fit a statistical model; and then the second part is used to assess the adequacy of the model fit to the first part of the data. Data-division, however, suffers from two problems: (1) Dividing the data decreases the sample size and thus increases sampling error; and (2), even more disconcertingly, particularly in smaller samples, the results can vary substantially based on the random division of the data: See Harrell (2015, sec. 5.3) for this and other remarks about data-division and cross-validation.
Cross-validation speaks to both of these issues. In CV, the data are randomly divided as equally as possible into several, say \(k\), parts, called “folds.” The statistical model is fit \(k\) times, leaving each fold out in turn. Each fitted model is then used to predict the response variable for the omitted fold, computing some CV criterion or “cost” measure, such as the mean-squared error of prediction. The CV criterion is then averaged over the \(k\) folds. In the extreme \(k = n\), the number of cases in the data, thus omitting individual cases and refitting the model \(n\) times—a procedure termed “leave-one-out (LOO) cross-validation.”
Because the \(k\) models are each fit to \(n - 1\) cases, LOO CV produces a nearly unbiased estimate of prediction error. The \(n\) regression models are highly statistical dependent, however, based as they are on nearly the same data, and so the resulting estimate of prediction error has relatively large variance. In contrast, estimated prediction error for \(k\)-fold CV with \(k = 5\) or \(10\) (commonly employed choices) are somewhat biased but have smaller variance. It is also possible to correct \(k\)-fold CV for bias (see below).
Auto
dataThe data for this example are drawn from the ISLR2
package for R, associated with James et al.
(2021), and the presentation here is close (though not identical)
to that in the original source (James et al.,
2021, secs. 5.1, 5.3), and it demonstrates the use of the
cv()
function in the cv package.1
The Auto
dataset contains information about 392
cars:
data("Auto", package="ISLR2")
head(Auto)
#> mpg cylinders displacement horsepower weight acceleration year origin
#> 1 18 8 307 130 3504 12.0 70 1
#> 2 15 8 350 165 3693 11.5 70 1
#> 3 18 8 318 150 3436 11.0 70 1
#> 4 16 8 304 150 3433 12.0 70 1
#> 5 17 8 302 140 3449 10.5 70 1
#> 6 15 8 429 198 4341 10.0 70 1
#> name
#> 1 chevrolet chevelle malibu
#> 2 buick skylark 320
#> 3 plymouth satellite
#> 4 amc rebel sst
#> 5 ford torino
#> 6 ford galaxie 500
dim(Auto)
#> [1] 392 9
With the exception of origin
(which we don’t use here),
these variables are largely self-explanatory, except possibly for units
of measurement: for details see
help("Auto", package="ISLR2")
.
We’ll focus here on the relationship of mpg
(miles per
gallon) to horsepower
, as displayed in the following
scatterplot:
mpg
vs horsepower
for the Auto
data
The relationship between the two variables is monotone, decreasing, and nonlinear. Following James et al. (2021), we’ll consider approximating the relationship by a polynomial regression, with the degree of the polynomial \(p\) ranging from 1 (a linear regression) to 10.2 Polynomial fits for \(p = 1\) to \(5\) are shown in the following figure:
plot(mpg ~ horsepower, data=Auto)
horsepower <- with(Auto,
seq(min(horsepower), max(horsepower),
length=1000))
for (p in 1:5){
m <- lm(mpg ~ poly(horsepower,p), data=Auto)
mpg <- predict(m, newdata=data.frame(horsepower=horsepower))
lines(horsepower, mpg, col=p + 1, lty=p, lwd=2)
}
legend("topright", legend=1:5, col=2:6, lty=1:5, lwd=2,
title="Degree", inset=0.02)
mpg
vs horsepower
for the Auto
data
The linear fit is clearly inappropriate; the fits for \(p = 2\) (quadratic) through \(4\) are very similar; and the fit for \(p = 5\) probably over-fits the data by
chasing one or two relatively high mpg
values at the
right.
The following graph shows two measures of estimated squared error as a function of polynomial-regression degree: The mean-squared error (MSE), defined as \(\mathsf{MSE} = \sum (y_i - \widehat{y}_i)^2/n\), and the usual unbiased estimated error variance, defined as \(\widehat{\sigma}^2 = \sum (y_i - \widehat{y}_i)^2/(n - p - 1)\). The former necessarily declines with \(p\) (or, more strictly, can’t increase with \(p\)), while the latter gets slightly larger for the largest values of \(p\), with the “best” value, by a small margin, for \(p = 7\).
library("cv") # for mse() and other functions
se <- mse <- numeric(10)
for (p in 1:10){
m <- lm(mpg ~ poly(horsepower, p), data=Auto)
mse[p] <- mse(Auto$mpg, fitted(m))
se[p] <- summary(m)$sigma
}
plot(c(1, 10), range(mse, se^2), type="n",
xlab="Degree of polynomial, p",
ylab="Estimated Squared Error")
lines(1:10, mse, lwd=2, lty=1, col=2, pch=16, type="b")
lines(1:10, se^2, lwd=2, lty=2, col=3, pch=17, type="b")
legend("topright", inset=0.02,
legend=c(expression(hat(sigma)^2), "MSE"),
lwd=2, lty=2:1, col=3:2, pch=17:16)
Estimated squared error as a function of polynomial degree, \(p\)
The code for this graph uses the mse()
function from the
cv package to compute the MSE for each fit.
cv()
The generic cv()
function has an "lm"
method, which by default performs \(k =
10\)-fold CV:
m.auto <- lm(mpg ~ poly(horsepower, 2), data=Auto)
summary(m.auto)
#>
#> Call:
#> lm(formula = mpg ~ poly(horsepower, 2), data = Auto)
#>
#> Residuals:
#> Min 1Q Median 3Q Max
#> -14.714 -2.594 -0.086 2.287 15.896
#>
#> Coefficients:
#> Estimate Std. Error t value Pr(>|t|)
#> (Intercept) 23.446 0.221 106.1 <2e-16 ***
#> poly(horsepower, 2)1 -120.138 4.374 -27.5 <2e-16 ***
#> poly(horsepower, 2)2 44.090 4.374 10.1 <2e-16 ***
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#> Residual standard error: 4.37 on 389 degrees of freedom
#> Multiple R-squared: 0.688, Adjusted R-squared: 0.686
#> F-statistic: 428 on 2 and 389 DF, p-value: <2e-16
cv(m.auto)
#> R RNG seed set to 398475
#> 10-Fold Cross Validation
#> method: Woodbury
#> criterion: mse
#> cross-validation criterion = 19.238
#> bias-adjusted cross-validation criterion = 19.224
#> full-sample criterion = 18.985
The "lm"
method by default uses mse()
as
the CV criterion and the Woodbury matrix identity to update the
regression with each fold deleted without having literally to refit the
model. Computational details are discussed in the final section of this
vignette. The function reports the CV estimate of MSE, a biased-adjusted
estimate of the MSE (the bias adjustment is explained in the final
section), and the MSE is also computed for the original, full-sample
regression. Because the division of the data into 10 folds is random,
cv()
explicitly (randomly) generates and saves a seed for
R’s pseudo-random number generator, to make the results replicable. The
user can also specify the seed directly via the seed
argument to cv()
.
To perform LOO CV, we can set the k
argument to
cv()
to the number of cases in the data, here
k=392
, or, more conveniently, to k="loo"
or
k="n"
:
cv(m.auto, k="loo")
#> n-Fold Cross Validation
#> method: hatvalues
#> criterion: mse
#> cross-validation criterion = 19.248
For LOO CV of a linear model, cv()
by default uses the
hatvalues from the model fit to the full data for the LOO updates, and
reports only the CV estimate of MSE. Alternative methods are to use the
Woodbury matrix identity or the “naive” approach of literally refitting
the model with each case omitted. All three methods produce exact
results for a linear model (within the precision of floating-point
computations):
cv(m.auto, k="loo", method="naive")
#> n-Fold Cross Validation
#> method: naive
#> criterion: mse
#> cross-validation criterion = 19.248
#> bias-adjusted cross-validation criterion = 19.248
#> full-sample criterion = 18.985
cv(m.auto, k="loo", method="Woodbury")
#> n-Fold Cross Validation
#> method: Woodbury
#> criterion: mse
#> cross-validation criterion = 19.248
#> bias-adjusted cross-validation criterion = 19.248
#> full-sample criterion = 18.985
The "naive"
and "Woodbury"
methods also
return the bias-adjusted estimate of MSE and the full-sample MSE, but
bias isn’t an issue for LOO CV.
This is a small regression problem and all three computational
approaches are essentially instantaneous, but it is still of interest to
investigate their relative speed. In this comparison, we include the
cv.glm()
function from the boot package,
which takes the naive approach, and for which we have to fit the linear
model as an equivalent Gaussian GLM. We use the
microbenchmark()
function from the package of the same name
for the timings (Mersmann, 2023):
m.auto.glm <- glm(mpg ~ poly(horsepower, 2), data=Auto)
boot::cv.glm(Auto, m.auto.glm)$delta
#> [1] 19.248 19.248
microbenchmark::microbenchmark(
hatvalues = cv(m.auto, k="loo"),
Woodbury = cv(m.auto, k="loo", method="Woodbury"),
naive = cv(m.auto, k="loo", method="naive"),
cv.glm = boot::cv.glm(Auto, m.auto.glm),
times=10
)
#> Warning in microbenchmark::microbenchmark(hatvalues = cv(m.auto, k = "loo"), :
#> less accurate nanosecond times to avoid potential integer overflows
#> Unit: microseconds
#> expr min lq mean median uq max neval cld
#> hatvalues 964.69 967.64 1129.8 1160.3 1196.3 1340.6 10 a
#> Woodbury 10483.78 10762.46 11264.0 10867.0 11363.8 14419.1 10 a
#> naive 215370.33 216389.64 229553.3 218345.7 220837.6 276001.9 10 b
#> cv.glm 377467.48 380847.03 405010.5 383950.0 437946.5 495947.4 10 c
On our computer, using the hatvalues is about an order of magnitude faster than employing Woodbury matrix updates, and more than two orders of magnitude faster than refitting the model.3
The cv()
function also has a method that can be applied
to a list of regression models for the same data, composed using the
models()
function. For \(k\)-fold CV, the same folds are used for
the competing models, which reduces random error in their comparison.
This result can also be obtained by specifying a common seed for R’s
random-number generator while applying cv()
separately to
each model, but employing a list of models is more convenient for both
\(k\)-fold and LOO CV (where there is
no random component to the composition of the \(n\) folds).
We illustrate with the polynomial regression models of varying degree
for the Auto
data (discussed previously), beginning by
fitting and saving the 10 models:
for (p in 1:10){
assign(paste0("m.", p),
lm(mpg ~ poly(horsepower, p), data=Auto))
}
objects(pattern="m\\.[0-9]")
#> [1] "m.1" "m.10" "m.2" "m.3" "m.4" "m.5" "m.6" "m.7" "m.8" "m.9"
summary(m.2) # for example, the quadratic fit
#>
#> Call:
#> lm(formula = mpg ~ poly(horsepower, p), data = Auto)
#>
#> Residuals:
#> Min 1Q Median 3Q Max
#> -14.714 -2.594 -0.086 2.287 15.896
#>
#> Coefficients:
#> Estimate Std. Error t value Pr(>|t|)
#> (Intercept) 23.446 0.221 106.1 <2e-16 ***
#> poly(horsepower, p)1 -120.138 4.374 -27.5 <2e-16 ***
#> poly(horsepower, p)2 44.090 4.374 10.1 <2e-16 ***
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#> Residual standard error: 4.37 on 389 degrees of freedom
#> Multiple R-squared: 0.688, Adjusted R-squared: 0.686
#> F-statistic: 428 on 2 and 389 DF, p-value: <2e-16
We then apply cv()
to the list of 10 models (the
data
argument is required):
# 10-fold CV
cv.auto.10 <- cv(models(m.1, m.2, m.3, m.4, m.5,
m.6, m.7, m.8, m.9, m.10),
data=Auto, seed=2120)
cv.auto.10[1:2] # for the linear and quadratic models
#>
#> Model model.1:
#> 10-Fold Cross Validation
#> method: Woodbury
#> cross-validation criterion = 24.246
#> bias-adjusted cross-validation criterion = 24.23
#> full-sample criterion = 23.944
#>
#> Model model.2:
#> 10-Fold Cross Validation
#> method: Woodbury
#> cross-validation criterion = 19.346
#> bias-adjusted cross-validation criterion = 19.327
#> full-sample criterion = 18.985
# LOO CV
cv.auto.loo <- cv(models(m.1, m.2, m.3, m.4, m.5,
m.6, m.7, m.8, m.9, m.10),
data=Auto, k="loo")
#> Warning in cv.lm(model[[i]], data = data, criterion = criterion, k = k, : seed
#> ignored for n-fold CV
#> Warning in cv.lm(model[[i]], data = data, criterion = criterion, k = k, : seed
#> ignored for n-fold CV
#> Warning in cv.lm(model[[i]], data = data, criterion = criterion, k = k, : seed
#> ignored for n-fold CV
#> Warning in cv.lm(model[[i]], data = data, criterion = criterion, k = k, : seed
#> ignored for n-fold CV
#> Warning in cv.lm(model[[i]], data = data, criterion = criterion, k = k, : seed
#> ignored for n-fold CV
#> Warning in cv.lm(model[[i]], data = data, criterion = criterion, k = k, : seed
#> ignored for n-fold CV
#> Warning in cv.lm(model[[i]], data = data, criterion = criterion, k = k, : seed
#> ignored for n-fold CV
#> Warning in cv.lm(model[[i]], data = data, criterion = criterion, k = k, : seed
#> ignored for n-fold CV
#> Warning in cv.lm(model[[i]], data = data, criterion = criterion, k = k, : seed
#> ignored for n-fold CV
#> Warning in cv.lm(model[[i]], data = data, criterion = criterion, k = k, : seed
#> ignored for n-fold CV
cv.auto.loo[1:2] # linear and quadratic models
#>
#> Model model.1:
#> n-Fold Cross Validation
#> method: hatvalues
#> cross-validation criterion = 24.232
#> Model model.2:
#> n-Fold Cross Validation
#> method: hatvalues
#> cross-validation criterion = 19.248
Because we didn’t supply names for the models in the calls to the
models()
function, the names model.1
,
model.2
, etc., are supplied by the function.
Finally, we extract and graph the adjusted MSEs for \(10\)-fold CV and the MSEs for LOO CV:
cv.mse.10 <- sapply(cv.auto.10, function(x) x[["adj CV crit"]])
cv.mse.loo <- sapply(cv.auto.loo, function(x) x[["CV crit"]])
plot(c(1, 10), range(cv.mse.10, cv.mse.loo), type="n",
xlab="Degree of polynomial, p",
ylab="Cross-Validated MSE")
lines(1:10, cv.mse.10, lwd=2, lty=1, col=2, pch=16, type="b")
lines(1:10, cv.mse.loo, lwd=2, lty=2, col=3, pch=17, type="b")
legend("topright", inset=0.02,
legend=c("10-Fold CV", "LOO CV"),
lwd=2, lty=2:1, col=3:2, pch=17:16)
Cross-validated 10-fold and LOO MSE as a function of polynomial degree, \(p\)
Alternatively, we can use the plot()
method for
"cvModList"
objects to compare the models, though with
separate graphs for 10-fold and LOO CV:
plot(cv.auto.10, main="Polynomial Regressions, 10-Fold CV",
axis.args=list(labels=1:10), xlab="Degree of Polynomial, p")
plot(cv.auto.loo, main="Polynomial Regressions, LOO CV",
axis.args=list(labels=1:10), xlab="Degree of Polynomial, p")
Cross-validated 10-fold and LOO MSE as a function of polynomial degree, \(p\)
In this example, 10-fold and LOO CV produce generally similar results, and also results that are similar to those produced by the estimated error variance \(\widehat{\sigma}^2\) for each model, reported above (except for the highest-degree polynomials, where the CV results more clearly suggest over-fitting).
Mroz
dataThe Mroz
data set from the carData
package (associated with Fox & Weisberg,
2019) has been used by several authors to illustrate binary
logistic regression; see, in particular Fox &
Weisberg (2019). The data were originally drawn from the U.S.
Panel Study of Income Dynamics and pertain to married women. Here are a
few cases in the data set:
data("Mroz", package="carData")
head(Mroz, 3)
#> lfp k5 k618 age wc hc lwg inc
#> 1 yes 1 0 32 no no 1.2102 10.91
#> 2 yes 0 2 30 no no 0.3285 19.50
#> 3 yes 1 3 35 no no 1.5141 12.04
tail(Mroz, 3)
#> lfp k5 k618 age wc hc lwg inc
#> 751 no 0 0 43 no no 0.88814 9.952
#> 752 no 0 0 60 no no 1.22497 24.984
#> 753 no 0 3 39 no no 0.85321 28.363
The response variable in the logistic regression is lfp
,
labor-force participation, a factor coded "yes"
or
"no"
. The remaining variables are predictors:
k5
, number of children 5 years old of younger in the
woman’s household;k618
, number of children between 6 and 18 years
old;age
, in years;wc
, wife’s college attendance, "yes"
or
"no"
;hc
, husband’s college attendance;lwg
, the woman’s log wage rate if she is employed, or
her imputed wage rate, if she is not (a
variable that Fox & Weisberg, 2019 show is problematically
defined); andinc
, family income, in $1000s, exclusive of wife’s
income.We use the glm()
function to fit a binary logistic
regression to the Mroz
data:
m.mroz <- glm(lfp ~ ., data=Mroz, family=binomial)
summary(m.mroz)
#>
#> Call:
#> glm(formula = lfp ~ ., family = binomial, data = Mroz)
#>
#> Coefficients:
#> Estimate Std. Error z value Pr(>|z|)
#> (Intercept) 3.18214 0.64438 4.94 7.9e-07 ***
#> k5 -1.46291 0.19700 -7.43 1.1e-13 ***
#> k618 -0.06457 0.06800 -0.95 0.34234
#> age -0.06287 0.01278 -4.92 8.7e-07 ***
#> wcyes 0.80727 0.22998 3.51 0.00045 ***
#> hcyes 0.11173 0.20604 0.54 0.58762
#> lwg 0.60469 0.15082 4.01 6.1e-05 ***
#> inc -0.03445 0.00821 -4.20 2.7e-05 ***
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#> (Dispersion parameter for binomial family taken to be 1)
#>
#> Null deviance: 1029.75 on 752 degrees of freedom
#> Residual deviance: 905.27 on 745 degrees of freedom
#> AIC: 921.3
#>
#> Number of Fisher Scoring iterations: 4
BayesRule(ifelse(Mroz$lfp == "yes", 1, 0),
fitted(m.mroz, type="response"))
#> [1] 0.30677
In addition to the usually summary output for a GLM, we show the
result of applying the BayesRule()
function from the
cv package to predictions derived from the fitted
model. Bayes rule, which predicts a “success” in a binary regression
model when the fitted probability of success [i.e., \(\phi = \Pr(y = 1)\)] is \(\widehat{\phi} \ge .5\) and a “failure” if
\(\widehat{\phi} \lt .5\).4 The first
argument to BayesRule()
is the binary {0, 1} response, and
the second argument is the predicted probability of success.
BayesRule()
returns the proportion of predictions that are
in error, as appropriate for a “cost” function.
In this example, the fitted logistic regression incorrectly predicts 31% of the responses; we expect this estimate to be optimistic given that the model is used to “predict” the data to which it is fit.
The "glm"
method for cv()
is largely
similar to the "lm"
method, although the default algorithm,
selected explicitly by method="exact"
, refits the model
with each fold removed (and is thus equivalent to
method="naive"
for "lm"
models). For
generalized linear models, method="Woodbury"
or (for LOO
CV) method="hatvalues"
provide approximate results (see the
last section of the vignette for details):
cv(m.mroz, criterion=BayesRule, seed=248)
#> R RNG seed set to 248
#> 10-Fold Cross Validation
#> method: exact
#> criterion: BayesRule
#> cross-validation criterion = 0.32404
#> bias-adjusted cross-validation criterion = 0.31952
#> full-sample criterion = 0.30677
cv(m.mroz, criterion=BayesRule, seed=248, method="Woodbury")
#> R RNG seed set to 248
#> 10-Fold Cross Validation
#> method: Woodbury
#> criterion: BayesRule
#> cross-validation criterion = 0.32404
#> bias-adjusted cross-validation criterion = 0.31926
#> full-sample criterion = 0.30677
To ensure that the two methods use the same 10 folds, we specify the
seed for R’s random-number generator explicitly; here, and as is common
in our experience, the "exact"
and "Woodbury"
algorithms produce nearly identical results. The CV estimates of
prediction error are slightly higher than the estimate based on all of
the cases.
Here are results of applying LOO CV to the Mroz model, using both the exact and the approximate methods:
cv(m.mroz, k="loo", criterion=BayesRule)
#> n-Fold Cross Validation
#> method: exact
#> criterion: BayesRule
#> cross-validation criterion = 0.32005
#> bias-adjusted cross-validation criterion = 0.3183
#> full-sample criterion = 0.30677
cv(m.mroz, k="loo", criterion=BayesRule, method="Woodbury")
#> n-Fold Cross Validation
#> method: Woodbury
#> criterion: BayesRule
#> cross-validation criterion = 0.32005
#> bias-adjusted cross-validation criterion = 0.3183
#> full-sample criterion = 0.30677
cv(m.mroz, k="loo", criterion=BayesRule, method="hatvalues")
#> n-Fold Cross Validation
#> method: hatvalues
#> criterion: BayesRule
#> cross-validation criterion = 0.32005
To the number of decimal digits shown, the three methods produce identical results for this example.
As for linear models, we report some timings for the various
cv()
methods of computation in LOO CV as well as for the
cv.glm()
function from the boot package
(which, recall, refits the model with each case removed, and thus is
comparable to cv()
with method="exact"
):
microbenchmark::microbenchmark(
hatvalues=cv(m.mroz, k="loo", criterion=BayesRule, method="hatvalues"),
Woodbury=cv(m.mroz, k="loo", criterion=BayesRule, method="Woodbury"),
exact=cv(m.mroz, k="loo", criterion=BayesRule),
cv.glm=boot::cv.glm(Mroz, m.mroz,
cost=BayesRule),
times=10)
#> Unit: milliseconds
#> expr min lq mean median uq max neval
#> hatvalues 1.2892 1.3212 1.3524 1.3414 1.3998 1.4421 10
#> Woodbury 39.0116 40.4098 41.0037 40.9399 41.5998 44.2752 10
#> exact 1750.1111 1753.7966 1778.0720 1757.7403 1802.9704 1855.2706 10
#> cv.glm 2008.0623 2038.1417 2081.1113 2090.7808 2106.6058 2137.6914 10
#> cld
#> a
#> b
#> c
#> d
There is a substantial time penalty associated with exact computations.
The fundamental analogy for cross-validation is to the collection of new data. That is, predicting the response in each fold from the model fit to data in the other folds is like using the model fit to all of the data to predict the response for new cases from the values of the predictors for those new cases. As we explained, the application of this idea to independently sampled cases is straightforward—simply partition the data into random folds of equal size and leave each fold out in turn, or, in the case of LOO CV, simply omit each case in turn.
In contrast, mixed-effects models are fit to dependent data, in which cases as clustered, such as hierarchical data, where the clusters comprise higher-level units (e.g., students clustered in schools), or longitudinal data, where the clusters are individuals and the cases repeated observations on the individuals over time.5
We can think of two approaches to applying cross-validation to clustered data:
Treat CV as analogous to predicting the response for one or more cases in a newly observed cluster. In this instance, the folds comprise one or more whole clusters; we refit the model with all of the cases in clusters in the current fold removed; and then we predict the response for the cases in clusters in the current fold. These predictions are based only on fixed effects because the random effects for the omitted clusters are presumably unknown, as they would be for data on cases in newly observed clusters.
Treat CV as analogous to predicting the response for a newly observed case in an existing cluster. In this instance, the folds comprise one or more individual cases, and the predictions can use both the fixed and random effects.
Following their use by Raudenbush & Bryk (2002), data from the 1982 High School and Beyond (HSB) survey have become a staple of the literature on mixed-effects models. The HSB data are used by Fox & Weisberg (2019, sec. 7.2.2) to illustrate the application of linear mixed models to hierarchical data, and we’ll closely follow their example here.
The HSB data are included in the MathAchieve
and
MathAchSchool
data sets in the nlme
package (Pinheiro & Bates, 2000).
MathAchieve
includes individual-level data on 7185 students
in 160 high schools, and MathAchSchool
includes
school-level data:
data("MathAchieve", package="nlme")
dim(MathAchieve)
#> [1] 7185 6
head(MathAchieve, 3)
#> Grouped Data: MathAch ~ SES | School
#> School Minority Sex SES MathAch MEANSES
#> 1 1224 No Female -1.528 5.876 -0.428
#> 2 1224 No Female -0.588 19.708 -0.428
#> 3 1224 No Male -0.528 20.349 -0.428
tail(MathAchieve, 3)
#> Grouped Data: MathAch ~ SES | School
#> School Minority Sex SES MathAch MEANSES
#> 7183 9586 No Female 1.332 19.641 0.627
#> 7184 9586 No Female -0.008 16.241 0.627
#> 7185 9586 No Female 0.792 22.733 0.627
data("MathAchSchool", package="nlme")
dim(MathAchSchool)
#> [1] 160 7
head(MathAchSchool, 2)
#> School Size Sector PRACAD DISCLIM HIMINTY MEANSES
#> 1224 1224 842 Public 0.35 1.597 0 -0.428
#> 1288 1288 1855 Public 0.27 0.174 0 0.128
tail(MathAchSchool, 2)
#> School Size Sector PRACAD DISCLIM HIMINTY MEANSES
#> 9550 9550 1532 Public 0.45 0.791 0 0.059
#> 9586 9586 262 Catholic 1.00 -2.416 0 0.627
The first few students are in school number 1224 and the last few in school 9586.
We’ll use only the School
, SES
(students’
socioeconomic status), and MathAch
(their score on a
standardized math-achievement test) variables in the
MathAchieve
data set, and Sector
("Catholic"
or "Public"
) in the
MathAchSchool
data set.
Some data-management is required before fitting a mixed-effects model to the HSB data, for which we use the dplyr package (Wickham, François, Henry, Müller, & Vaughan, 2023):
library("dplyr")
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#> filter, lag
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, setequal, union
MathAchieve %>% group_by(School) %>%
summarize(mean.ses = mean(SES)) -> Temp
Temp <- merge(MathAchSchool, Temp, by="School")
HSB <- merge(Temp[, c("School", "Sector", "mean.ses")],
MathAchieve[, c("School", "SES", "MathAch")], by="School")
names(HSB) <- tolower(names(HSB))
HSB$cses <- with(HSB, ses - mean.ses)
In the process, we created two new school-level variables:
meanses
, which is the average SES for students in each
school; and cses
, which is school-average SES centered at
its mean. For details, see Fox & Weisberg
(2019, sec. 7.2.2).
Still following Fox and Weisberg, we proceed to use the
lmer()
function in the lme4 package (Bates, Mächler, Bolker, & Walker, 2015) to
fit a mixed model for math achievement to the HSB data:
library("lme4")
#> Loading required package: Matrix
hsb.lmer <- lmer(mathach ~ mean.ses*cses + sector*cses
+ (cses | school), data=HSB)
summary(hsb.lmer, correlation=FALSE)
#> Linear mixed model fit by REML ['lmerMod']
#> Formula: mathach ~ mean.ses * cses + sector * cses + (cses | school)
#> Data: HSB
#>
#> REML criterion at convergence: 46504
#>
#> Scaled residuals:
#> Min 1Q Median 3Q Max
#> -3.159 -0.723 0.017 0.754 2.958
#>
#> Random effects:
#> Groups Name Variance Std.Dev. Corr
#> school (Intercept) 2.380 1.543
#> cses 0.101 0.318 0.39
#> Residual 36.721 6.060
#> Number of obs: 7185, groups: school, 160
#>
#> Fixed effects:
#> Estimate Std. Error t value
#> (Intercept) 12.128 0.199 60.86
#> mean.ses 5.333 0.369 14.45
#> cses 2.945 0.156 18.93
#> sectorCatholic 1.227 0.306 4.00
#> mean.ses:cses 1.039 0.299 3.48
#> cses:sectorCatholic -1.643 0.240 -6.85
We can then cross-validate at the cluster (i.e., school) level,
cv(hsb.lmer, k=10, clusterVariables="school", seed=5240)
#> R RNG seed set to 5240
#> 10-Fold Cross Validation based on 160 {school} clusters
#> cross-validation criterion = 39.133
#> bias-adjusted cross-validation criterion = 39.125
#> full-sample criterion = 39.006
or at the case (i.e., student) level,
cv(hsb.lmer, seed=1575)
#> R RNG seed set to 1575
#> Warning in checkConv(attr(opt, "derivs"), opt$par, ctrl = control$checkConv, :
#> Model failed to converge with max|grad| = 0.00587228 (tol = 0.002, component 1)
#> boundary (singular) fit: see help('isSingular')
#> 10-Fold Cross Validation
#> cross-validation criterion = 37.445
#> bias-adjusted cross-validation criterion = 37.338
#> full-sample criterion = 36.068
For cluster-level CV, the clusterVariables
argument
tells cv()
how the clusters are defined. Were there more
than one clustering variable, say classes within schools, these would be
provided as a character vector of variable names:
clusterVariables = c("school", "class")
. For cluster-level
CV, the default is k = "loo"
, that is, leave one cluster
out at a time; we instead specify k = 10
folds of clusters,
each fold therefore comprising \(160/10 =
16\) schools.
If the clusterVariables
argument is omitted, then
case-level CV is employed, with k = 10
folds as the
default, here each with \(7185/10 \approx
719\) students. Notice that one of the 10 models refit with a
fold removed failed to converge. Convergence problems are common in
mixed-effects modeling. The apparent issue here is that an estimated
variance component is close to or equal to 0, which is at a boundary of
the parameter space. That shouldn’t disqualify the fitted model for the
kind of prediction required for cross-validation.
There is also a cv()
method for linear mixed models fit
by the lme()
function in the nlme package,
and the arguments for cv()
in this case are the same as for
a model fit by lmer()
or glmer()
. We
illustrate with the mixed model fit to the HSB data:
library("nlme")
#>
#> Attaching package: 'nlme'
#> The following object is masked from 'package:lme4':
#>
#> lmList
#> The following object is masked from 'package:dplyr':
#>
#> collapse
#> The following object is masked from 'package:cv':
#>
#> getResponse
hsb.lme <- lme(mathach ~ mean.ses*cses + sector*cses,
random = ~ cses | school, data=HSB,
control=list(opt="optim"))
summary(hsb.lme)
#> Linear mixed-effects model fit by REML
#> Data: HSB
#> AIC BIC logLik
#> 46525 46594 -23252
#>
#> Random effects:
#> Formula: ~cses | school
#> Structure: General positive-definite, Log-Cholesky parametrization
#> StdDev Corr
#> (Intercept) 1.541177 (Intr)
#> cses 0.018174 0.006
#> Residual 6.063492
#>
#> Fixed effects: mathach ~ mean.ses * cses + sector * cses
#> Value Std.Error DF t-value p-value
#> (Intercept) 12.1282 0.19920 7022 60.886 0e+00
#> mean.ses 5.3367 0.36898 157 14.463 0e+00
#> cses 2.9421 0.15122 7022 19.456 0e+00
#> sectorCatholic 1.2245 0.30611 157 4.000 1e-04
#> mean.ses:cses 1.0444 0.29107 7022 3.588 3e-04
#> cses:sectorCatholic -1.6421 0.23312 7022 -7.044 0e+00
#> Correlation:
#> (Intr) men.ss cses sctrCt mn.ss:
#> mean.ses 0.256
#> cses 0.000 0.000
#> sectorCatholic -0.699 -0.356 0.000
#> mean.ses:cses 0.000 0.000 0.295 0.000
#> cses:sectorCatholic 0.000 0.000 -0.696 0.000 -0.351
#>
#> Standardized Within-Group Residuals:
#> Min Q1 Med Q3 Max
#> -3.170106 -0.724877 0.014892 0.754263 2.965498
#>
#> Number of Observations: 7185
#> Number of Groups: 160
cv(hsb.lme, k=10, clusterVariables="school", seed=5240)
#> R RNG seed set to 5240
#> 10-Fold Cross Validation based on 160 {school} clusters
#> cross-validation criterion = 39.133
#> bias-adjusted cross-validation criterion = 39.125
#> full-sample criterion = 39.006
cv(hsb.lme, seed=1575)
#> R RNG seed set to 1575
#> 10-Fold Cross Validation
#> cross-validation criterion = 37.442
#> bias-adjusted cross-validation criterion = 37.402
#> full-sample criterion = 36.147
We used the same random-number generator seeds as in the previous
example cross-validating the model fit by lmer()
, and so
the same folds are employed in both cases.6 The estimated
covariance components and fixed effects in the summary output differ
slightly between the lmer()
and lme()
solutions, although both functions seek to maximize the REML criterion.
This is, of course, to be expected when different algorithms are used
for numerical optimization. To the precision reported, the cluster-level
CV results for the lmer()
and lme()
models are
identical, while the case-level CV results are very similar but not
identical.
We introduce an artificial data set that exemplifies aspects of cross-validation particular to hierarchical models. Using this data set, we show that model comparisons employing cluster-based and those employing case-based cross-validation may not agree on a “best” model. Furthermore, commonly used measures of fit, such as mean-squared error, do not necessarily become smaller as models become larger, even when the models are nested, and even when the measure of fit is computed for the whole data set.
Consider a researcher studying improvement in a skill, yodeling, for example, among students enrolled in a four-year yodeling program. The plan is to measure each student’s skill level at the beginning of the program and every year thereafter until the end of the program, resulting in five annual measurements for each student. It turns out that yodeling appeals to students of all ages, and students enrolling in the program range in age from 20 to 70. Moreover, participants’ untrained yodeling skill is similar at all ages, as is their rate of progress with training. All students complete the four-year program.
The researcher, who has more expertise in yodeling than in modeling, decides to model the response, \(y\), yodeling skill, as a function of age, \(x\), reasoning that students get older during their stay in the program, and (incorrectly) that age can serve as a proxy for elapsed time. The researcher knows that a mixed model should be used to account for clustering due to the expected similarity of measurements taken from each student.
We start by generating the data, using parameters consistent with the description above and meant to highlight the issues that arise in cross-validating mixed-effects models:7
# Parameters:
set.seed(9693)
Nb <- 100 # number of groups
Nw <- 5 # number of individuals within groups
Bb <- 0 # between-group regression coefficient on group mean
SDre <- 2.0 # between-group SD of random level relative to group mean of x
SDwithin <- 0.5 # within group SD
Bw <- 1 # within group effect of x
Ay <- 10 # intercept for response
Ax <- 20 # starting level of x
Nx <- Nw*10 # number of distinct x values
Data <- data.frame(
group = factor(rep(1:Nb, each=Nw)),
x = Ax + rep(1:Nx, length.out = Nw*Nb)
) |>
within(
{
xm <- ave(x, group, FUN = mean) # within-group mean
y <- Ay +
Bb * xm + # contextual effect
Bw * (x - xm) + # within-group effect
rnorm(Nb, sd=SDre)[group] + # random level by group
rnorm(Nb*Nw, sd=SDwithin) # random error within groups
}
)
Here is a scatterplot of the data for a representative group of 10 (without loss of generality, the first 10) of 100 students, showing the 95% concentration ellipse for each cluster:8
library("lattice")
library("latticeExtra")
plot <- xyplot(y ~ x, data=Data[1:Nx, ], group=group,
ylim=c(4, 16),
par.settings=list(superpose.symbol=list(pch=1, cex=0.7))) +
layer(panel.ellipse(..., center.cex=0))
plot # display graph
Hierarchical data set, showing the first 10 of 100 students.
The between-student effect of age is 0 but the within-student effect is 1. Due to the large variation in ages between students, the least-squares regression of yodeling skill on age (for the 500 observations among all 100 students) produces an estimated slope close to 0 (though with a small \(p\)-value), because the slope is heavily weighted toward the between-student effect:
summary(lm(y ~ x, data=Data))
#>
#> Call:
#> lm(formula = y ~ x, data = Data)
#>
#> Residuals:
#> Min 1Q Median 3Q Max
#> -5.771 -1.658 -0.089 1.552 7.624
#>
#> Coefficients:
#> Estimate Std. Error t value Pr(>|t|)
#> (Intercept) 9.05043 0.34719 26.07 <2e-16 ***
#> x 0.02091 0.00727 2.87 0.0042 **
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#> Residual standard error: 2.35 on 498 degrees of freedom
#> Multiple R-squared: 0.0163, Adjusted R-squared: 0.0143
#> F-statistic: 8.26 on 1 and 498 DF, p-value: 0.00422
The initial mixed-effects model that we fit to the data is a simple random-intercepts model:
# random intercept only:
mod.0 <- lmer(y ~ 1 + (1 | group), Data)
summary(mod.0)
#> Linear mixed model fit by REML ['lmerMod']
#> Formula: y ~ 1 + (1 | group)
#> Data: Data
#>
#> REML criterion at convergence: 2103.1
#>
#> Scaled residuals:
#> Min 1Q Median 3Q Max
#> -2.0351 -0.7264 -0.0117 0.7848 2.0438
#>
#> Random effects:
#> Groups Name Variance Std.Dev.
#> group (Intercept) 2.90 1.70
#> Residual 2.71 1.65
#> Number of obs: 500, groups: group, 100
#>
#> Fixed effects:
#> Estimate Std. Error t value
#> (Intercept) 10.002 0.186 53.9
We will shortly consider three other, more complex, mixed models; because of data-management considerations, it is convenient to fit them now, but we defer discussion of these models:
# effect of x and random intercept:
mod.1 <- lmer(y ~ x + (1 | group), Data)
# effect of x, contextual (student) mean of x, and random intercept:
mod.2 <- lmer(y ~ x + xm + (1 | group), Data)
# equivalent to y ~ I(x - xm) + xm + (1 | group)
# model generating the data (where Bb = 0)
mod.3 <- lmer(y ~ I(x - xm) + (1 | group), Data)
We proceed to obtain predictions from the random-intercept model
(mod.0
) and the other models (mod.1
,
mod.2
, and mod.3
) based on fixed effects
alone, as would be used for cross-validation based on clusters (i.e.,
students), and for fixed and random effects—so-called best linear
unbiased predictions or BLUPs—as would be used for cross-validation
based on cases (i.e., occasions within students):
Data <- within(Data, {
fit_mod0.fe <- predict(mod.0, re.form = ~ 0) # fixed effects only
fit_mod0.re <- predict(mod.0) # fixed and random effects (BLUPs)
fit_mod1.fe <- predict(mod.1, re.form = ~ 0)
fit_mod1.re <- predict(mod.1)
fit_mod2.fe <- predict(mod.2, re.form = ~ 0)
fit_mod2.re <- predict(mod.2)
fit_mod3.fe <- predict(mod.3, re.form = ~ 0)
fit_mod3.re <- predict(mod.3)
})
We then prepare the data for plotting:
Data_long <- reshape(Data[1:Nx, ], direction = "long", sep = ".",
timevar = "effect", varying = grep("\\.", names(Data[1:Nx, ])))
Data_long$id <- 1:nrow(Data_long)
Data_long <- reshape(Data_long, direction = "long", sep = "_",
timevar = "modelcode", varying = grep("_", names(Data_long)))
Data_long$model <- factor(
c("~ 1", "~ 1 + x", "~ 1 + x + xm", "~ 1 + I(x - xm)")
[match(Data_long$modelcode, c("mod0", "mod1", "mod2", "mod3"))]
)
Predictions based on the random-intercept model mod.0
for the first 10 students are shown in the following graph:
(plot +
xyplot(fit ~ x, subset(Data_long, modelcode == "mod0" & effect == "fe"),
groups=group, type="l", lwd=2) +
xyplot(fit ~ x, subset(Data_long, modelcode == "mod0" & effect == "re"),
groups=group, type="l", lwd=2, lty=3)
) |> update(
main="Model: y ~ 1 + (1 | group)",
key=list(
corner=c(0.05, 0.05),
text=list(c("fixed effects only","fixed and random")),
lines=list(lty=c(1, 3))))
Predictions from the random intercept model.
The fixed-effect predictions for the various individuals are identical—the estimated fixed-effects intercept or estimated general mean of \(y\)—while the BLUPs are the sums of the fixed-effects intercept and the random intercepts, and are only slightly shrunken towards the general mean. Because in our artificial data there is no population relationship between age and skill, the fixed-effect-only predictions and the BLUPs are not very different.
Our next model, mod.1
, includes a fixed intercept and
fixed effect of x
along with a random intercept:
summary(mod.1)
#> Linear mixed model fit by REML ['lmerMod']
#> Formula: y ~ x + (1 | group)
#> Data: Data
#>
#> REML criterion at convergence: 1564.5
#>
#> Scaled residuals:
#> Min 1Q Median 3Q Max
#> -2.9016 -0.6350 0.0188 0.5541 2.8293
#>
#> Random effects:
#> Groups Name Variance Std.Dev.
#> group (Intercept) 192.941 13.890
#> Residual 0.257 0.507
#> Number of obs: 500, groups: group, 100
#>
#> Fixed effects:
#> Estimate Std. Error t value
#> (Intercept) -33.9189 1.5645 -21.7
#> x 0.9653 0.0158 61.0
#>
#> Correlation of Fixed Effects:
#> (Intr)
#> x -0.460
Predictions from this model appear in the following graph:
(plot +
xyplot(fit ~ x, subset(Data_long, modelcode == "mod1" & effect == "fe"),
groups=group, type="l", lwd=2) +
xyplot(fit ~ x, subset(Data_long, modelcode == "mod1" & effect == "re"),
groups=group, type="l", lwd=2, lty=3)
) |> update(
main="Model: y ~ 1 + x + (1 | group)",
ylim=c(-15, 35),
key=list(
corner=c(0.95, 0.05),
text=list(c("fixed effects only","fixed and random")),
lines=list(lty=c(1, 3))))
Predictions from the model with random intercepts and \(x\) as a fixed-effect predictor.
The BLUPs fit the observed data very closely, but predictions based
on the fixed effects alone, with a common intercept and slope for all
clusters, are very poor—indeed, much worse than the fixed-effects-only
predictions based on the simpler random-intercept model,
mod.0
. We therefore anticipate (and show later in this
section) that case-based cross-validation will prefer mod1
to mod0
, but that cluster-based cross-validation will
prefer mod0
to mod1
.
Our third model, mod.2
, includes the contextual effect
of \(x\)—that is, the cluster mean
xm
—along with \(x\) and
the intercept in the fixed-effect part of the model, and a random
intercept:
summary(mod.2)
#> Linear mixed model fit by REML ['lmerMod']
#> Formula: y ~ x + xm + (1 | group)
#> Data: Data
#>
#> REML criterion at convergence: 1169.2
#>
#> Scaled residuals:
#> Min 1Q Median 3Q Max
#> -2.9847 -0.6375 0.0019 0.5568 2.7325
#>
#> Random effects:
#> Groups Name Variance Std.Dev.
#> group (Intercept) 3.399 1.844
#> Residual 0.255 0.505
#> Number of obs: 500, groups: group, 100
#>
#> Fixed effects:
#> Estimate Std. Error t value
#> (Intercept) 9.4787 0.6171 15.4
#> x 0.9915 0.0160 62.1
#> xm -0.9800 0.0206 -47.7
#>
#> Correlation of Fixed Effects:
#> (Intr) x
#> x 0.000
#> xm -0.600 -0.777
This model is equivalent to fitting
y ~ I(x - xm) + xm + (1 | group)
, which is the model that
generated the data once the coefficient of the contextual predictor
xm
is set to 0 (as it is in mod.3
, discussed
below).
Predictions from model mod.2
appear in the following
graph:
(plot +
xyplot(fit ~ x, subset(Data_long, modelcode == "mod2" & effect == "fe"),
groups=group, type="l", lwd=2) +
xyplot(fit ~ x, subset(Data_long, modelcode == "mod2" & effect == "re"),
groups=group, type="l", lwd=2, lty=3)
) |> update(
main="Model: y ~ 1 + x + xm + (1 | group)",
ylim=c(4, 16),
key=list(
corner=c(0.05, 0.05),
text=list(c("fixed effects only","fixed and random")),
lines=list(lty=c(1, 3))))
Predictors from the model with random intercepts, \(x\), and the group (student) mean of \(x\) as predictors.
Depending on the estimated variance parameters of the model, a mixed
model like mod.2
will apply varying degrees of shrinkage to
the random-intercept BLUPs that correspond to variation in the heights
of the parallel fitted lines for the individual students. In our
contrived data, the mod.2
applies little shrinkage,
allowing substantial variability in the heights of the fitted lines,
which closely approach the observed values for each student. The fit of
the mixed model mod.2
is consequently similar to that of a
fixed-effects model with age and a categorical predictor for individual
students (i.e., treating students as a factor, and not shown here).
The mixed model mod.2
therefore fits individual
observations well, and we anticipate a favorable assessment using
individual-based cross-validation. In contrast, the large variability in
the BLUPs results in larger residuals for predictions based on fixed
effects alone, and so we expect that cluster-based cross-validation
won’t show an advantage for model mod.2
compared to the
smaller model mod.0
, which includes only fixed and random
intercepts.
Had the mixed model applied considerable shrinkage, then neither cluster-based nor case-based cross-validation would show much improvement over the random-intercept-only model. In our experience, the degree of shrinkage does not vary smoothly as parameters are changed but tends to be “all or nothing,” and near the tipping point, the behavior of estimates can be affected considerably by the choice of algorithm used to fit the model.
Finally, mod.3
directly estimates the model used to
generate the data. As mentioned, it is a constrained version of
mod.2
, with the coefficient of xm
set to 0,
and with x
expressed as a deviation from the cluster mean
xm
:
summary(mod.3)
#> Linear mixed model fit by REML ['lmerMod']
#> Formula: y ~ I(x - xm) + (1 | group)
#> Data: Data
#>
#> REML criterion at convergence: 1163.2
#>
#> Scaled residuals:
#> Min 1Q Median 3Q Max
#> -2.9770 -0.6320 0.0063 0.5603 2.7249
#>
#> Random effects:
#> Groups Name Variance Std.Dev.
#> group (Intercept) 3.391 1.842
#> Residual 0.255 0.505
#> Number of obs: 500, groups: group, 100
#>
#> Fixed effects:
#> Estimate Std. Error t value
#> (Intercept) 10.002 0.185 53.9
#> I(x - xm) 0.992 0.016 62.1
#>
#> Correlation of Fixed Effects:
#> (Intr)
#> I(x - xm) 0.000
The predictions from mod.3
are therefore similar to
those from mod.2
:
(plot +
xyplot(fit ~ x, subset(Data_long, modelcode == "mod3" & effect == "fe"),
groups=group, type="l", lwd=2) +
xyplot(fit ~ x, subset(Data_long, modelcode == "mod3" & effect == "re"),
groups=group, type="l", lwd=2, lty=3)
) |> update(
main="Model: y ~ 1 + I(x - xm) + (1 | group)",
ylim=c(4, 16),
key=list(
corner=c(0.05, 0.05),
text=list(c("fixed effects only","fixed and random")),
lines=list(lty=c(1, 3))))
Predictions from the estimated model generating the data.
We next carry out case-based cross-validation, which, as we have
explained, is based on both fixed and predicted random effects (i.e.,
BLUPs), and cluster-based cross-validation, which is based on fixed
effects only. In order to reduce between-model random variability in
comparisons of models, we apply cv()
to the list of models
created by the models()
function (introduced previously),
performing cross-validation with the same folds for each model:
modlist <- models("~ 1"=mod.0, "~ 1 + x"=mod.1,
"~ 1 + x + xm"=mod.2, "~ 1 + I(x - xm)"=mod.3)
cvs_clusters <- cv(modlist, data=Data, cluster="group", k=10, seed=6449)
plot(cvs_clusters, main="Model Comparison, Cluster-Based CV")
10-fold cluster-based cross-validation comparing random intercept models with varying fixed effects.
cvs_cases <- cv(modlist, data=Data, seed=9693)
plot(cvs_cases, main="Model Comparison, Case-Based CV")
10-fold case-based cross-validation comparing random intercept models with varying fixed effects.
In summary, model mod.1
, with \(x\) alone and without the contextual mean
of \(x\), is assessed as fitting very
poorly by cluster-based CV, but relatively much better by case-based CV.
Model mod.2
, which includes both \(x\) and its contextual mean, produces
better results using both cluster-based and case-based CV. The
data-generating model, mod.3
, which includes the fixed
effect of x - xm
in place of separate terms in
x
and xm
, isn’t distinguishable from model
mod.2
, which includes x
and xm
separately, even though mod.2
has an unnecessary parameter
(recall that the population coefficient of xm
is 0 when
x
is expressed as deviations from the contextual mean).
These conclusions are consistent with our observations based on graphing
predictions from the various models, and they illustrate the
desirability of assessing mixed-effect models at different hierarchical
levels.
Assuming that the number of cases \(n\) is a multiple of the number of folds \(k\)—a slightly simplifying assumption—the number of possible partitions of cases into folds is \(\frac{n!}{[(n/k)!]^k}\), a number that grows very large very quickly. For example, for \(n = 10\) and \(k = 5\), so that the folds are each of size \(n/k = 2\), there are \(113,400\) possible partitions; for \(n=100\) and \(k=5\), where \(n/k = 20\), still a small problem, the number of possible partitions is truly astronomical, \(1.09\times 10^{66}\).
Because the partition into folds that’s employed is selected
randomly, the resulting CV criterion estimates are subject to sampling
error. (An exception is LOO cross-validation, which is not at all
random.) To get a sense of the magnitude of the sampling error, we can
repeat the CV procedure with different randomly selected partitions into
folds. All of the CV functions in the cv package are
capable of repeated cross-validation, with the number of repetitions
controlled by the reps
argument, which defaults to
1
.
Here, for example, is 10-fold CV for the Mroz logistic regression, repeated 5 times:
cv(m.mroz, criterion=BayesRule, seed=248, reps=5,
method="Woodbury")
#> R RNG seed set to 248
#> R RNG seed set to 68134
#> R RNG seed set to 767359
#> R RNG seed set to 556270
#> R RNG seed set to 882966
#>
#> Replicate 1:
#> 10-Fold Cross Validation
#> method: Woodbury
#> criterion: BayesRule
#> cross-validation criterion = 0.32005
#> bias-adjusted cross-validation criterion = 0.31301
#> full-sample criterion = 0.30677
#>
#> Replicate 2:
#> 10-Fold Cross Validation
#> method: Woodbury
#> criterion: BayesRule
#> cross-validation criterion = 0.31607
#> bias-adjusted cross-validation criterion = 0.3117
#> full-sample criterion = 0.30677
#>
#> Replicate 3:
#> 10-Fold Cross Validation
#> method: Woodbury
#> criterion: BayesRule
#> cross-validation criterion = 0.31474
#> bias-adjusted cross-validation criterion = 0.30862
#> full-sample criterion = 0.30677
#>
#> Replicate 4:
#> 10-Fold Cross Validation
#> method: Woodbury
#> criterion: BayesRule
#> cross-validation criterion = 0.32404
#> bias-adjusted cross-validation criterion = 0.31807
#> full-sample criterion = 0.30677
#>
#> Replicate 5:
#> 10-Fold Cross Validation
#> method: Woodbury
#> criterion: BayesRule
#> cross-validation criterion = 0.32404
#> bias-adjusted cross-validation criterion = 0.31926
#> full-sample criterion = 0.30677
#>
#> Average:
#> 10-Fold Cross Validation
#> method: Woodbury
#> criterion: BayesRule
#> cross-validation criterion = 0.31983 (0.003887)
#> bias-adjusted cross-validation criterion = 0.31394 (0.0040093)
#> full-sample criterion = 0.30677
When reps
> 1
, the result returned by
cv()
is an object of class "cvList"
—literally
a list of "cv"
objects. The results are reported for each
repetition and then averaged across repetitions, with the standard
deviations of the CV criterion and the biased-adjusted CV criterion
given in parentheses. In this example, there is therefore little
variation across repetitions, increasing our confidence in the
reliability of the results.
Notice that the seed that’s set in the cv()
command
pertains to the first repetition and the seeds for the remaining
repetitions are then selected pseudo-randomly.9 Setting the first
seed, however, makes the entire process easily replicable, and the seed
for each repetition is stored in the corresponding element of the
"cvList"
object (which isn’t, however, saved in the
example).
It’s also possible to replicate CV when comparing competing models
via the cv()
method for "modList"
objects.
Recall our comparison of polynomial regressions of varying degree fit to
the Auto
data; we performed 10-fold CV for each of 10
models. Here, we replicate that process 5 times for each model and graph
the results:
cv.auto.reps <- cv(models(m.1, m.2, m.3, m.4, m.5,
m.6, m.7, m.8, m.9, m.10),
data=Auto, seed=8004, reps=5)
plot(cv.auto.reps)
Replicated cross-validated 10-fold CV as a function of polynomial degree, \(p\)
The graph shows both the average CV criterion and its range for each of the competing models.
As Hastie, Tibshirani, & Friedman (2009, sec. 7.10.2: “The Wrong and Right Way to Do Cross-validation”) explain, if the whole data are used to select or fine-tune a statistical model, subsequent cross-validation of the model is intrinsically misleading, because the model is selected to fit the whole data, including the part of the data that remains when each fold is removed.
The following example is similar in spirit to one employed by Hastie et al. (2009). Suppose that we randomly generate \(n = 1000\) independent observations for a response variable variable \(y \sim N(\mu = 10, \sigma^2 = 0)\), and independently sample \(1000\) observations for \(p = 100\) “predictors,” \(x_1, \ldots, x_{100}\), each from \(x_j \sim N(0, 1)\). The response has nothing to do with the predictors and so the population linear-regression model \(y_i = \alpha + \beta_1 x_{i1} + \cdots + \beta_{100} x_{i,100} + \varepsilon_i\) has \(\alpha = 10\) and all \(\beta_j = 0\).
set.seed(24361) # for reproducibility
D <- data.frame(
y = rnorm(1000, mean=10),
X = matrix(rnorm(1000*100), 1000, 100)
)
head(D[, 1:6])
#> y X.1 X.2 X.3 X.4 X.5
#> 1 10.0316 -1.23886 -0.26487 -0.03539 -2.576973 0.811048
#> 2 9.6650 0.12287 -0.17744 0.37290 -0.935138 0.628673
#> 3 10.0232 -0.95052 -0.73487 -1.05978 0.882944 0.023918
#> 4 8.9910 1.13571 0.32411 0.11037 1.376303 -0.422114
#> 5 9.0712 1.49474 1.87538 0.10575 0.292140 -0.184568
#> 6 11.3493 -0.18453 -0.78037 -1.23804 -0.010949 0.691034
Least-squares provides accurate estimates of the regression constant \(\alpha = 10\) and the error variance \(\sigma^2 = 1\) for the “null model” including only the regression constant; moreover, the omnibus \(F\)-test of the correct null hypothesis that all of the \(\beta\)s are 0 for the “full model” with all 100 \(x\)s is associated with a large \(p\)-value:
m.full <- lm(y ~ ., data=D)
m.null <- lm(y ~ 1, data=D)
anova(m.null, m.full)
#> Analysis of Variance Table
#>
#> Model 1: y ~ 1
#> Model 2: y ~ X.1 + X.2 + X.3 + X.4 + X.5 + X.6 + X.7 + X.8 + X.9 + X.10 +
#> X.11 + X.12 + X.13 + X.14 + X.15 + X.16 + X.17 + X.18 + X.19 +
#> X.20 + X.21 + X.22 + X.23 + X.24 + X.25 + X.26 + X.27 + X.28 +
#> X.29 + X.30 + X.31 + X.32 + X.33 + X.34 + X.35 + X.36 + X.37 +
#> X.38 + X.39 + X.40 + X.41 + X.42 + X.43 + X.44 + X.45 + X.46 +
#> X.47 + X.48 + X.49 + X.50 + X.51 + X.52 + X.53 + X.54 + X.55 +
#> X.56 + X.57 + X.58 + X.59 + X.60 + X.61 + X.62 + X.63 + X.64 +
#> X.65 + X.66 + X.67 + X.68 + X.69 + X.70 + X.71 + X.72 + X.73 +
#> X.74 + X.75 + X.76 + X.77 + X.78 + X.79 + X.80 + X.81 + X.82 +
#> X.83 + X.84 + X.85 + X.86 + X.87 + X.88 + X.89 + X.90 + X.91 +
#> X.92 + X.93 + X.94 + X.95 + X.96 + X.97 + X.98 + X.99 + X.100
#> Res.Df RSS Df Sum of Sq F Pr(>F)
#> 1 999 974
#> 2 899 888 100 85.2 0.86 0.82
summary(m.null)
#>
#> Call:
#> lm(formula = y ~ 1, data = D)
#>
#> Residuals:
#> Min 1Q Median 3Q Max
#> -3.458 -0.681 0.019 0.636 2.935
#>
#> Coefficients:
#> Estimate Std. Error t value Pr(>|t|)
#> (Intercept) 9.9370 0.0312 318 <2e-16 ***
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#> Residual standard error: 0.987 on 999 degrees of freedom
Next, using the stepAIC()
function in the
MASS package (Venables &
Ripley, 2002), let us perform a forward stepwise regression to
select a “best” model, starting with the null model, and using AIC as
the model-selection criterion (see the help page for
stepAIC()
for details):10
m.select <- MASS::stepAIC(m.null,
direction="forward", trace=FALSE,
scope=list(lower=~1, upper=formula(m.full)))
summary(m.select)
#>
#> Call:
#> lm(formula = y ~ X.99 + X.90 + X.87 + X.40 + X.65 + X.91 + X.53 +
#> X.45 + X.31 + X.56 + X.61 + X.60 + X.46 + X.35 + X.92, data = D)
#>
#> Residuals:
#> Min 1Q Median 3Q Max
#> -3.262 -0.645 0.024 0.641 3.118
#>
#> Coefficients:
#> Estimate Std. Error t value Pr(>|t|)
#> (Intercept) 9.9372 0.0310 320.80 <2e-16 ***
#> X.99 -0.0910 0.0308 -2.95 0.0032 **
#> X.90 -0.0820 0.0314 -2.62 0.0090 **
#> X.87 -0.0694 0.0311 -2.24 0.0256 *
#> X.40 -0.0476 0.0308 -1.55 0.1221
#> X.65 -0.0552 0.0315 -1.76 0.0795 .
#> X.91 0.0524 0.0308 1.70 0.0894 .
#> X.53 -0.0492 0.0305 -1.61 0.1067
#> X.45 0.0554 0.0318 1.74 0.0818 .
#> X.31 0.0452 0.0311 1.46 0.1457
#> X.56 0.0543 0.0327 1.66 0.0972 .
#> X.61 -0.0508 0.0317 -1.60 0.1091
#> X.60 -0.0513 0.0319 -1.61 0.1083
#> X.46 0.0516 0.0327 1.58 0.1153
#> X.35 0.0470 0.0315 1.49 0.1358
#> X.92 0.0443 0.0310 1.43 0.1533
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#> Residual standard error: 0.973 on 984 degrees of freedom
#> Multiple R-squared: 0.0442, Adjusted R-squared: 0.0296
#> F-statistic: 3.03 on 15 and 984 DF, p-value: 8.34e-05
mse(D$y, fitted(m.select))
#> [1] 0.93063
The resulting model has 15 predictors, a very modest \(R^2 = .044\), but a small \(p\)-value for its omnibus \(F\)-test (which, of course, is entirely spurious because the same data were used to select and test the model). The MSE for the selected model is smaller than the true error variance \(\sigma^2 = 1\), as is the estimated error variance for the selected model, \(\widehat{\sigma}^2 = 0.973^2 = 0.947\).
If we cross-validate the selected model, we also obtain an optimistic estimate of its predictive power:
cv(m.select, seed=2529)
#> R RNG seed set to 2529
#> 10-Fold Cross Validation
#> method: Woodbury
#> criterion: mse
#> cross-validation criterion = 0.95937
#> bias-adjusted cross-validation criterion = 0.95785
#> full-sample criterion = 0.93063
The cvSelect()
function in the cv
package allows us to cross-validate the whole model-selection procedure.
The first argument to cvSelect()
is a model-selection
function capable of refitting the model with a fold omitted and
returning a CV criterion. The selectStepAIC()
function,
also in cv and based on stepAIC()
, is
suitable for use with cvSelect()
:
cv.select <- cvSelect(selectStepAIC, data=D, seed=3791,
model=m.null, direction="forward",
scope=list(lower=~1,
upper=formula(m.full)))
#> R RNG seed set to 3791
cv.select
#> 10-Fold Cross Validation
#> cross-validation criterion = 1.0687
#> bias-adjusted cross-validation criterion = 1.0612
#> full-sample criterion = 0.93063
The other arguments to cvSelect()
are:
data
, the data set to which the model is fit;seed
, an optional seed for R’s pseudo-random-number
generator; as for cv()
, if the seed isn’t supplied by the
user, a seed is randomly selected and saved;model
argument, the direction
of
model selection, and the scope
of models considered (from
the model with only a regression constant to the model with all 100
predictors).By default, cvSelect()
performs 10-fold CV, and produces
an estimate of MSE for the model-selection procedure even
larger than the true error variance, \(\sigma^2 = 1\).
Also by default, when the number of folds is 10 or fewer,
cvSelect()
saves the coefficients of the selected models.
In this example, the compareFolds()
function reveals that
the variables retained by the model-selection process in the several
folds are quite different:
compareFolds(cv.select)
#> (Intercept) X.87 X.90 X.99 X.91 X.54 X.53 X.56
#> Fold 1 9.9187 -0.0615 -0.0994 -0.0942 0.0512 0.0516
#> Fold 2 9.9451 -0.0745 -0.0899 -0.0614 0.0587 0.0673
#> Fold 3 9.9423 -0.0783 -0.0718 -0.0987 0.0601 0.0512
#> Fold 4 9.9410 -0.0860 -0.0831 -0.0867 0.0570 -0.0508
#> Fold 5 9.9421 -0.0659 -0.0849 -0.1004 0.0701 0.0511 -0.0487 0.0537
#> Fold 6 9.9633 -0.0733 -0.0874 -0.0960 0.0555 0.0629 -0.0478
#> Fold 7 9.9279 -0.0618 -0.0960 -0.0838 0.0533 -0.0464
#> Fold 8 9.9453 -0.0610 -0.0811 -0.0818 0.0497 -0.0612 0.0560
#> Fold 9 9.9173 -0.0663 -0.0894 -0.1100 0.0504 0.0524 0.0747
#> Fold 10 9.9449 -0.0745 -0.0906 -0.0891 0.0535 0.0482 -0.0583 0.0642
#> X.40 X.45 X.65 X.68 X.92 X.15 X.26 X.46 X.60
#> Fold 1 -0.0590 -0.0456 0.0658 0.0608
#> Fold 2 0.0607 0.0487
#> Fold 3 -0.0496 -0.0664 0.0494
#> Fold 4 -0.0597 0.0579 -0.0531 0.0519 -0.0566 -0.0519
#> Fold 5 0.0587 0.0527 -0.0603
#> Fold 6 -0.0596 0.0552 0.0474
#> Fold 7 0.0572 0.0595
#> Fold 8 0.0547 -0.0617 0.0453 0.0493 -0.0613 0.0591 0.0703 -0.0588
#> Fold 9 -0.0552 0.0573 -0.0635 0.0492 -0.0513 0.0484 -0.0507
#> Fold 10 -0.0558 0.0529 0.0710
#> X.61 X.8 X.28 X.29 X.31 X.35 X.70 X.89 X.17
#> Fold 1 -0.0490 0.0616 -0.0537 0.0638
#> Fold 2 0.0671 0.0568 0.0523
#> Fold 3 -0.0631 0.0616
#> Fold 4 0.0659 -0.0549 0.0527 0.0527
#> Fold 5 0.0425 0.0672 0.0613 0.0493
#> Fold 6 0.0559 -0.0629 0.0498 0.0487
#> Fold 7 0.0611 0.0472
#> Fold 8 -0.0719 0.0586
#> Fold 9 0.0525
#> Fold 10 -0.0580 0.0603
#> X.25 X.4 X.64 X.81 X.97 X.11 X.2 X.33 X.47
#> Fold 1 0.0604 0.0575
#> Fold 2 0.0478 0.0532 0.0518
#> Fold 3 0.0574 0.0473
#> Fold 4 0.0628
#> Fold 5 0.0518
#> Fold 6 0.0521
#> Fold 7 0.0550
#> Fold 8
#> Fold 9 0.0556 0.0447
#> Fold 10 0.0516
#> X.6 X.72 X.73 X.77 X.79 X.88
#> Fold 1 0.0476
#> Fold 2 0.0514
#> Fold 3
#> Fold 4 -0.0473
#> Fold 5 0.0586 0.07
#> Fold 6 -0.0489
#> Fold 7
#> Fold 8
#> Fold 9
#> Fold 10
For a contrasting example we apply model selection to Mroz’s logistic
regression for married women’s labor-force participation. First, recall
the logistic regression model that we fit to the Mroz
data:
summary(m.mroz)
#>
#> Call:
#> glm(formula = lfp ~ ., family = binomial, data = Mroz)
#>
#> Coefficients:
#> Estimate Std. Error z value Pr(>|z|)
#> (Intercept) 3.18214 0.64438 4.94 7.9e-07 ***
#> k5 -1.46291 0.19700 -7.43 1.1e-13 ***
#> k618 -0.06457 0.06800 -0.95 0.34234
#> age -0.06287 0.01278 -4.92 8.7e-07 ***
#> wcyes 0.80727 0.22998 3.51 0.00045 ***
#> hcyes 0.11173 0.20604 0.54 0.58762
#> lwg 0.60469 0.15082 4.01 6.1e-05 ***
#> inc -0.03445 0.00821 -4.20 2.7e-05 ***
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#> (Dispersion parameter for binomial family taken to be 1)
#>
#> Null deviance: 1029.75 on 752 degrees of freedom
#> Residual deviance: 905.27 on 745 degrees of freedom
#> AIC: 921.3
#>
#> Number of Fisher Scoring iterations: 4
Applying stepwise model selection Mroz’s logistic regression, using
BIC as the model-selection criterion (via the argument
k=log(nrow(Mroz))
to stepAIC()
) selects 5 of
the 7 original predictors:
m.mroz.sel <- MASS::stepAIC(m.mroz, k=log(nrow(Mroz)),
trace=FALSE)
summary(m.mroz.sel)
#>
#> Call:
#> glm(formula = lfp ~ k5 + age + wc + lwg + inc, family = binomial,
#> data = Mroz)
#>
#> Coefficients:
#> Estimate Std. Error z value Pr(>|z|)
#> (Intercept) 2.9019 0.5429 5.35 9.0e-08 ***
#> k5 -1.4318 0.1932 -7.41 1.3e-13 ***
#> age -0.0585 0.0114 -5.13 2.9e-07 ***
#> wcyes 0.8724 0.2064 4.23 2.4e-05 ***
#> lwg 0.6157 0.1501 4.10 4.1e-05 ***
#> inc -0.0337 0.0078 -4.32 1.6e-05 ***
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#> (Dispersion parameter for binomial family taken to be 1)
#>
#> Null deviance: 1029.75 on 752 degrees of freedom
#> Residual deviance: 906.46 on 747 degrees of freedom
#> AIC: 918.5
#>
#> Number of Fisher Scoring iterations: 3
BayesRule(Mroz$lfp == "yes",
predict(m.mroz.sel, type="response"))
#> [1] 0.31873
The Bayes rule applied to the selected model misclassifies 32% of the
cases in the Mroz
data.
Cross-validating the selected model produces a similar, slightly larger, estimate of misclassification, about 33%:
cv(m.mroz.sel, criterion=BayesRule, seed=345266)
#> R RNG seed set to 345266
#> 10-Fold Cross Validation
#> method: exact
#> criterion: BayesRule
#> cross-validation criterion = 0.33068
#> bias-adjusted cross-validation criterion = 0.33332
#> full-sample criterion = 0.31873
Is this estimate of predictive performance optimistic?
We proceed to apply the model-selection procedure by cross-validation, producing more or less the same result:
m.mroz.sel.cv <- cvSelect(selectStepAIC, Mroz,
seed=6681,
criterion=BayesRule,
model=m.mroz,
AIC=FALSE)
#> R RNG seed set to 6681
m.mroz.sel.cv
#> 10-Fold Cross Validation
#> cross-validation criterion = 0.33068
#> bias-adjusted cross-validation criterion = 0.33452
#> full-sample criterion = 0.31873
Setting AIC=FALSE
in the call to cvSelect()
uses the BIC rather than the AIC as the model-selection criterion. As it
turns out, exactly the same predictors are selected when each of the 10
folds are omitted, and the several coefficient estimates are very
similar, as we show using compareFolds()
:
compareFolds(m.mroz.sel.cv)
#> (Intercept) age inc k5 lwg wcyes
#> Fold 1 2.5014 -0.0454 -0.0388 -1.3613 0.5653 0.85
#> Fold 2 3.0789 -0.0659 -0.0306 -1.5335 0.6923 0.79
#> Fold 3 3.0141 -0.0595 -0.0305 -1.3994 0.5428 0.86
#> Fold 4 2.7251 -0.0543 -0.0354 -1.4474 0.6298 1.09
#> Fold 5 2.7617 -0.0566 -0.0320 -1.4752 0.6324 0.74
#> Fold 6 3.0234 -0.0621 -0.0348 -1.4537 0.6618 0.94
#> Fold 7 2.9615 -0.0600 -0.0351 -1.4127 0.5835 0.97
#> Fold 8 2.9598 -0.0603 -0.0329 -1.3865 0.6210 0.69
#> Fold 9 3.2481 -0.0650 -0.0381 -1.4138 0.6093 0.94
#> Fold 10 2.7724 -0.0569 -0.0295 -1.4503 0.6347 0.85
In this example, therefore, we appear to obtain a realistic estimate of model performance directly from the selected model, because there is little added uncertainty induced by model selection.
The cv package also provides a
cvSelect()
procedure, selectTrans()
, for
choosing transformations of the predictors and the response in
regression.
Some background: As Weisberg (2014, sec. 8.2) explains, there are technical advantages to having (numeric) predictors in linear regression analysis that are themselves linearly related. If the predictors aren’t linearly related, then the relationships between them can often be straightened by power transformations. Transformations can be selected after graphical examination of the data, or by analytic methods. Once the relationships between the predictors are linearized, it can be advantageous similarly to transform the response variable towards normality.
Selecting transformations analytically raises the possibility of automating the process, as would be required for cross-validation. One could, in principle, apply graphical methods to select transformations for each fold, but because a data analyst couldn’t forget the choices made for previous folds, the process wouldn’t really be applied independently to the folds.
To illustrate, we adapt an example appearing in several places in
Fox & Weisberg (2019) (for example in
Chapter 3 on transforming data), using data on the prestige and other
characteristics of 102 Canadian occupations circa 1970. The data are in
the Prestige
data frame in the carData
package:
data("Prestige", package="carData")
head(Prestige)
#> education income women prestige census type
#> gov.administrators 13.11 12351 11.16 68.8 1113 prof
#> general.managers 12.26 25879 4.02 69.1 1130 prof
#> accountants 12.77 9271 15.70 63.4 1171 prof
#> purchasing.officers 11.42 8865 9.11 56.8 1175 prof
#> chemists 14.62 8403 11.68 73.5 2111 prof
#> physicists 15.64 11030 5.13 77.6 2113 prof
summary(Prestige)
#> education income women prestige census
#> Min. : 6.38 Min. : 611 Min. : 0.00 Min. :14.8 Min. :1113
#> 1st Qu.: 8.45 1st Qu.: 4106 1st Qu.: 3.59 1st Qu.:35.2 1st Qu.:3120
#> Median :10.54 Median : 5930 Median :13.60 Median :43.6 Median :5135
#> Mean :10.74 Mean : 6798 Mean :28.98 Mean :46.8 Mean :5402
#> 3rd Qu.:12.65 3rd Qu.: 8187 3rd Qu.:52.20 3rd Qu.:59.3 3rd Qu.:8312
#> Max. :15.97 Max. :25879 Max. :97.51 Max. :87.2 Max. :9517
#> type
#> bc :44
#> prof:31
#> wc :23
#> NA's: 4
#>
#>
The variables in the Prestige
data set are:
education
: average years of education for incumbents in
the occupation, from the 1971 Canadian Census.income
: average dollars of annual income for the
occupation, from the Census.women
: percentage of occupational incumbents who were
women, also from the Census.prestige
: the average prestige rating of the occupation
on a 0–100 “thermometer” scale, in a Canadian social survey conducted
around the same time.type
, type of occupation, and census
, the
Census occupational code, which are not used in our example.The object of a regression analysis for the Prestige
data (and their original purpose) is to predict occupational prestige
from the other variables in the data set.
A scatterplot matrix (using the scatterplotMatrix()
function in the car package) of the numeric variables
in the data reveals that the distributions of income
and
women
are positively skewed, and that some of the
relationships among the three predictors, and between the predictors and
the response (i.e., prestige
), are nonlinear:
library("car")
#> Loading required package: carData
scatterplotMatrix(~ prestige + income + education + women,
data=Prestige, smooth=list(spread=FALSE))
Scatterplot matrix for the Prestige
data.
The powerTransform()
function in the
car package transforms variables towards multivariate
normality by a generalization of Box and Cox’s maximum-likelihood-like
approach (Box & Cox, 1964). Several
“families” of power transformations can be used, including the original
Box-Cox family, simple powers (and roots), and two adaptations of the
Box-Cox family to data that may include negative values and zeros: the
Box-Cox-with-negatives family and the Yeo-Johnson family; see Weisberg (2014, Chapter 8), and Fox & Weisberg (2019, Chapter 3) for
details. Because women
has some zero values, we use the
Yeo-Johnson family:
trans <- powerTransform( cbind(income, education, women) ~ 1,
data=Prestige, family="yjPower")
summary(trans)
#> yjPower Transformations to Multinormality
#> Est Power Rounded Pwr Wald Lwr Bnd Wald Upr Bnd
#> income 0.2678 0.33 0.1051 0.4304
#> education 0.5162 1.00 -0.2822 1.3145
#> women 0.1630 0.16 0.0112 0.3149
#>
#> Likelihood ratio test that all transformation parameters are equal to 0
#> LRT df pval
#> LR test, lambda = (0 0 0) 15.739 3 0.00128
We thus have evidence of the desirability of transforming
income
(by the \(1/3\)
power) and women
(by the \(0.16\) power—which is close to the “0”
power, i.e., the log transformation), but not education
.
Applying the “rounded” power transformations makes the predictors
better-behaved:
P <- Prestige[, c("prestige", "income", "education", "women")]
(lambdas <- trans$roundlam)
#> income education women
#> 0.33000 1.00000 0.16302
names(lambdas) <- c("income", "education", "women")
for (var in c("income", "education", "women")){
P[, var] <- yjPower(P[, var], lambda=lambdas[var])
}
summary(P)
#> prestige income education women
#> Min. :14.8 Min. :22.2 Min. : 6.38 Min. :0.00
#> 1st Qu.:35.2 1st Qu.:44.2 1st Qu.: 8.45 1st Qu.:1.73
#> Median :43.6 Median :50.3 Median :10.54 Median :3.36
#> Mean :46.8 Mean :50.8 Mean :10.74 Mean :3.50
#> 3rd Qu.:59.3 3rd Qu.:56.2 3rd Qu.:12.65 3rd Qu.:5.59
#> Max. :87.2 Max. :83.6 Max. :15.97 Max. :6.83
scatterplotMatrix(~ prestige + income + education + women,
data=P, smooth=list(spread=FALSE))
Scatterplot matrix for the Prestige
data with the
predictors transformed.
Comparing the MSE for the regressions with the original and transformed predictors shows a advantage to the latter:
m.pres <- lm(prestige ~ income + education + women, data=Prestige)
m.pres.trans <- lm(prestige ~ income + education + women, data=P)
mse(Prestige$prestige, fitted(m.pres))
#> [1] 59.153
mse(P$prestige, fitted(m.pres.trans))
#> [1] 50.6
Similarly, component+residual plots for the two regressions, produced
by the crPlots()
function in the car
package, suggest that the partial relationship of prestige
to income
is more nearly linear in the transformed data,
but the transformation of women
fails to capture what
appears to be a slight quadratic partial relationship; the partial
relationship of prestige
to education
is close
to linear in both regressions:
Component+residual plots for the Prestige
regression with
the original predictors.
Component+residual plots for the Prestige
regression with
transformed predictors.
Having transformed the predictors towards multinormality, we now
consider whether there’s evidence for transforming the response (using
powerTransform()
for Box and Cox’s original method), and we
discover that there’s not:
summary(powerTransform(m.pres.trans))
#> bcPower Transformation to Normality
#> Est Power Rounded Pwr Wald Lwr Bnd Wald Upr Bnd
#> Y1 1.0194 1 0.6773 1.3615
#>
#> Likelihood ratio test that transformation parameter is equal to 0
#> (log transformation)
#> LRT df pval
#> LR test, lambda = (0) 32.217 1 1.38e-08
#>
#> Likelihood ratio test that no transformation is needed
#> LRT df pval
#> LR test, lambda = (1) 0.012384 1 0.911
The selectTrans()
function in the cv
package automates the process of selecting predictor and response
transformations. The function takes a data
set and
“working” model
as arguments, along with the candidate
predictors
and response
for transformation,
and the transformation family
to employ. If the
predictors
argument is missing then only the response is
transformed, and if the response
argument is missing, only
the supplied predictors are transformed. The default family
for transforming the predictors is "bcPower"
—the original
Box-Cox family—as is the default family.y
for transforming
the response; here we specify family="yjPower
because of
the zeros in women
. selectTrans()
returns the
result of applying a lack-of-fit criterion to the model after the
selected transformation is applied, with the default
criterion=mse
:
selectTrans(data=Prestige, model=m.pres,
predictors=c("income", "education", "women"),
response="prestige", family="yjPower")
#> [1] 50.6
selectTrans()
also takes an optional
indices
argument, making it suitable for doing computations
on a subset of the data (i.e., a CV fold), and hence for use with
cvSelect()
(see ?selectTrans
for details):
cvs <- cvSelect(selectTrans, data=Prestige, model=m.pres, seed=1463,
predictors=c("income", "education", "women"),
response="prestige",
family="yjPower")
#> R RNG seed set to 1463
cvs
#> 10-Fold Cross Validation
#> cross-validation criterion = 54.487
#> bias-adjusted cross-validation criterion = 54.308
#> full-sample criterion = 50.6
cv(m.pres, seed=1463) # untransformed model with same folds
#> R RNG seed set to 1463
#> 10-Fold Cross Validation
#> method: Woodbury
#> criterion: mse
#> cross-validation criterion = 63.293
#> bias-adjusted cross-validation criterion = 63.073
#> full-sample criterion = 59.153
compareFolds(cvs)
#> lam.education lam.income lam.women lamda.y
#> Fold 1 1.000 0.330 0.330 1
#> Fold 2 1.000 0.330 0.169 1
#> Fold 3 1.000 0.330 0.330 1
#> Fold 4 1.000 0.330 0.330 1
#> Fold 5 1.000 0.330 0.000 1
#> Fold 6 1.000 0.330 0.330 1
#> Fold 7 1.000 0.330 0.330 1
#> Fold 8 1.000 0.330 0.000 1
#> Fold 9 1.000 0.330 0.000 1
#> Fold 10 1.000 0.330 0.000 1
The results suggest that the predictive power of the transformed
regression is reliably greater than that of the untransformed regression
(though in both case, the cross-validated MSE is considerably higher
than the MSE computed for the whole data). Examining the selected
transformations for each fold reveals that the predictor
education
and the response prestige
are never
transformed; that the \(1/3\) power is
selected for income
in all of the folds; and that the
transformation selected for women
varies narrowly across
the folds between the \(0\)th power
(i.e., log) and the \(1/3\) power.
The CV functions in the cv package are all capable
of performing parallel computations by setting the ncores
argument (specifying the number of computer cores to be used) to a
number > 1
(which is the default). Parallel computation
can be advantageous for large problems, reducing the execution time of
the program.
To illustrate, let’s time model selection in Mroz’s logistic regression, repeating the computation as performed previously and then doing it in parallel using 2 cores:
system.time(m.mroz.sel.cv <- cvSelect(selectStepAIC, Mroz,
seed=6681,
criterion=BayesRule,
model=m.mroz,
AIC=FALSE))
#> R RNG seed set to 6681
#> user system elapsed
#> 0.237 0.003 0.241
system.time(m.mroz.sel.cv.p <- cvSelect(selectStepAIC, Mroz,
seed=6681,
criterion=BayesRule,
model=m.mroz,
AIC=FALSE,
ncores=2))
#> R RNG seed set to 6681
#> user system elapsed
#> 0.036 0.007 1.471
all.equal(m.mroz.sel.cv, m.mroz.sel.cv.p)
#> [1] TRUE
In this small problem, the parallel computation is actually slower, because there is an overhead cost to parallelization, but we can see that it produces the same result as before.
The most straightforward way to implement cross-validation in R for
statistical modeling functions that are written in the canonical manner
is to use update()
to refit the model with each fold
removed. This is the approach taken in the default method for
cv()
, and it is appropriate if the cases are independently
sampled. Refitting the model in this manner for each fold is generally
feasible when the number of folds in modest, but can be prohibitively
costly for leave-one-out cross-validation when the number of cases is
large.
The "lm"
and "glm"
methods for
cv()
take advantage of computational efficiencies by
avoiding refitting the model with each fold removed. Consider, in
particular, the weighted linear model \(\mathbf{y}_{n \times 1} = \mathbf{X}_{n \times
p}\boldsymbol{\beta}_{p \times 1} + \boldsymbol{\varepsilon}_{n \times
1}\), where \(\boldsymbol{\varepsilon}
\sim \mathbf{N}_n \left(\mathbf{0}, \sigma^2 \mathbf{W}^{-1}_{n \times
n}\right)\). Here, \(\mathbf{y}\) is the response vector, \(\mathbf{X}\) the model matrix, and \(\boldsymbol{\varepsilon}\) the error
vector, each for \(n\) cases, and \(\boldsymbol{\beta}\) is the vector of \(p\) population regression coefficients. The
errors are assumed to be multivariately normally distributed with 0
means and covariance matrix \(\sigma^2
\mathbf{W}^{-1}\), where \(\mathbf{W} =
\mathrm{diag}(w_i)\) is a diagonal matrix of inverse-variance
weights. For the linear model with constant error variance, the weight
matrix is taken to be \(\mathbf{W} =
\mathbf{I}_n\), the order-\(n\)
identity matrix.
The weighted-least-squares (WLS) estimator of \(\boldsymbol{\beta}\) is (see, e.g., Fox, 2016, sec. 12.2.2) 11 \[ \mathbf{b}_{\mathrm{WLS}} = \left( \mathbf{X}^T \mathbf{W} \mathbf{X} \right)^{-1} \mathbf{X}^T \mathbf{W} \mathbf{y} \]
Fitted values are then \(\widehat{\mathbf{y}} = \mathbf{X}\mathbf{b}_{\mathrm{WLS}}\).
The LOO fitted value for the \(i\)th case can be efficiently computed by \(\widehat{y}_{-i} = y_i - e_i/(1 - h_i)\) where \(h_i = \mathbf{x}^T_i \left( \mathbf{X}^T \mathbf{W} \mathbf{X} \right)^{-1} \mathbf{x}_i\) (the so-called “hatvalue”). Here, \(\mathbf{x}^T_i\) is the \(i\)th row of \(\mathbf{X}\), and \(\mathbf{x}_i\) is the \(i\)th row written as a column vector. This approach can break down when one or more hatvalues are equal to 1, in which case the formula for \(\widehat{y}_{-i}\) requires division by 0.
To compute cross-validated fitted values when the folds contain more than one case, we make use of the Woodbury matrix identify ("Woodbury matrix identity", 2023), \[ \left(\mathbf{A}_{m \times m} + \mathbf{U}_{m \times k} \mathbf{C}_{k \times k} \mathbf{V}_{k \times m} \right)^{-1} = \mathbf{A}^{-1} - \mathbf{A}^{-1}\mathbf{U} \left(\mathbf{C}^{-1} + \mathbf{VA}^{-1}\mathbf{U} \right)^{-1} \mathbf{VA}^{-1} \] where \(\mathbf{A}\) is a nonsingular order-\(n\) matrix. We apply this result by letting \[\begin{align*} \mathbf{A} &= \mathbf{X}^T \mathbf{W} \mathbf{X} \\ \mathbf{U} &= \mathbf{X}_\mathbf{j}^T \\ \mathbf{V} &= - \mathbf{X}_\mathbf{j} \\ \mathbf{C} &= \mathbf{W}_\mathbf{j} \\ \end{align*}\] where the subscript \(\mathbf{j} = (i_{j1}, \ldots, i_{jm})^T\) represents the vector of indices for the cases in the \(j\)th fold, \(j = 1, \ldots, k\). The negative sign in \(\mathbf{V} = - \mathbf{X}_\mathbf{j}\) reflects the removal, rather than addition, of the cases in \(\mathbf{j}\).
Applying the Woodbury identity isn’t quite as fast as using the
hatvalues, but it is generally much faster than refitting the model. A
disadvantage of the Woodbury identity, however, is that it entails
explicit matrix inversion and thus may be numerically unstable. The
inverse of \(\mathbf{A} = \mathbf{X}^T
\mathbf{W} \mathbf{X}\) is available directly in the
"lm"
object, but the second term on the right-hand side of
the Woodbury identity requires a matrix inversion with each fold
deleted. (In contrast, the inverse of each \(\mathbf{C} = \mathbf{W}_\mathbf{j}\) is
straightforward because \(\mathbf{W}\)
is diagonal.)
The Woodbury identity also requires that the model matrix be of full rank. We impose that restriction in our code by removing redundant regressors from the model matrix for all of the cases, but that doesn’t preclude rank deficiency from surfacing when a fold is removed. Rank deficiency of \(\mathbf{X}\) doesn’t disqualify cross-validation because all we need are fitted values under the estimated model.
glm()
computes the maximum-likelihood estimates for a
generalized linear model by iterated weighted least squares (see, e.g., Fox & Weisberg, 2019, sec.
6.12). The last iteration is therefore just a WLS fit of the
“working response” on the model matrix using “working weights.” Both the
working weights and the working response at convergence are available
from the information in the object returned by glm()
.
We then treat re-estimation of the model with a case or cases deleted
as a WLS problem, using the hatvalues or the Woodbury matrix identity.
The resulting fitted values for the deleted fold aren’t exact—that is,
except for the Gaussian family, the result isn’t identical to what we
would obtain by literally refitting the model—but in our (limited)
experience, the approximation is very good, especially for LOO CV, which
is when we would be most tempted to use it. Nevertheless, because these
results are approximate, the default for the "glm"
cv()
method is to perform the exact computation, which
entails refitting the model with each fold omitted.
Let \(\mathrm{CV}(\mathbf{y}, \widehat{\mathbf{y}})\) represent a cross-validation cost criterion, such as mean-squared error, computed for all of the \(n\) values of the response \(\mathbf{y}\) based on fitted values \(\widehat{\mathbf{y}}\) from the model fit to all of the data. We divide the \(n\) cases into \(k\) folds of approximately \(n_j \approx n/k\) cases each, where \(n = \sum n_j\). As above, let \(\mathbf{j}\) denote the indices of the cases in the \(j\)th fold.
Now define \(\mathrm{CV}_j = \mathrm{CV}(\mathbf{y}, \widehat{\mathbf{y}}^{(j)})\) and \(\mathrm{CV}_j^{(j)} = \mathrm{CV}(\mathbf{y}_{\mathbf{j}}, \widehat{\mathbf{y}}^{(j)}_{\mathbf{j}})\). The superscript \((j)\) on \(\widehat{\mathbf{y}}^{(j)}\) and \(\widehat{\mathbf{y}}^{(j)}_{\mathbf{j}}\) represents fitted values computed from the model with fold \(j\) omitted, respectively for all of the cases (i.e., \(\mathrm{CV}_j\)) and for the cases only in the \(j\)th fold (\(\mathrm{CV}_j^{(j)}\)).
Then the cross-validation criterion is the weighted average of the \(\mathrm{CV}_j^{(j)}\) \[ \mathrm{CV} = \frac{1}{n} \sum_{j = 1}^{k} n_j \mathrm{CV}_j^{(j)} \] Following Davison & Hinkley (1997, pp. 293–295), the bias-adjusted cross-validation criterion is \[ \mathrm{CV}_{\mathrm{adj}} = \mathrm{CV} + \mathrm{CV}(\mathbf{y}, \widehat{\mathbf{y}}) - \frac{1}{n} \sum_{j=1}^{k} n_j \mathrm{CV}_j \]
James et al. (2021) use
the cv.glm()
function in the boot package
(Canty & Ripley, 2022; Davison & Hinkley,
1997). Despite its name, cv.glm()
is an independent
function and not a method of a cv()
generic function.↩︎
Although it serves to illustrate the use of CV, a
polynomial is probably not the best choice here. Consider, for example
the scatterplot for log-transformed mpg
and
horsepower
, produced by
plot(mpg ~ horsepower, data=Auto, log="xy")
(execution of
which is left to the reader).↩︎
Out of impatience, we asked
microbenchmark()
to execute each command only 10 times
rather than the default 100. With the exception of the last columns, the
output is self-explanatory. The last column shows which methods have
average timings that are statistically distinguishable. Because of the
small number of repetitions (i.e., 10), the "hatvalues"
and
"Woodbury"
methods aren’t distinguishable, but the
difference between these methods persists when we perform more
repetitions—we invite the reader to redo this computation with the
default times=100
repetitions.↩︎
BayesRule()
does some error checking;
BayesRule2()
is similar, but omits the error checking, and
so can be faster for large problems.↩︎
There are, however, more complex situations that give
rise to so-called crossed (rather than nested) random
effects. For example, consider students within classes within schools.
In primary schools, students typically are in a single class, and so
classes are nested within schools. In secondary schools, however,
students typically take several classes and students who are together in
a particular class may not be together in other classes; consequently,
random effects based on classes within schools are crossed. Although the
lmer()
function in the lme4 package is
capable of modeling both nested and crossed random effects, the
cv()
methods for mixed models in the cv
package pertain only to nested random effects. It may be possible to
cross-validate mixed models with crossed random effects, but we don’t
know how to do it.↩︎
The observant reader will notice that we set the
argument control=list(opt="optim")
in the call to
lme()
, changing the optimizer employed from the default
"nlminb"
. We did this because with the default optimizer,
lme()
encountered the same convergence issue as
lmer()
, but rather than issuing a warning,
lme()
failed, reporting an error. As it turns out, setting
the optimizer to "optim"
avoids this problem.↩︎
We invite the interested reader to experiment with varying the parameters of our example.↩︎
We find it convenient to use the lattice (Sarkar, 2008) and latticeExtra (Sarkar & Andrews, 2022) packages for this and other graphs in this section.↩︎
Because of the manner in which the computation is
performed, the order of the replicates in the "cvList"
object returned by cv()
isn’t the same as the order in
which the replicates are computed. Each element of the result, however,
is a "cv"
object with the correct random-number seed saved,
and so this technical detail can be safely ignored. The individual
"cv"
objects are printed in the order in which they are
stored rather than the order in which they are computed.↩︎
It’s generally advantageous to start with the largest model, here the one with 100 predictors, and proceed by backward elimination. In this demonstration, however, where all of the \(\beta\)s are really 0, the selected model will be small, and so we proceed by forward selection from the null model to save computing time.↩︎
This is a definitional formula, which assumes that the
model matrix \(\mathbf{X}\) is of full
column rank, and which can be subject to numerical instability when
\(\mathbf{X}\) is ill-conditioned.
lm()
uses the singular-value decomposition of the model
matrix to obtain computationally more stable results.↩︎