flare: A Family of Lasso Regression Methods

Description

The package "flare" provides a family of regression methods (Lasso, Dantzig Selector, LAD Lasso, SQRT Lasso, and Lq Lasso) and their extensions to sparse precision matrix estimation (TIGER and CLIME using L1) in high dimensions. We adopt the alternating direction method of multipliers and convert the original optimization problem into a sequence of L1-penalized least squares minimization problems with linearization and multi-stage screening of variables. Missing values can be tolerated for Dantzig selector in the design matrix and response vector, and for CLIME in the data matrix. Computation is memory-optimized using sparse matrix output. In addition, we provide convenient tools for regularization parameter selection and visualization.

Details

Author(s)

Xingguo Li, Tuo Zhao, Lie Wang , Xiaoming Yuan and Han Liu
Maintainer: Tuo Zhao <tourzhao@gatech.edu>

References

1. E. Candes and T. Tao. The Dantzig selector: Statistical estimation when p is much larger than n. Annals of Statistics, 2007.
2. A. Belloni, V. Chernozhukov and L. Wang. Pivotal recovery of sparse signals via conic programming. Biometrika, 2012.
3. L. Wang. L1 penalized LAD estimator for high dimensional linear regression. Journal of Multivariate Analysis, 2012.
4. J. Liu and J. Ye. Efficient L1/Lq Norm Regularization. Technical Report, 2010.
5. T. Cai, W. Liu and X. Luo. A constrained \ell_1 minimization approach to sparse precision matrix estimation. Journal of the American Statistical Association, 2011.
6. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers. Foundations and Trends in Machine Learning, 2011.
7. H. Liu and L. Wang. TIGER: A tuning-insensitive approach for optimally estimating large undirected graphs. Technical Report, 2012.
8. B. He and X. Yuan. On non-ergodic convergence rate of Douglas-Rachford alternating direction method of multipliers. Technical Report, 2012.

See Also

sugm and slim.

Extract Model Coefficients for an object with S3 class "slim"

Description

Extract estimated regression coefficient vectors from the solution path.

Usage

## S3 method for class 'slim'
coef(object, lambda.idx = c(1:3), beta.idx = c(1:3), ...)

Arguments

Author(s)

Xingguo Li, Tuo Zhao, Lie Wang, Xiaoming Yuan and Han Liu
Maintainer: Tuo Zhao <tourzhao@gatech.edu>

See Also

slim and flare-package.

The Bardet-Biedl syndrome Gene expression data from Scheetz et al. (2006)

Description

Gene expression data (20 genes for 120 samples) from the microarray experiments of mammalianeye tissue samples of Scheetz et al. (2006).

Usage

data(eyedata)

Format

The format is a list containing conatins a matrix and a vector. 1. x - an 120 by 200 matrix, which represents the data of 120 rats with 200 gene probes. 2. y - a 120-dimensional vector of, which represents the expression level of TRIM32 gene.

Details

This data set contains 120 samples with 200 predictors

Author(s)

Xingguo Li, Tuo Zhao, Lie Wang, Xiaoming Yuan and Han Liu
Maintainer: Tuo Zhao <tourzhao@gatech.edu>

References

1. T. Scheetz, k. Kim, R. Swiderski, A. Philp, T. Braun, K. Knudtson, A. Dorrance, G. DiBona, J. Huang, T. Casavant, V. Sheffield, E. Stone .Regulation of gene expression in the mammalian eye and its relevance to eye disease. Proceedings of the National Academy of Sciences of the United States of America, 2006.

See Also

flare-package.

Examples

data(eyedata)
image(x)

Internal flare functions

Description

Internal flare functions

Usage

sugm.likelihood(Sigma, Omega)
sugm.tracel2(Sigma, Omega)
sugm.cv(obj, loss=c("likelihood", "tracel2"), fold=5)
part.cv(n, fold)
sugm.clime.ladm.scr(Sigma, lambda, nlambda, n, d, maxdf, rho, shrink, prec, 
                    max.ite, verbose)
sugm.tiger.ladm.scr(data, n, d, maxdf, rho, lambda, shrink, prec, 
                    max.ite, verbose)
slim.lad.ladm.scr.btr(Y, X, lambda, nlambda, n, d, maxdf, rho, max.ite, prec, 
                      intercept, verbose)
slim.sqrt.ladm.scr(Y, X, lambda, nlambda, n, d, maxdf, rho, max.ite, prec, 
                   intercept, verbose)
slim.dantzig.ladm.scr(Y, X, lambda, nlambda, n, d, maxdf, rho, max.ite, prec, 
                      intercept, verbose)
slim.lq.ladm.scr.btr(Y, X, q, lambda, nlambda, n, d, maxdf, rho, max.ite, prec, 
                     intercept, verbose)
slim.lasso.ladm.scr(Y, X, lambda, nlambda, n, d, maxdf, max.ite, prec, 
                    intercept, verbose)

Arguments

Details

These are not intended for use by users.

Author(s)

Xingguo Li, Tuo Zhao, Lie Wang, Xiaoming Yuan and Han Liu
Maintainer: Tuo Zhao <tourzhao@gatech.edu>

See Also

sugm, slim and flare-package.

Plot Function for "roc"

Description

Plot the ROC curve for an object with S3 class "roc"

Usage

## S3 method for class 'roc'
plot(x, ...)

Arguments

Author(s)

Xingguo Li, Tuo Zhao, Lie Wang, Xiaoming Yuan and Han Liu
Maintainer: Tuo Zhao <tourzhao@gatech.edu>

See Also

sugm.roc, sugm and flare-package.

Plot Function for "select"

Description

Plot the optimal graph by model selection.

Usage

## S3 method for class 'select'
plot(x, ...)

Arguments

Author(s)

Xingguo Li, Tuo Zhao, Lie Wang, Xiaoming Yuan and Han Liu
Maintainer: Tuo Zhao <tourzhao@gatech.edu>

See Also

sugm and sugm.select

Plot Function for "sim"

Description

Visualize the covariance matrix, the empirical covariance matrix, the adjacency matrix and the graph pattern of the true graph structure.

Usage

## S3 method for class 'sim'
plot(x, ...)

Arguments

Author(s)

Xingguo Li, Tuo Zhao, Lie Wang, Xiaoming Yuan and Han Liu
Maintainer: Tuo Zhao <tourzhao@gatech.edu>

See Also

sugm.generator, sugm and flare-package

Plot Function for "slim"

Description

Visualize the solution path of regression estimates corresponding to regularization parameters.

Usage

## S3 method for class 'slim'
plot(x, ...)

Arguments

Author(s)

Xingguo Li, Tuo Zhao, Lie Wang, Xiaoming Yuan and Han Liu
Maintainer: Tuo Zhao <tourzhao@gatech.edu>

See Also

slim and flare-package.

Plot Function for "sugm"

Description

Plot sparsity level information and 3 typical sparse graphs from the graph path.

Usage

## S3 method for class 'sugm'
plot(x, align = FALSE, ...)

Arguments

Author(s)

Xingguo Li, Tuo Zhao, Lie Wang, Xiaoming Yuan and Han Liu
Maintainer: Tuo Zhao <tourzhao@gatech.edu>

See Also

sugm and flare-package

Prediction for an object with S3 class "slim"

Description

Predict responses for new design data.

Usage

## S3 method for class 'slim'
predict(object, newdata, lambda.idx = c(1:3), Y.pred.idx = c(1:5), ...)

Arguments

Details

predict.slim produces predicted values of the responses of the newdata from the estimated beta values in the object, i.e.

\hat{Y} = \hat{\beta}_0 + X_{new} \hat{\beta}.

Value

Author(s)

Xingguo Li, Tuo Zhao, Lie Wang, Xiaoming Yuan and Han Liu
Maintainer: Tuo Zhao <tourzhao@gatech.edu>

See Also

slim and flare-package.

Examples

## load library
library(flare)
## generate data
set.seed(123)
n = 100
d = 200
d1 = 10
rho0 = 0.3
lambda = c(3:1)*sqrt(log(d)/n)
Sigma = matrix(0,nrow=d,ncol=d)
Sigma[1:d1,1:d1] = rho0
diag(Sigma) = 1
mu = rep(0,d)
X = mvrnorm(n=2*n,mu=mu,Sigma=Sigma)
X.fit = X[1:n,]
X.pred = X[(n+1):(2*n),]
eps = rt(n=n,df=n-1)
beta = c(rep(sqrt(1/3),3),rep(0,d-3))
Y.fit = X.fit%*%beta+eps

## Regression with "dantzig".
out=slim(X=X.fit,Y=Y.fit,lambda=lambda,method = "lq",q=1)

## Display results
Y=predict(out,X.pred)

Print Function for an object with S3 class "roc"

Description

Print true positive rates, false positive rates, area under the curve, and maximum F1 score.

Usage

## S3 method for class 'roc'
print(x, ...)

Arguments

Author(s)

Xingguo Li, Tuo Zhao, Lie Wang, Xiaoming Yuan and Han Liu
Maintainer: Tuo Zhao <tourzhao@gatech.edu>

See Also

sugm.roc, sugm and flare-package

Print Function for an object with S3 class "select"

Description

Print information about model usage, graph dimension, selection criterion, and sparsity level of the optimal graph.

Usage

## S3 method for class 'select'
print(x, ...)

Arguments

Author(s)

Xingguo Li, Tuo Zhao, Lie Wang, Xiaoming Yuan and Han Liu
Maintainer: Tuo Zhao <tourzhao@gatech.edu>

See Also

sugm.select, sugm and flare-package

Print Function for an object with S3 class "sim"

Description

Print the information about the sample size, the dimension, the pattern and sparsity of the true graph structure.

Usage

## S3 method for class 'sim'
print(x, ...)

Arguments

Author(s)

Xingguo Li, Tuo Zhao, Lie Wang, Xiaoming Yuan and Han Liu
Maintainer: Tuo Zhao <tourzhao@gatech.edu>

See Also

sugm and sugm.generator

Print Function for an object with S3 class "slim"

Description

Print a summary of the information about an object with S3 class "slim".

Usage

## S3 method for class 'slim'
print(x, ...)

Arguments

Details

This call simply outlines the options used for computing a slim object.

Author(s)

Xingguo Li, Tuo Zhao, Lie Wang, Xiaoming Yuan and Han Liu
Maintainer: Tuo Zhao <tourzhao@gatech.edu>

See Also

slim and flare-package.

Print Function for an object with S3 class "sugm"

Description

Print a summary of information about an object with S3 class "sugm".

Usage

## S3 method for class 'sugm'
print(x, ...)

Arguments

Details

This call simply outlines the options used for computing a sugm object.

Author(s)

Xingguo Li, Tuo Zhao, Lie Wang, Xiaoming Yuan and Han Liu
Maintainer: Tuo Zhao <tourzhao@gatech.edu>

See Also

sugm and flare-package.

Sparse Linear Regression using Nonsmooth Loss Functions and L1 Regularization

Description

The function "slim" implements a family of Lasso variants for estimating high-dimensional sparse linear models, including Dantzig Selector, LAD Lasso, SQRT Lasso, and Lq Lasso. We adopt the alternating direction method of multipliers (ADMM) and convert the original optimization problem into a sequence of L1-penalized least squares minimization problems, which can be efficiently solved by combining linearization and multi-stage screening of variables. Missing values can be tolerated for Dantzig selector in the design matrix and response vector.

Usage

slim(X, Y, lambda = NULL, nlambda = NULL, 
     lambda.min.value = NULL, lambda.min.ratio = NULL, 
     rho = 1, method="lq", q = 2, res.sd = FALSE, 
     prec = 1e-5, max.ite = 1e5, verbose = TRUE)

Arguments

Details

Standard Lasso

\min {\frac{1}{2n}}|| Y - X \beta ||_2^2 + \lambda || \beta ||_1

Dantzig selector solves the following optimization problem

\min || \beta ||_1, \quad \textrm{s.t. } || X'(Y - X \beta) ||_{\infty} < \lambda

L_q loss Lasso solves the following optimization problem

\min n^{-\frac{1}{q}}|| Y - X \beta ||_q + \lambda || \beta ||_1

where 1<= q <=2. Lq Lasso is equivalent to LAD Lasso and SQRT Lasso when q=1 and q=2, respectively.

Value

An object with S3 class "slim" is returned:

Author(s)

Xingguo Li, Tuo Zhao, Lie Wang, Xiaoming Yuan and Han Liu
Maintainer: Tuo Zhao <tourzhao@gatech.edu>

References

1. E. Candes and T. Tao. The Dantzig selector: Statistical estimation when p is much larger than n. Annals of Statistics, 2007.
2. A. Belloni, V. Chernozhukov and L. Wang. Pivotal recovery of sparse signals via conic programming. Biometrika, 2012.
3. L. Wang. L1 penalized LAD estimator for high dimensional linear regression. Journal of Multivariate Analysis, 2012.
4. J. Liu and J. Ye. Efficient L1/Lq Norm Regularization. Technical Report, 2010.
5. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers. Foundations and Trends in Machine Learning, 2011.
6. B. He and X. Yuan. On non-ergodic convergence rate of Douglas-Rachford alternating direction method of multipliers. Technical Report, 2012.

See Also

flare-package, print.slim, plot.slim, coef.slim and predict.slim.

Examples

## load library
library(flare)
## generate data
n = 50
d = 100
X = matrix(rnorm(n*d), n, d)
beta = c(3,2,0,1.5,rep(0,d-4))
eps = rnorm(n)
Y = X%*%beta + eps
nlamb = 5
ratio = 0.3

## Regression with "dantzig", general "lq" and "lasso" respectively
out1 = slim(X=X,Y=Y,nlambda=nlamb,lambda.min.ratio=ratio,method="dantzig")
out2 = slim(X=X,Y=Y,nlambda=nlamb,lambda.min.ratio=ratio,method="lq",q=1)
out3 = slim(X=X,Y=Y,nlambda=nlamb,lambda.min.ratio=ratio,method="lq",q=1.5)
out4 = slim(X=X,Y=Y,nlambda=nlamb,lambda.min.ratio=ratio,method="lq",q=2)
out5 = slim(X=X,Y=Y,nlambda=nlamb,lambda.min.ratio=ratio,method="lasso")

## Display results
print(out4)
plot(out4)
coef(out4)

High-dimensional Sparse Undirected Graphical Models

Description

The function "sugm" estimates sparse undirected graphical models (Gaussian precision matrices) in high dimensions. Two procedures are implemented using a column-wise regression scheme: (1) Tuning-Insensitive Graph Estimation and Regression based on square-root Lasso ("tiger"); and (2) The Constrained L1 Minimization for Sparse Precision Matrix Estimation ("clime"). The optimization algorithm is based on the alternating direction method of multipliers (ADMM), linearization, and multi-stage screening. Missing values can be tolerated for CLIME when the input is a data matrix. Computation is memory-optimized using sparse matrix output.

Usage

sugm(data, lambda = NULL, nlambda = NULL, lambda.min.ratio = NULL, 
     rho = NULL, method = "tiger", sym = "or", shrink = NULL, 
     prec = 1e-4, max.ite = 1e4, standardize = FALSE, 
     perturb = TRUE, verbose = TRUE)

Arguments

Details

CLIME solves the following minimization problem

\min || \Omega ||_1 \quad \textrm{s.t. } || S \Omega - I ||_\infty \le \lambda,

where ||\cdot||_1 and ||\cdot||_\infty are element-wise 1-norm and \infty-norm respectively.

"tiger" solves the following minimization problem

\min ||X-XB||_{2,1} + \lambda ||B||_1 \quad \textrm{s.t. } B_{jj} = 0,

where ||\cdot||_{1} and ||\cdot||_{2,1} are element-wise 1-norm and L_{2,1}-norm respectively.

Value

An object with S3 class "sugm" is returned:

Author(s)

Xingguo Li, Tuo Zhao, Lie Wang, Xiaoming Yuan and Han Liu
Maintainer: Tuo Zhao <tourzhao@gatech.edu>

References

1. T. Cai, W. Liu and X. Luo. A constrained L1 minimization approach to sparse precision matrix estimation. Journal of the American Statistical Association, 2011.
2. H. Liu, L. Wang. TIGER: A tuning-insensitive approach for optimally estimating large undirected graphs. Technical Report, 2012.
3. B. He and X. Yuan. On non-ergodic convergence rate of Douglas-Rachford alternating direction method of multipliers. Technical Report, 2012.

See Also

flare-package, sugm.generator, sugm.select, sugm.plot, sugm.roc, plot.sugm, plot.select, plot.roc, plot.sim, print.sugm, print.select, print.roc and print.sim.

Examples

## load package required
library(flare)

## generating data
n = 50
d = 50
D = sugm.generator(n=n,d=d,graph="band",g=1)
plot(D)

## sparse precision matrix estimation with method "clime"
out1 = sugm(D$data, method = "clime")
plot(out1)
sugm.plot(out1$path[[4]])

## sparse precision matrix estimation with method "tiger"
out2 = sugm(D$data, method = "tiger")
plot(out2)
sugm.plot(out2$path[[5]])

Data generator for sparse undirected graph estimation.

Description

Implements the data generation from multivariate normal distributions with different graph structures, including "random", "hub", "cluster", "band", and "scale-free".

Usage

sugm.generator(n = 200, d = 50, graph = "random", v = NULL, u = NULL,
      g = NULL, prob = NULL, seed = NULL, vis = FALSE, verbose = TRUE)

Arguments

Details

Given the adjacency matrix theta, the graph patterns are generated as below:

(I) "random": Each pair of off-diagonal elements is randomly set to theta[i,j]=theta[j,i]=1 for i!=j with probability prob, and 0 otherwise. It results in about d*(d-1)*prob/2 edges in the graph.

(II)"hub":The row/columns are evenly partitioned into g disjoint groups. Each group is associated with a "center" row i in that group. Each pair of off-diagonal elements are set theta[i,j]=theta[j,i]=1 for i!=j if j also belongs to the same group as i and 0 otherwise. It results in d - g edges in the graph.

(III)"cluster": The rows/columns are evenly partitioned into g disjoint groups. Each pair of off-diagonal elements is set to theta[i,j]=theta[j,i]=1 for i!=j with probability prob if both i and j belong to the same group, and 0 otherwise. It results in about g*(d/g)*(d/g-1)*prob/2 edges in the graph.

(IV)"band": The off-diagonal elements are set to theta[i,j]=1 if 1<=|i-j|<=g and 0 otherwise. It results in (2d-1-g)*g/2 edges in the graph.

(V) "scale-free": The graph is generated using B-A algorithm. The initial graph has two connected nodes and each new node is connected to only one node in the existing graph with the probability proportional to the degree of the each node in the existing graph. It results in d edges in the graph.

The adjacency matrix theta has all diagonal elements equal to 0. To obtain a positive definite covariance matrix, the smallest eigenvalue of theta*v (denoted by e) is computed. Then we set the covariance matrix equal to cov2cor(solve(theta*v+(|e|+0.1+u)*I)) to generate multivariate normal data.

Value

An object with S3 class "sim" is returned:

Author(s)

Xingguo Li, Tuo Zhao, Lie Wang, Xiaoming Yuan and Han Liu
Maintainer: Tuo Zhao <tourzhao@gatech.edu>

See Also

flare and flare-package

Examples

## load package required
library(flare)

## band graph with bandwidth 3
L = sugm.generator(graph = "band", g = 3)
plot(L)

## random sparse graph
L = sugm.generator(vis = TRUE)

## hub graph with 6 hubs
L = sugm.generator(graph = "hub", g = 6, vis = TRUE)

## cluster graph with 8 clusters
L = sugm.generator(graph = "cluster", g = 8, vis = TRUE)

## scale-free graphs
L = sugm.generator(graph="scale-free", vis = TRUE)

Graph Visualization from an Adjacency Matrix

Description

Implements graph visualization using an adjacency matrix and automatically organizes a 2D embedding layout.

Usage

sugm.plot(G, epsflag = FALSE, graph.name = "default", cur.num = 1, 
          location)

Arguments

Details

The user can change cur.num to plot several figures and select the best one. The implementation is based on the popular package "igraph".

Author(s)

Xingguo Li, Tuo Zhao, Lie Wang, Xiaoming Yuan and Han Liu
Maintainer: Tuo Zhao <tourzhao@gatech.edu>

See Also

flare and flare-package

Examples

## load package required
library(flare)

## visualize the hub graph
L = sugm.generator(graph = "hub")
sugm.plot(L$theta)

## visualize the band graph
L = sugm.generator(graph = "band",g=5)
sugm.plot(L$theta)

## visualize the cluster graph
L = sugm.generator(graph = "cluster")
sugm.plot(L$theta)

## Not run: 
#show working directory
getwd()
#plot 5 graphs and save the plots as eps files in the working directory  
sugm.plot(L$theta, epsflag = TRUE, cur.num = 5)

## End(Not run)

Draw ROC Curve for an object with S3 class "sugm"

Description

Draws ROC curve for a graph path according to the true graph structure.

Usage

sugm.roc(path, theta, verbose = TRUE)

Arguments

Details

To avoid the horizontal oscillation, false positive rates is automatically sorted in the ascent oder and true positive rates also follow the same order.

Value

An object with S3 class "roc" is returned:

Note

For a lasso regression, the number of nonzero coefficients is at most n-1. If d>>n, even when regularization parameter is very small, the estimated graph may still be sparse. In this case, the AUC may not be a good choice to evaluate the performance.

Author(s)

Xingguo Li, Tuo Zhao, Lie Wang, Xiaoming Yuan and Han Liu
Maintainer: Tuo Zhao <tourzhao@gatech.edu>

See Also

sugm and flare-package

Examples

## load package required
library(flare)

#generate data
L = sugm.generator(d = 30, graph = "random", prob = 0.1)
out1 = sugm(L$data, lambda=10^(seq(log10(.4), log10(0.03), length.out=20)))

#draw ROC curve
Z1 = sugm.roc(out1$path,L$theta)

#Maximum F1 score
max(Z1$F1)

Model selection for high-dimensional undirected graphical models

Description

Implements regularization parameter selection for high-dimensional undirected graphical models. Supported approaches are Stability Approach to Regularization Selection ("stars") and cross-validation ("cv").

Usage

sugm.select(est, criterion = "stars", stars.subsample.ratio = NULL,
            stars.thresh = 0.1, rep.num = 20, fold = 5,
            loss="likelihood", verbose = TRUE)

Arguments

Details

Stability Approach to Regularization Selection (STARS) selects an optimal regularization parameter by variability across subsamples, and tends to over-select edges in Gaussian graphical models. In addition to selecting regularization parameters, STARS can provide an additional merged graph estimate based on edge frequencies across subsamples. K-fold cross-validation is also available for selecting lambda, using the following losses:

likelihood: Tr(\Sigma \Omega) - \log|\Omega|

tracel2: Tr(diag(\Sigma \Omega - I)^2).

Value

An object with S3 class "select" is returned:

and anything else included in the input est.

Note

Model selection is not available when the data input is a sample covariance matrix.

Author(s)

Xingguo Li, Tuo Zhao, Lie Wang, Xiaoming Yuan and Han Liu
Maintainer: Tuo Zhao <tourzhao@gatech.edu>

References

1. T. Cai, W. Liu and X. Luo. A constrained \ell_1 minimization approach to sparse precision matrix estimation. Journal of the American Statistical Association, 2011.
2. B. He and X. Yuan. On non-ergodic convergence rate of Douglas-Rachford alternating direction method of multipliers. Technical Report, 2012.

See Also

sugm and flare-package.

Examples

## load package required
library(flare)

#generate data
L = sugm.generator(d = 10, graph="hub")
out1 = sugm(L$data)

#model selection using stars
#out1.select1 = sugm.select(out1, criterion = "stars", stars.thresh = 0.1)
#plot(out1.select1)

#model selection using cross validation
out1.select2 = sugm.select(out1, criterion = "cv")
plot(out1.select2)